<!-----

Conversion time: 0.426 seconds.


Using this Markdown file:

1. Paste this output into your source file.
2. See the notes and action items below regarding this conversion run.
3. Check the rendered output (headings, lists, code blocks, tables) for proper
   formatting and use a linkchecker before you publish this page.

Conversion notes:

* Docs to Markdown version 1.0β35
* Thu Mar 21 2024 00:22:11 GMT-0700 (PDT)
* Source doc: PII Detection Library
* This is a partial selection. Check to make sure intra-doc links work.
----->



# PII Library 


## Problem Literature : 

According to NIST, these are what considered to be PII, sorted in the order by criticality :



* Full name
* Home address
* Email address
* Telephone number
* Social security number
* National Identification Number
* Passport number
* Driver's license number
* Credit card numbers
* Date of birth
* Log in details
* Gender
* Race
* Device IDs
* IP addresses

This is the standard list of what is identified as PII, however, the list is NOT exhaustive and subjected to change based on the internet laws of operation of a region. Also, in certain cases its a grey area. For example, a web cookie is not considered PII, however storing a customer id, makes it a PII.

Task at hand is to have a system where we feed data and the system reports back the PII information found in that data. There are already some systems available which does pretty much the same. Some options that were considered for are : 



* [https://pypi.org/project/piicatcher/](https://pypi.org/project/piicatcher/)
* [https://github.com/solvvy/redact-pii](https://github.com/solvvy/redact-pii)
* [https://pypi.org/project/piianalyzer/0.1.0/#description](https://pypi.org/project/piianalyzer/0.1.0/#description)

The base idea behind all of these modules is 1:1 regex matching for detection and redaction. Although, that's precisely the way such a problem should be solved, however, they are not full proof and does not guarantee 100 % detection. The goal here is to understand how to increase the detection efficiency for this technique. In order to understand the complexity of the detection, we will iterate through the list of PII data and discuss detection techniques, for each one of them in the increasing order of detection difficulty.



1. **IP addresses,Device IDs, Email address, Passport number, Social Security number, Gender, Race**: They are straight forward regex detection, as their formats are standard across all platforms, devices and region. 1 to 1 regex matching.
2. **Credit card numbers ,Telephone number, National Identification Number, Driver's license number, Dates, Addresses** : These are particularly difficult to detect with just one regex on its own, as incase of National IDs, every nation has their own format. 
3. **Full name (Entity Names), Login Details :** These are the hardest to detect. Most of the libraries stated above is either doing static lookups from standard name databases or using contextual NLP for detecting entity names.

Based on this, we can see there are clearly three categories in which PII can be split based on their detection mechanisms, baring the grey areas of dealing with authentication tokens and web cookies with customer id. 


## Proposed Solution : 

The solution is divided into two phases : 


## Phase 1 : (token extractor)



* Every stream is matched against list of regexes of each category. Difference is, instead of matching 1-1 regex for each category, it matches to a bunch of regexes for 1 category and returns a probability score for each of the identified tokens. The probability score for a token is determined  by number of regex where this token was returned vs total number of regexes. For example : 

**Sample Text **contains same phone number in the following formats (world wide standards) : 

E164 format: +918197800510

Original format: 81978 00510

National format: 081978 00510

International format: +91 81978 00510

Out-of-country format from US: 011 91 81978 00510

Out-of-country format from Switzerland: 00 91 81978 00510

**Regex 1** : ^(\(?\+?[0-9]*\)?)?[0-9_\- \(\)]*$/igm

Validates all to be correct.

**Regex 2** : (\d?[^\s\w]*(?:\(?\d{3}\)?\W*)?\d{3}\W*\d{4}) -> Regex from CommonRegex lib.

Validates 2 correctly.

 \
Determining the probability for the individual tokens **_(Number of regexes matched / Total  number of regexes)_** : 



* +918197800510  =  1
*  81978 00510 = 1
*  081978 00510  = 0.5
* +91 81978 00510 =  0.5
* 011 91 81978 00510 = 0.5
* 00 91 81978 0051 = 0.5

This way, the system will be able to determine any and all supported forms of phone number detections. This is a quick preview of the potential PII detected. Phase 2 is what will confirm the determination. The probability score is needed for phase 2 evaluation. Similar evaluations are to be in place for all the categories of the PII data. In case certain detection fails, for any category of data, it's all about finding the right regexes to be added to the list, which validates detects the information.


## Phase 2: (token assertion)

Phase 2 incorporates spacy rule based matching model mechanism [https://spacy.io/usage/rule-based-matching](https://spacy.io/usage/rule-based-matching).  Here, the tokens / analysis received from phase 1, becomes the labelled training data which will be used for training our model and our test data, is the original stream passed in the library. The NLP model will serve as an assertion mechanism for the data passed from phase 2, till the point where test data results in equivalent or better analysis than the regex matches. Also, this is the only mechanism via which named entities can be detected. 

Reasons to use the rule based matching model is that the model will be trained faster and better for labelled data. Given that the training data (tokens from phase 1) are already noise free, model shows a better response than training via unclean data. Furthermore, passing the unclean or actual text as test data, provides assertion instantly hence helping the model to be updated automatically without any intervention. 


### Typical Response example from PII Analyzer Library : 

[https://jsoneditoronline.org/?id=d0373eeba1294f5f88ce34b9ac2a8258](https://jsoneditoronline.org/?id=d0373eeba1294f5f88ce34b9ac2a8258)