Behind the Mask: Demographic bias in name detection for PII masking

Courtney Mansfield, Amandalynne Paullada, Kristen HowellACL 2022

TL;DR

This paper investigates demographic biases in off-the-shelf PII masking systems, specifically focusing on name detection. The authors found that while all tested systems exhibit performance disparities across racial/ethnic groups, names associated with Black and Asian/Pacific Islander individuals are disproportionately more likely to be missed, leading to potential privacy risks.

What Problem Does This Paper Address?

In an era of massive data collection, protecting personally identifiable information (PII) is crucial. PII masking, a common technique to redact sensitive data like names, often relies on machine learning. However, these systems can exhibit biases, meaning they might perform better for certain demographic groups than others. This paper addresses the critical question of whether PII masking systems, particularly for name detection, are fair across different racial, ethnic, and gender groups. Historically, privacy concepts have been centered around dominant groups, leaving marginalized communities more vulnerable to surveillance and harm, including algorithmic surveillance. This research aims to quantify these disparities to highlight potential privacy risks for specific populations.

Methods

The researchers evaluated three off-the-shelf PII masking systems: Amazon Web Services (AWS) Comprehend, Google Cloud Platform Data Loss Prevention (GCP DLP), and an open-source ROBERTa-based system called Presidio. They employed a sentence perturbation technique, a common method in NLP research, to test model performance. This involved using pre-defined templates and filling name slots with names sampled from various demographic datasets. The primary evaluation metric was the False Negative Rate (FNR), which measures how often a system fails to detect a name that is present. They also calculated the False Negative Equality Difference (FNED) to quantify disparities across demographic subgroups and used statistical tests (Friedman and Wilcoxon signed-rank tests) to assess the significance of these differences.

Datasets

The study utilized several datasets to obtain names and associated demographic information:

  • LAR Dataset: This dataset, from Tzioumis (2018), contains aggregate names with self-reported race/ethnicity from US Loan Application Registrars. It includes 4.2K given names from 2.6M observations.
  • NYC Data: Derived from the New York City Department of Health and Mental Hygiene's civil birth registration data (NYC Open Data, 2013), this dataset contains 1.8K given names from 1.2M observations (2011-2018). It includes the sex of the baby, allowing for intersectional analysis.
  • Congress Data: This dataset comprises the given and family names of US Congress members, providing self-reported demographic information.
  • Templates: A set of 32 templates were collected from real-world customer service messaging conversations to create test sentences.

The paper also discusses efforts to create frequency-matched subsets of the LAR dataset (LAR1 and LAR2) to control for name popularity and ensure fair comparisons across demographic groups.

Key Insights and Ideas

  • Name Ambiguity is a Double-Edged Sword: Names with multiple word senses (e.g., words that can also be verbs or adjectives) are harder to mask, leading to higher FNRs. However, this ambiguity disproportionately impacts certain groups, with Black names showing a significantly larger performance disparity when they have multiple senses.
  • Short Names and API/Black Groups: There appears to be a higher penalty for short names in the API and Black groups, suggesting that name length alone doesn't explain all disparities.
  • Context Matters for System Performance: The performance differences between groups are not always consistent across systems or datasets. For example, AWS struggles with API names in LAR2 but not in NYC, highlighting the importance of evaluating systems across diverse contexts.
  • Beyond Binary Gender: The study found that names not strongly associated with a particular gender had the highest error rates, underscoring the need to consider categories beyond the traditional male/female binary in bias evaluations.

Key Results

  • Significant Disparities: All tested PII masking systems demonstrated significant differences in error rates based on demographics.
  • Highest Error Rates for API and Black Individuals: Names associated with Asian/Pacific Islander (API) and Black individuals consistently showed the highest false negative rates across most systems and datasets.
  • Presidio Shows Less Bias: The open-source Presidio system, based on RoBERTa, exhibited fewer significant demographic discrepancies compared to the commercial AWS and GCP systems.
  • Specific System Performance:
    • AWS and GCP showed significant differences in FNR for API-associated names in the LAR2 dataset, with better performance on White-associated names.
  • Presidio performed poorly on Black-associated names in LAR1, with significant differences compared to Hispanic and White groups.
  • AWS performed best on Black-associated names in LAR1 but worse on Hispanic-associated names.
  • GCP performed worst on White-associated names in LAR1.
  • Gender Differences: While generally smaller than race/ethnicity differences, gender-based disparities were significant. Names not strongly associated with a gender had the highest error rates.
  • Intersectional Impact: In the NYC dataset, Black female-associated names had the highest FNR, with error rates averaging 13.7% higher than the full dataset.

Limitations and Future Work

  • Data Limitations: The study acknowledges limitations in the availability of demographic data, particularly for Indigenous and mixed-race individuals. The sampled names were also almost exclusively from US-born citizens.
  • Proxy for Race/Ethnicity: The paper uses names as a proxy for race/ethnicity and gender, acknowledging that this is not a perfect representation and that these categories are socially constructed.
  • Future Work: The authors suggest collaborating with the public to develop a database where individuals can voluntarily contribute their name and self-identified information for research. They also emphasize the responsibility of the NLP community to develop more equitable systems for PII masking and data protection.

Summary updated

BibTeX
@inproceedings{mansfield2022behind,
  author = {Courtney Mansfield and Amandalynne Paullada and Kristen Howell},
  title = {{Behind the Mask: Demographic bias in name detection for PII masking}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure