Log-linear Guardedness and its Implications

Shauli Ravfogel, Yoav Goldberg, Ryan CotterellACL 2023

TL;DR

This paper investigates "log-linear guardedness," a technique for removing protected attributes (like gender) from text representations. They show that while this method can prevent a binary downstream classifier from recovering the protected attribute, it fails to do so for multiclass classifiers, highlighting a significant limitation of current concept erasure techniques.

What Problem Does This Paper Address?

Neural networks learn representations of text that can encode human-interpretable concepts, including sensitive attributes like gender. "Concept erasure" aims to remove these attributes from the representations to mitigate bias. A common approach is "linear concept erasure," which aims to make the protected attribute unpredictable from the representation using linear models. The assumption is that if a protected attribute is removed from the representation, downstream classifiers trained on these modified representations will also be oblivious to that attribute. However, the implications of this "guardedness" for downstream classifiers, especially those that are not strictly binary, are not fully understood. This paper aims to theoretically analyze and empirically validate the effectiveness of log-linear guardedness in preventing information leakage to downstream classifiers.

Methods

The authors introduce a formal definition of "log-linear guardedness" based on V-information, a generalization of mutual information that considers a specific family of predictive models (in this case, log-linear models). They then analyze the implications of this guardedness for downstream binary and multiclass classifiers.

  • V-information: This is a measure of information leakage, but restricted to a specific family of models (V). Log-linear guardedness means V-information is small with respect to a log-linear model family.
  • Binary Downstream Classifier Analysis: They prove that for binary classification tasks, if representations are log-linearly guarded with respect to a binary protected attribute (using a specific family of discretized log-linear models), then a downstream binary log-linear classifier cannot leak information about the protected attribute.
  • Multiclass Downstream Classifier Analysis: In contrast, they demonstrate that for multiclass classification, log-linear guardedness can be broken. They show that even if representations are guarded, a multiclass softmax classifier can still recover substantial information about the protected attribute. This is illustrated with theoretical examples and figures showing how linear separators can be constructed to recover the attribute.
  • Empirical Evaluation: They use the RLACE method (Ravfogel et al., 2022a) to approximate log-linear guardedness on a biography dataset. They then measure different forms of V-information to assess leakage to downstream profession prediction tasks (both binary and multiclass).

Datasets

The paper uses the Bias in Bios dataset (De-Arteaga et al., 2019) for their empirical evaluation. This dataset consists of short biographies annotated with gender and profession. They use the train-dev-test split provided by Ravfogel et al. (2020).

Key Insights and Ideas

  • The "Log-Linear" Assumption is Crucial: The effectiveness of concept erasure is highly dependent on the type of downstream classifier. What works for a simple linear classifier might fail for a more complex one.
  • Binary vs. Multiclass Divide: The paper reveals a critical distinction: log-linear guardedness provides strong protection against binary downstream classifiers but offers limited guarantees against multiclass classifiers. This is a key takeaway for practitioners.
  • Softmax is the Culprit: The non-linearity introduced by softmax in multiclass classifiers is identified as the mechanism that allows information leakage, even when the underlying representations are "guarded" against linear prediction.
  • Intrinsic vs. Extrinsic Bias: The work touches upon the relationship between intrinsic bias (in representations) and extrinsic bias (in model predictions), suggesting that achieving intrinsic guardedness doesn't automatically guarantee extrinsic fairness.

Key Results

  • Binary Case: Theoretically proven that log-linear guardedness prevents information leakage to binary downstream log-linear classifiers under specific assumptions. Empirically, they show that downstream profession classifiers achieve accuracy less than 2% above majority accuracy on the protected attribute.
  • Multiclass Case: Empirically demonstrated that multiclass softmax classifiers can recover almost all information about the protected attribute (gender) when using guarded representations. This is shown to be possible with as few as 4-8 softmax entries (hidden neurons).
  • RLACE Effectiveness: The RLACE method, used to approximate log-linear guardedness, shows good but imperfect protection against binary adversaries. For multiclass adversaries, the leakage is significantly higher.

Limitations and Future Work

  • Specific Model Family: The theoretical analysis of binary guardedness relies on a specific family of "d-discretized log-linear models." The authors acknowledge that their proof for the binary case has a restriction and they don't have a proof for the opposite implication.
  • V-information Focus: The paper's theoretical framework is based on V-information. Future work could explore alternative extrinsic bias measures and notions of guardedness.
  • Linearity Assumption: The focus on linear models and linear interventions limits the generality of the conclusions. Extending the analysis to other predictive families is a promising direction.
  • Real-World Deployment: The authors caution that while their work provides theoretical insights, real-world deployment of concept erasure methods requires careful consideration of the specific adversary, data, and fairness metrics. They also highlight that gender is a non-binary concept, and the task of removing it has ethical implications.

Summary updated

BibTeX
@inproceedings{ravfogel2023log,
  author = {Shauli Ravfogel and Yoav Goldberg and Ryan Cotterell},
  title = {{Log-linear Guardedness and its Implications}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure