Removing biases or spurious correlations from neural networks is a major challenge in AI safety and fairness. Deep learning models often "cheat" by using spurious correlations (e.g., using the presence of the word "not" as a shortcut to predict "contradiction" in text) or encode sensitive attributes like gender and race. Researchers have developed methods to "scrub" these concepts from a model's internal representations.
Most of these methods rely on a probing classifier: a small auxiliary model trained to detect the unwanted concept within the main model's hidden layers. The logic is straightforward: "If a probe can't find the concept, it must be gone." However, this paper proves that popular "concept removal" techniques—like adversarial training and null-space projection—often fail because they rely on probing classifiers that are fundamentally unreliable. Instead of just finding the "bad" concept, these probes get distracted by correlated "good" features, leading the removal process to accidentally destroy task-relevant information or even increase the model's reliance on the very bias it was meant to erase.
The Flaw in Probing Classifiers
The authors analyze two primary classes of removal methods:
- Post-hoc Removal (INLP): Specifically Iterative Null-space Projection. This method trains a linear probe to find the concept and then projects the model's representations into the "null space" of that probe to effectively erase the concept's dimensions.
- Adversarial Removal: This involves training the main model and a concept-detecting "adversary" simultaneously. The main model is penalized if the adversary can successfully guess the sensitive concept.
The core issue is that probes are "non-causal." A probe doesn't necessarily find the features caused by the concept; it finds any features predictive of the concept label. If the main task is correlated with the concept, the probe will hijack main-task features.
Because probes use a mix of "good" and "bad" features, removal methods like INLP end up "mixing" these dimensions. When you try to delete the "bad" dimension, you accidentally delete the "good" one too. Perhaps most surprisingly, applying these removal methods to a model that doesn't even use the concept can actually introduce bias or destroy its accuracy, simply because the training data contains correlations.
In adversarial training, the model might not actually remove the concept. Instead, it might just find a representation that makes the probe slightly less accurate while still keeping the concept features available for the main task to use.
Empirical Evidence of Failure
The study uses a mix of real-world and controlled environments to test these theories:
- MultiNLI: A natural language inference dataset where "negation" words are often spuriously correlated with the "contradiction" label.
- Twitter-PAN16 & Twitter-AAE: Datasets used to detect user mentions or sentiment, where author gender or race act as sensitive concepts to be removed.
- Synthetic-Text: A custom dataset where sentence length is artificially correlated with a specific task, allowing the authors to perform "interventions" to see if the model's internal logic actually changed.
The results reveal a stark reality for concept removal techniques:
- Total Collapse: In experiments on MultiNLI and Twitter datasets, post-hoc removal (INLP) eventually destroyed all task-relevant information. It reduced the main model to "random guess" accuracy before it successfully removed the concept.
- Early Stopping Backfires: Researchers often stop removal early to save accuracy. The authors found that this often results in a model that is more reliant on the spurious concept than the original un-scrubbed model.
- Adversarial Failure: Adversarial removal failed to significantly reduce the Spuriousness Score (a metric introduced to measure reliance on a concept) across most datasets. The "scrubbed" models still performed significantly worse on minority groups where the spurious correlation didn't hold.
Conclusion and Future Directions
While the theoretical proofs assume a frozen encoder, the empirical results suggest the findings hold even when the encoder is fine-tuned. The evidence strongly suggests that using probing classifiers as a proxy for concept removal is fundamentally flawed.
Instead of trying to "fix" representations via probes, the community should look toward alternative approaches:
- Counterfactual Data Augmentation: Changing the input text itself to break correlations during training.
- Algorithmic Fairness: Focusing on the fairness of the final predictions rather than trying to police the internal hidden layers, which are notoriously difficult to interpret and manipulate reliably.
Summary updated