Abstract
Neural network models trained on text data have been found to encode undesirable linguistic or sensitive concepts in their representation. Removing such concepts is non-trivial because of a complex relationship between the concept, text input, and the learnt representation. Recent work has proposed post-hoc and adversarial methods to remove such unwanted concepts from a model's representation. Through an extensive theoretical and empirical analysis, we show that these methods can be counter-productive: they are unable to remove the concepts entirely, and in the worst case may end up destroying all task-relevant features. The reason is the methods' reliance on a probing classifier as a proxy for the concept. Even under the most favorable conditions for learning a probing classifier when a concept's relevant features in representation space alone can provide 100% accuracy, we prove that a probing classifier is likely to use non-concept features and thus post-hoc or adversarial methods will fail to remove the concept correctly. These theoretical implications are confirmed by experiments on models trained on synthetic, Multi-NLI, and Twitter datasets. For sensitive applications of concept removal such as fairness, we recommend caution against using these methods and propose a spuriousness metric to gauge the quality of the final classifier.
Introduction
Deep learning models often achieve high performance by latching onto spurious correlations rather than learning the underlying logic of a task. For instance, a Natural Language Inference (NLI) model might associate the presence of the word "not" with a "contradiction" label, regardless of the sentence's actual meaning. Beyond linguistic shortcuts, models can inadvertently encode sensitive attributes like race or gender, leading to biased and unfair outcomes.
To address this, researchers have developed concept removal methods. The prevailing logic suggests that if a small "probing classifier" can detect a concept within a model's internal layers, that information can be systematically scrubbed. This paper challenges this assumption, demonstrating that probing classifiers are fundamentally unreliable proxies for concept detection and removal, often leading to the destruction of task-relevant information or the reinforcement of existing biases.
Methods of Concept Removal
The study analyzes two primary classes of removal techniques:
- Post-hoc Removal (Iterative Null Space Projection - INLP): This method operates on a pre-trained model by projecting its internal representations into the "null space" of a probing classifier. The goal is to flatten the data such that the probe can no longer distinguish the concept.
- Adversarial Removal (AR): This involves a min-max game during training. The model attempts to minimize the loss on the primary task while simultaneously maximizing the loss of an adversarial probe trying to predict the sensitive concept.
Using a rigorous theoretical framework based on max-margin classifiers, the authors prove why these methods fail when the primary task and the target concept are correlated—a condition nearly universal in real-world datasets.
Experimental Setup
The researchers evaluated these methods across four diverse settings:
- MultiNLI: Investigating the spurious correlation between negation words (e.g., never, no) and the "contradiction" label.
- Twitter-PAN16: A tweet-mention detection task where gender is the sensitive concept to be removed.
- Twitter-AAE: A sentiment analysis task where race (African American English vs. Standard American English) is the sensitive concept.
- Synthetic-Text: A controlled environment where the task is number detection and the spurious concept is sentence length, allowing for precise interventions.
Key Insights and Theoretical Failures
The "Lazy Probe" Problem
The core failure of these methods lies in the nature of the probes themselves. Even if a concept is perfectly represented in the data, a probing classifier is "lazy"; it will utilize any correlated features to boost its accuracy. Because the probe relies on an "unclean" mix of features, the subsequent removal process based on that probe becomes similarly imprecise.
Feature Mixing and Task Collapse
When attempting to project away a concept using a flawed probe, the method does not just remove the concept; it "mixes" it with task-relevant features. In extreme cases, this leads to a total collapse of the model, where the main task accuracy drops to the level of a random guess because the signal was deleted along with the noise.
The Adversarial Backfire
Surprisingly, adversarial training can introduce bias into a previously "clean" model. If labels and concepts are correlated, the adversary forces the model to alter its representations in ways that may actually increase its reliance on spurious features to satisfy the primary task objective while evading the adversary.
Results and Metrics
Total Collapse and Incomplete Removal
In INLP experiments, as the correlation between the task and the concept increased, task accuracy frequently plummeted. Conversely, adversarial removal rarely achieved a complete removal of the concept, often plateauing at a level proportional to the correlation present in the training data.
The Spuriousness Score
Because probe accuracy is a misleading signal of success, the authors propose the Spuriousness Score. This metric measures the performance gap on "minority groups" (where the correlation between task and concept is broken) compared to a truly robust model. This provides a more accurate reflection of how much the model still relies on the "removed" concept.
Conclusion
The paper demonstrates that common heuristics, such as stopping removal when task accuracy begins to drop, are dangerous; at these points, the model is often more reliant on spurious concepts than it was initially.
For high-stakes fairness applications, the authors recommend extreme caution when using probe-based removal. Instead of post-hoc scrubbing, they suggest that counterfactual data augmentation—altering the input text to break correlations at the source—remains a more promising, though more labor-intensive, path toward building robust and fair neural representations.
Summary updated