TL;DR
Projection-based methods aim to remove specific information (like gender) from data representations. This paper reveals that instead of truly removing information, these methods inject strong statistical dependencies between data points, making the transformed data highly structured and potentially reconstructible. This has significant implications for how we interpret and use such transformed datasets, especially for privacy and downstream analysis.
What Problem Does This Paper Address?
In natural language processing (NLP) and machine learning, there's a growing need to control what information is encoded in data representations. This is crucial for applications like domain adaptation (making models generalize across different data sources) and, more recently, for removing sensitive attributes like gender or race from representations to promote fairness or privacy. While adversarial methods have been used for this, projection-based methods, like Iterative Nullspace Projection (INLP), offer a more stable and efficient alternative. These methods aim to transform representations so that a specific concept (e.g., gender) becomes unrecoverable. However, the paper questions the extent to which this information is truly "removed" and what happens to the dataset's underlying structure after such a transformation. Specifically, it asks: does applying these methods lead to a dataset where the concept is statistically independent of the representations, or does it introduce other, perhaps unintended, dependencies?
Methods
The paper focuses on projection-based concept removal methods, particularly Iterative Nullspace Projection (INLP) and a related method called Mean Projection (MP). The core idea behind these methods is to project the data representations into a subspace that is orthogonal to the direction identified by a classifier trained to predict the concept to be removed.
- Theoretical Analysis: The authors provide a theoretical analysis, primarily focusing on MP (which is equivalent to INLP with a nearest centroid classifier). They prove that after applying MP, the class centroids in the transformed space become identical. This leads to a situation where instances are either misclassified or lie exactly on the decision boundary of a leave-one-out classifier trained on the remaining data. This demonstrates a fundamental departure from independent and identically distributed (i.i.d.) data.
- Empirical Investigation: To complement the theory, the authors conduct experiments using INLP. They analyze prediction accuracy, the distribution of predicted probabilities, neighborhood structures, and the reconstructibility of original labels.
Datasets
The experiments utilize both synthetic and real-world datasets:
- Synthetic Data: Generated from a standard isotropic multivariate Gaussian distribution with balanced labels. This was used to investigate INLP's behavior when the original data has no inherent signal representing the concept.
- Real-World Data: Six domains from the sentiment classification corpus collected by Blitzer et al. (2007). These datasets associate documents with positive or negative polarity labels.
- Representations: The paper uses two types of representations:
- TF-IDF weighted bag-of-words (BoW) with a feature dimension of 210.
- BERT model outputs (specifically, the [CLS] token embedding) with a dimension of 768.
Key Insights and Ideas
- Injected Dependencies, Not Removal: Projection-based methods don't truly remove information about the concept; instead, they inject strong statistical dependencies between instances in the transformed dataset.
- Adversarial Arrangement: The transformed representation space becomes highly structured, with instances tending to be located near instances of the opposite label. This is counter-intuitive if the goal is independence.
- Reconstructibility: Even if the original data was random and unrelated to the target concept, the injected structure can allow the original grouping (labels) to be reconstructed using anti-clustering methods.
- Non-i.i.d. Nature: The core finding is that the assumption of i.i.d. instances is violated after projection, which has significant implications for statistical analyses that rely on this assumption.
Key Results
- Prediction Accuracy: Cross-validated accuracies for predicting the removed concept drop significantly below chance level after just a few INLP iterations. For BoW representations, this drop is almost immediate, while for BERT, it takes more iterations, especially with larger datasets.
- Predicted Probabilities: The distribution of predicted probabilities for models trained on projected data differs markedly from models trained on truly independent data.
- Neighborhood Structure: Experiments show that for a significant proportion of instances, their nearest neighbor in the transformed space belongs to the opposite class. This tendency is more pronounced when the dimensionality of the data is much larger than the number of instances (d >> n).
- Recovering Labels: Anti-clustering algorithms applied to the projected datasets can often perfectly reconstruct the original label groupings, demonstrating that the information, while obscured, is still present in the data's structure.
Limitations and Future Work
- Theoretical Scope: The theoretical analysis was limited to Mean Projection (MP) and nearest centroid classifiers. A more general formal justification for the adversarial arrangement is desired.
- Dimensionality vs. Dataset Size: The role of dimensionality (d) in relation to dataset size (n) is acknowledged as important but not fully integrated into the theoretical analysis.
- Broader Method Analysis: The empirical investigation focused on INLP. Future work could explore a wider range of concept removal methods, including more recent projection-based and adversarial approaches.
- Privacy Implications: While not claiming to be privacy-preserving, the paper highlights that projection methods should not be viewed as such, as information can be reconstructed. The authors note that some methods (like Xu et al., 2017) explicitly considered privacy, but this was not investigated here.
- Statistical Analysis Impact: The violation of i.i.d. assumptions means that statistical analyses requiring them (e.g., text-based causal inference) applied to projected data could be invalid.
Summary updated