Abstract
Current methods for evaluating model invariance to nuisance factors—variables like product IDs that should not influence sentiment classification—are often inadequate. This paper introduces two new robustness metrics: Generalization under Target Bias (GTB) and Generalization onto Unknown (GU), alongside a mutual information-based filtering technique to generate intentionally biased datasets. Our findings suggest that traditional metrics like identifiability are flawed proxies for true invariance. We demonstrate that while models can be robust to unseen nuisance factors, they are highly sensitive to target bias, leaking nuisance information into their decision-making processes.
Introduction
In machine learning, achieving invariance to nuisance variables—features that are correlated with the label but should not influence the prediction—is a critical goal. In sentiment analysis, a model should ideally classify reviews based on linguistic sentiment rather than specific Product IDs. However, existing evaluation frameworks lack the necessary metrics and datasets to rigorously test this invariance. This paper addresses these gaps by introducing novel metrics and a systematic data-filtering approach to create "stress-test" environments that reveal where a model's logic breaks down.
Methods
We introduce two primary metrics for assessing invariance:
- Generalization under Target Bias (GTB): This metric measures performance on a balanced test set after training on a dataset where the label $Y$ and nuisance factor $S$ are artificially correlated. A non-invariant model will exploit this correlation, leading to poor generalization on unbiased data.
- Generalization onto Unknown (GU): This assesses the model's ability to generalize to values of the nuisance factor not encountered during training, such as new products.
To generate the required biased datasets, we employ an iterative filtering method. By estimating the mutual information (MI) between features $X$ and the nuisance factor $S$, we selectively remove samples to increase the dependency between $X$ and $S$, thereby forcing the model to contend with higher levels of entanglement between signal and noise.
Experimental Setup
We evaluate our approach using four standard sentiment analysis datasets: Yelp and three subsets of Amazon reviews (Books, Electronics, and Music). For each dataset, we identify the 50 most reviewed products and treat the Product ID as the nuisance factor. We then apply our filtering method to create three levels of biased subsets ($D_0, D_1, D_2$) to evaluate model sensitivity to increasing levels of target bias.
Results and Discussion
Our experiments reveal that traditional identifiability metrics—which measure how easily a nuisance factor can be decoded from hidden representations—are poor indicators of invariance. Even when identifiability is low, models may still rely on nuisance factors for prediction.
As shown in Figure 4, performance on unbiased test sets (GTB) consistently declines as the mutual information between text and Product ID increases. Conversely, models remain relatively robust in the GU setting, maintaining high accuracy on unseen products. This suggests that while models can generalize to new instances of a nuisance factor, they struggle to ignore that factor when it provides a "shortcut" during training.
The table in Figure 6 provides detailed performance values, confirming the trend of decreasing robustness as bias is intensified through our filtering process. The "leakage" of nuisance information into the decision process becomes more pronounced as the dataset becomes more biased, highlighting the limitations of current neural architectures in learning truly invariant representations.
Conclusion
This study underscores the importance of dataset construction in evaluating model invariance. By intentionally introducing bias, we can more accurately measure a model's reliance on nuisance factors. Our results indicate that current NLP models are highly susceptible to target bias, even when they appear robust to unseen nuisance factors. Future work will explore more efficient filtering techniques and the development of metrics based on conditional probability modifications to further refine our understanding of invariant representations.
Summary updated