On the Importance of Feature Separability in Predicting Out-Of-Distribution Error

Renchunzi Xie, Hongxin Wei, Lei Feng, Yuzhou Cao, Bo An

Predicting Model Performance on Unseen Data: The Power of Feature Separability

In the world of machine learning, we often train models on one set of data (the training set) and hope they perform well on new, unseen data (the test set). However, when the test data comes from a different distribution than the training data—a phenomenon known as Out-of-Distribution (OOD) shift—performance often takes a hit.

The challenge is: how can we predict a model's accuracy on OOD data without having access to the ground-truth labels?

What Problem Does This Paper Address?

Most existing methods for predicting OOD performance rely on measuring the "distance" between the training and test distributions. The logic is intuitive: the further the test data is from the training data, the worse the model should perform.

However, the authors of this paper demonstrate that this isn't always true. As shown in Figure 1, two different datasets can have the same "distance" from the training set (measured via Fréchet or MMD distance) but result in wildly different accuracies.

Figure 1: Distribution Gap Vs. OOD accuracy on CIFAR10-C via (a) Fréchet distance [Dowson and Landau, 1982] and (b) MMD [Gretton et al., 2006]. All colors indicate 14 types of corruption. The test accuracy varies significantly on shifted datasets with the same distribution distances.

This suggests that the "distribution gap" isn't the whole story. Instead, this paper proposes looking at feature separability—how well the model's internal representations distinguish between different classes.

The Method: Dispersion Score

The authors introduce the Dispersion Score, a simple, training-free metric to estimate OOD accuracy. The process is straightforward:

  1. Extract Features: Pass the unlabeled OOD data through the pre-trained model to get feature representations.
  2. Assign Pseudo-labels: Use the model's own predictions as "pseudo-labels" to group the features into clusters.
  3. Calculate Dispersion: Measure the average distance between each cluster's center and the overall center of all data points.

Mathematically, a higher Dispersion Score means the model is still "separating" the different classes well in its internal feature space, even if the input data looks different from what it saw during training.

Key Insights and Ideas

  • Inter-class vs. Intra-class: In representation learning, we usually want "high inter-class dispersion" (classes are far apart) and "high intra-class compactness" (samples within a class are close together). The authors found that only inter-class dispersion strongly correlates with OOD accuracy. Surprisingly, how "tight" the clusters are (compactness) has almost no impact on the final error.
  • Visual Evidence: Using t-SNE visualizations (as seen in Figure 2), the authors show that as data becomes more "corrupted" (higher severity), the clusters in the feature space become less distinct and begin to overlap, directly corresponding to a drop in accuracy.
Figure 2: t-SNE visualization of feature representation on training and OOD test datasets (CIFAR- 10C) with contrast corruption under different severity levels. The first left figure shows feature representation of the training dataset, while the rest of three figures illustrate those of the OOD datasets. From this figure, we observe that different clusters tends to be more separated as the corruption severity level gets smaller.
  • Efficiency: Unlike some state-of-the-art methods (like ProjNorm) that require fine-tuning the model on the test data, the Dispersion Score is "training-free." It only requires a single forward pass through the network.

Key Results

The researchers tested their method on standard datasets like CIFAR-10, CIFAR-100, and TinyImageNet across various corruptions.

  • Superior Accuracy: On TinyImageNet, the Dispersion Score improved the $R^2$ (a measure of how well the score predicts the actual error) from 0.847 to 0.970—a 14.5% improvement over previous methods.
  • Efficiency: When using a large model like WRN-50-2, the previous best method (ProjNorm) took 575 seconds to estimate performance. The Dispersion Score took only 11 seconds.
  • Sample Efficiency: As shown in Figure 4, the Dispersion Score remains highly accurate even with very small sample sizes (as few as 50 examples), whereas other methods' performance drops significantly.
Figure 4: Prediction performance vs. sample size

Limitations and Future Work

While the Dispersion Score is highly effective for "natural" distribution shifts (like blur, noise, or weather changes), it struggles with adversarial attacks. In the case of adversarial perturbations, the model actually separates the features more (increasing the Dispersion Score) even as the accuracy plummets because the model is being "tricked" into confidently misclassifying the data. Future work could look into adapting the concept of feature separability to better handle these malicious cases.

Summary updated

Abstract

Estimating the generalization performance is practically challenging on out-of-distribution (OOD) data without ground-truth labels. While previous methods emphasize the connection between distribution difference and OOD accuracy, we show that a large domain gap not necessarily leads to a low test accuracy. In this paper, we investigate this problem from the perspective of feature separability empirically and theoretically. Specifically, we propose a dataset-level score based upon feature dispersion to estimate the test accuracy under distribution shift. Our method is inspired by desirable properties of features in representation learning: high inter-class dispersion and high intra-class compactness. Our analysis shows that inter-class dispersion is strongly correlated with the model accuracy, while intra-class compactness does not reflect the generalization performance on OOD data. Extensive experiments demonstrate the superiority of our method in both prediction performance and computational efficiency.
BibTeX
@inproceedings{xie2023importance,
  author = {Renchunzi Xie and Hongxin Wei and Lei Feng and Yuzhou Cao and Bo An},
  title = {{On the Importance of Feature Separability in Predicting Out-Of-Distribution Error}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=A86JTXllHa},
}
Figure