Predicting Model Performance on Unseen Data: The Power of Feature Separability
In the world of machine learning, we often train models on one set of data (the training set) and hope they perform well on new, unseen data (the test set). However, when the test data comes from a different distribution than the training data—a phenomenon known as Out-of-Distribution (OOD) shift—performance often takes a hit.
The challenge is: how can we predict a model's accuracy on OOD data without having access to the ground-truth labels?
What Problem Does This Paper Address?
Most existing methods for predicting OOD performance rely on measuring the "distance" between the training and test distributions. The logic is intuitive: the further the test data is from the training data, the worse the model should perform.
However, the authors of this paper demonstrate that this isn't always true. As shown in Figure 1, two different datasets can have the same "distance" from the training set (measured via Fréchet or MMD distance) but result in wildly different accuracies.
This suggests that the "distribution gap" isn't the whole story. Instead, this paper proposes looking at feature separability—how well the model's internal representations distinguish between different classes.
The Method: Dispersion Score
The authors introduce the Dispersion Score, a simple, training-free metric to estimate OOD accuracy. The process is straightforward:
- Extract Features: Pass the unlabeled OOD data through the pre-trained model to get feature representations.
- Assign Pseudo-labels: Use the model's own predictions as "pseudo-labels" to group the features into clusters.
- Calculate Dispersion: Measure the average distance between each cluster's center and the overall center of all data points.
Mathematically, a higher Dispersion Score means the model is still "separating" the different classes well in its internal feature space, even if the input data looks different from what it saw during training.
Key Insights and Ideas
- Inter-class vs. Intra-class: In representation learning, we usually want "high inter-class dispersion" (classes are far apart) and "high intra-class compactness" (samples within a class are close together). The authors found that only inter-class dispersion strongly correlates with OOD accuracy. Surprisingly, how "tight" the clusters are (compactness) has almost no impact on the final error.
- Visual Evidence: Using t-SNE visualizations (as seen in Figure 2), the authors show that as data becomes more "corrupted" (higher severity), the clusters in the feature space become less distinct and begin to overlap, directly corresponding to a drop in accuracy.
- Efficiency: Unlike some state-of-the-art methods (like ProjNorm) that require fine-tuning the model on the test data, the Dispersion Score is "training-free." It only requires a single forward pass through the network.
Key Results
The researchers tested their method on standard datasets like CIFAR-10, CIFAR-100, and TinyImageNet across various corruptions.
- Superior Accuracy: On TinyImageNet, the Dispersion Score improved the $R^2$ (a measure of how well the score predicts the actual error) from 0.847 to 0.970—a 14.5% improvement over previous methods.
- Efficiency: When using a large model like WRN-50-2, the previous best method (ProjNorm) took 575 seconds to estimate performance. The Dispersion Score took only 11 seconds.
- Sample Efficiency: As shown in Figure 4, the Dispersion Score remains highly accurate even with very small sample sizes (as few as 50 examples), whereas other methods' performance drops significantly.
Limitations and Future Work
While the Dispersion Score is highly effective for "natural" distribution shifts (like blur, noise, or weather changes), it struggles with adversarial attacks. In the case of adversarial perturbations, the model actually separates the features more (increasing the Dispersion Score) even as the accuracy plummets because the model is being "tricked" into confidently misclassifying the data. Future work could look into adapting the concept of feature separability to better handle these malicious cases.
Summary updated