Abstract
For the task of image classification, researchers work arduously to develop the next state-of-the-art (SOTA) model, each benchmarking their own performance against that of their predecessors and of their peers. Unfortunately, the metric used most frequently to describe a model's performance, average categorization accuracy, is often used in isolation. As the number of classes increases, such as in fine-grained visual categorization (FGVC), the amount of information conveyed by average accuracy alone dwindles. While its most glaring weakness is its failure to describe the model's performance on a class-by-class basis, average accuracy also fails to describe how performance may vary from one trained model of the same architecture, on the same dataset, to another (both averaged across all categories and at the per-class level). We first demonstrate the magnitude of these variations across models and across class distributions based on attributes of the data, comparing results on different visual domains and different per-class image distributions, including long-tailed distributions and few-shot subsets. We then analyze the impact various FGVC methods have on overall and per-class variance. From this analysis, we both highlight the importance of reporting and comparing methods based on information beyond overall accuracy, as well as point out techniques that mitigate variance in FGVC results.
Introduction
Fine-Grained Visual Categorization (FGVC) represents a significant challenge in computer vision, requiring models to distinguish between highly similar sub-categories, such as specific bird species, car models, or aircraft variants. Because the inter-class differences are subtle and intra-class variations can be high, these models are exceptionally sensitive to training conditions.
Currently, the primary metric for success is average categorization accuracy. However, this single number often masks significant inconsistencies. A model might achieve high average accuracy while failing completely on specific classes, or its performance might fluctuate wildly between identical training runs due to random initialization and data shuffling.
Quantifying Variance in FGVC
To address the limitations of average accuracy, this research introduces two critical metrics for evaluating model reliability:
- Overall Deviation: This measures the spread of accuracies across all classes in a single experiment. It reveals whether a model is "fair" across the entire dataset or if it possesses significant blind spots.
- Per-class Deviation: This measures how much the accuracy for a specific class fluctuates across multiple independent training runs. It indicates the stability and reproducibility of the training process.
The study conducted an extensive "stress test" across five standard benchmarks: CUB (Birds), Stanford Cars, Stanford Dogs, FGVC Aircraft, and NABirds. By running 10 trials for every configuration—varying optimizers, architectures, and loss functions—the researchers could observe the inherent volatility of these models.
The Deception of Averages
The core finding of the study is that high accuracy does not equate to consistency. In the NABirds dataset, which contains 555 classes, the researchers observed a "35% swing" where one ResNet-50 run was significantly more accurate on a specific class than an identical run of the same model.
As shown in the figure above, the "Overall mean" (dotted line) provides a very limited view of the model's true behavior. The "Per-class min/max" (red lines) demonstrate that for many classes, performance is highly unpredictable. Some runs reach near-perfect accuracy while others fail significantly on the same class, proving that average accuracy is a deceptive metric in isolation.
Key Insights: The "Lucky Draw" Hazard
The high variance observed in State-of-the-Art (SOTA) models suggests a "Lucky Draw" hazard. If researchers only report their best single run, they may be showcasing a statistical outlier rather than a robust improvement. The study found that the "worst" run of a top-tier model often performed worse than the "best" run of a lower-tier model, making single-run comparisons statistically unsound.
Furthermore, the research highlighted that in long-tailed datasets, reliability is driven more by the minimum number of images per class than the total dataset size. However, certain techniques, such as Maximum Entropy Regularization, were found to act as stabilizers, improving both accuracy and consistency compared to standard cross-entropy loss.
Conclusion and Recommendations
Average accuracy is an insufficient metric for FGVC. To ensure fair comparison and real-world reliability, the community must move toward standardized reporting that includes "Mean ± SD" over multiple trials. Future work should focus on developing variance-aware loss functions and ensembling strategies that specifically target high-variance classes to create more robust and predictable fine-grained classifiers.
Summary updated