Fair Comparison: Quantifying Variance in Results for Fine-grained Visual Categorization

Matthew Gwilliam, Adam Teuscher, Connor Anderson, Ryan Farrell

Abstract

For the task of image classification, researchers work arduously to develop the next state-of-the-art (SOTA) model, each benchmarking their own performance against that of their predecessors and of their peers. Unfortunately, the metric used most frequently to describe a model’s performance, average categorization accuracy, is often used in isolation. As the number of classes increases, such as in fine-grained visual categorization (FGVC), the amount of information conveyed by average accuracy alone dwindles. While its most glaring weakness is its failure to describe the model’s performance on a class-by-class basis, average accuracy also fails to describe how performance may vary from one trained model of the same architecture, on the same dataset, to another. We first demonstrate the magnitude of these variations across models and across class distributions based on attributes of the data, comparing results on different visual domains and different per-class image distributions, including long-tailed distributions and few-shot subsets. We then analyze the impact various FGVC methods have on overall and per-class variance. From this analysis, we both highlight the importance of reporting and comparing methods based on information beyond overall accuracy, as well as point out techniques that mitigate variance in FGVC results.

Introduction

In Fine-Grained Visual Categorization (FGVC), models are tasked with distinguishing between highly similar sub-categories, such as specific bird species, car makes, or aircraft models. Currently, the primary metric for success is "State-of-the-Art" (SOTA) average accuracy. However, this single number acts as a "black box" that obscures critical instabilities in model performance.

Fine-grained visual categorization requires distinguishing between highly similar classes, such as specific bird species.

This paper identifies two primary types of instability. First, per-class instability occurs when a model achieves high overall accuracy but fails significantly on specific classes. Second, training instability arises because random weight initialization and data shuffling can lead to "identical" models (same architecture and data) exhibiting non-trivial performance differences. Without quantifying this variance, researchers risk reporting "lucky" high-scoring runs as breakthroughs while dismissing potentially superior methods that suffered from "unlucky" initialization.

Methodology

Rather than proposing a new algorithm, this work introduces a rigorous evaluation protocol. The authors conducted extensive experiments by training multiple instances (typically 10 trials) of various models to measure:

  • Overall Deviation: The standard deviation of all per-class accuracy scores within a single experiment, indicating how performance is distributed across the dataset.
  • Per-class Deviation: The standard deviation of accuracy for a specific class across multiple training runs, measuring the model's consistency on that category.

The study evaluated these metrics across standard FGVC benchmarks including CUB-200-2011 (Birds), NABirds, Stanford Cars, Stanford Dogs, and FGVC Aircraft. Additionally, synthetic long-tailed and few-shot subsets of the Birds dataset were created to observe the impact of data scarcity on variance.

Results and Analysis

The "Lucky Draw" and Consistency

Analysis reveals that several high-performing SOTA methods, such as DCL and WSDAN, exhibit large variations between trials. This suggests that some reported literature results may stem from exceptionally lucky training runs. Furthermore, the study found that higher accuracy does not inherently guarantee lower variance. For instance, Label Smoothing, a common accuracy-boosting technique, actually increased overall deviation compared to the baseline, making performance less predictable across classes.

Accuracy distribution across classes showing per-class mean, deviation, and min/max ranges over multiple trials. The shaded region highlights the instability hidden by a single average accuracy metric.

Model and Optimizer Stability

While newer models often boast higher peak accuracies, older architectures like the Bilinear CNN (BCNN) proved remarkably consistent across trials. In applications where reliability is paramount, such consistency may be more valuable than peak performance.

Optimizer choice also significantly impacts stability. While Adam often reaches higher peak accuracies, Adadelta was found to produce models with lower standard deviations. Conversely, Max-Entropy Regularization emerged as a superior technique, improving both average accuracy and reducing variance compared to standard cross-entropy.

Data Distribution Impacts

In the NABirds dataset, the authors observed classes where one ResNet-50 instance was 35% more accurate than an identical instance due solely to random initialization. In long-tailed distributions, variance follows an inverse relationship with accuracy; as image counts per class drop, the "disagreement" between identical models increases sharply.

Conclusion

This research highlights the urgent need for the FGVC community to move beyond reporting single-run average accuracy. While training multiple trials is computationally intensive, reporting results from even 3 to 5 trials provides a much fairer basis for comparison. Future work should focus on developing loss functions specifically designed to minimize variance and investigating how ensemble methods can be optimized to mitigate the per-class instabilities identified here. Ensuring a "fair comparison" requires evaluating both the height of performance and the stability of the platform.

Summary updated

Abstract

For the task of image classification, researchers work arduously to develop the next state-of-the-art (SOTA) model, each bench-marking their own performance against that of their predecessors and of their peers. Unfortunately, the metric used most frequently to describe a model’s performance, average categorization accuracy, is often used in isolation. As the number of classes increases, such as in fine-grained visual categorization (FGVC), the amount of information conveyed by average accuracy alone dwindles. While its most glaring weakness is its failure to describe the model’s performance on a class-by-class basis, average accuracy also fails to describe how performance may vary from one trained model of the same architecture, on the same dataset, to another (both averaged across all categories and at the per-class level). We first demonstrate the magnitude of these variations across models and across class distributions based on attributes of the data, comparing results on different visual domains and different per-class image distributions, including long-tailed distributions and few-shot subsets. We then analyze the impact various FGVC methods have on overall and per-class variance. From this analysis, we both highlight the importance of reporting and comparing methods based on information beyond overall accuracy, as well as point out techniques that mitigate variance in FGVC results.
BibTeX
@inproceedings{gwilliam2021fair,
  author = {Matthew Gwilliam and Adam Teuscher and Connor Anderson and Ryan Farrell},
  title = {{Fair Comparison: Quantifying Variance in Results for Fine-grained Visual Categorization}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=0CINmkOOKq},
}
Figure