Introduction
In deep learning, ensembling—averaging the predictions of multiple models—is a standard technique for improving performance and uncertainty estimation. However, ensembling internal representations (embeddings) is non-trivial. Feature vectors from independently trained models are typically unaligned; a specific semantic feature might occupy different indices in different models, making direct averaging meaningless.
Researchers from Samsung AI and EPFL introduce MeTTA (Mean Embeddings with Test-Time Augmentation) to address this. Instead of ensembling across different models, MeTTA ensembles different "views" of the same input through a single model, ensuring feature alignment while improving robustness and accuracy.
The MeTTA Method
The core idea of MeTTA is to leverage the same model to process multiple augmented versions of a single input image. By averaging these representations, the resulting "mean embedding" becomes more invariant to the specific transformations applied during inference.
As shown in Figure 1, the process involves:
- Generating $S$ stochastic augmentations (e.g., random crops and flips) of an image $x$.
- Passing these $S$ versions through a pre-trained neural network.
- Computing the arithmetic mean of the resulting feature vectors.
Because all augmentations pass through the same network weights, the feature spaces are perfectly aligned, making the mean embedding a mathematically sound representation.
Addressing Representation Instability
Standard embeddings often suffer from a lack of invariance. Even if a model is trained with data augmentation, its final representations can "jitter" significantly with small changes in the input, such as a slight shift in a crop. This instability can lead to inconsistent classifications (e.g., misidentifying a "wombat" as a "koala" due to a specific crop).
MeTTA mitigates this by finding a more stable region in the feature space. The authors demonstrate that the mean embedding resides in a "favorable direction" relative to standard central-crop embeddings.
As illustrated in Figure 5, interpolating between a single augmentation (or central crop) and the mean embedding shows a clear trend: the negative log-likelihood decreases and accuracy increases as we move toward the mean. This suggests that the mean embedding is a more "correct" and calibrated representation of the underlying object identity.
Empirical Results on ImageNet
The effectiveness of MeTTA was validated using ImageNet-1K across various architectures and training paradigms:
- Self-Supervised Learning: For a ResNet-50 pre-trained with SimCLRv2, MeTTA (using 32 augmentations) improved top-1 linear evaluation accuracy from 71.7% to 73.8% (+2.1%).
- Supervised Learning: A standard supervised ResNet-50 saw an increase from 76.6% to 78.5% (+1.9%).
- Scalability: The benefit persists even for larger models. A ResNet-152 (3x width) achieved a gain of approximately +1.0%.
Beyond accuracy, MeTTA significantly improves model calibration, as evidenced by the reduction in negative log-likelihood.
Conclusion and Limitations
MeTTA provides a simple, "plug-and-play" method to boost the quality of pre-trained embeddings without retraining. It effectively trades additional inference-time computation for higher representation quality.
However, limitations remain. MeTTA is primarily designed for global image classification and may not directly translate to spatial tasks like object detection or segmentation, where averaging could blur coordinate-specific information. Additionally, the optimal augmentation policy for generating the mean embedding remains an area for future exploration.
Summary updated