Mean Embeddings with Test-Time Data Augmentation for Ensembling of Representations

Arsenii Ashukha, Andrei Atanov, Dmitry P. VetrovarXiv 2021 · arXiv

Introduction

In deep learning, ensembling—averaging the predictions of multiple models—is a standard technique for improving performance and uncertainty estimation. However, ensembling internal representations (embeddings) is non-trivial. Feature vectors from independently trained models are typically unaligned; a specific semantic feature might occupy different indices in different models, making direct averaging meaningless.

Researchers from Samsung AI and EPFL introduce MeTTA (Mean Embeddings with Test-Time Augmentation) to address this. Instead of ensembling across different models, MeTTA ensembles different "views" of the same input through a single model, ensuring feature alignment while improving robustness and accuracy.

The MeTTA Method

The core idea of MeTTA is to leverage the same model to process multiple augmented versions of a single input image. By averaging these representations, the resulting "mean embedding" becomes more invariant to the specific transformations applied during inference.

Overview of the MeTTA pipeline: multiple augmentations of an image are passed through the same DNN, and their resulting embeddings are averaged to produce a more robust mean embedding.

As shown in Figure 1, the process involves:

  1. Generating $S$ stochastic augmentations (e.g., random crops and flips) of an image $x$.
  2. Passing these $S$ versions through a pre-trained neural network.
  3. Computing the arithmetic mean of the resulting feature vectors.

Because all augmentations pass through the same network weights, the feature spaces are perfectly aligned, making the mean embedding a mathematically sound representation.

Addressing Representation Instability

Standard embeddings often suffer from a lack of invariance. Even if a model is trained with data augmentation, its final representations can "jitter" significantly with small changes in the input, such as a slight shift in a crop. This instability can lead to inconsistent classifications (e.g., misidentifying a "wombat" as a "koala" due to a specific crop).

MeTTA mitigates this by finding a more stable region in the feature space. The authors demonstrate that the mean embedding resides in a "favorable direction" relative to standard central-crop embeddings.

Negative log-likelihood and Accuracy as a function of the interpolation step between central-crop/augmentation embeddings and the mean embedding. Moving toward the mean embedding consistently improves performance.

As illustrated in Figure 5, interpolating between a single augmentation (or central crop) and the mean embedding shows a clear trend: the negative log-likelihood decreases and accuracy increases as we move toward the mean. This suggests that the mean embedding is a more "correct" and calibrated representation of the underlying object identity.

Empirical Results on ImageNet

The effectiveness of MeTTA was validated using ImageNet-1K across various architectures and training paradigms:

  • Self-Supervised Learning: For a ResNet-50 pre-trained with SimCLRv2, MeTTA (using 32 augmentations) improved top-1 linear evaluation accuracy from 71.7% to 73.8% (+2.1%).
  • Supervised Learning: A standard supervised ResNet-50 saw an increase from 76.6% to 78.5% (+1.9%).
  • Scalability: The benefit persists even for larger models. A ResNet-152 (3x width) achieved a gain of approximately +1.0%.

Beyond accuracy, MeTTA significantly improves model calibration, as evidenced by the reduction in negative log-likelihood.

Conclusion and Limitations

MeTTA provides a simple, "plug-and-play" method to boost the quality of pre-trained embeddings without retraining. It effectively trades additional inference-time computation for higher representation quality.

However, limitations remain. MeTTA is primarily designed for global image classification and may not directly translate to spatial tasks like object detection or segmentation, where averaging could blur coordinate-specific information. Additionally, the optimal augmentation policy for generating the mean embedding remains an area for future exploration.

Summary updated

Abstract

Averaging predictions over a set of models -- an ensemble -- is widely used to improve predictive performance and uncertainty estimation of deep learning models. At the same time, many machine learning systems, such as search, matching, and recommendation systems, heavily rely on embeddings. Unfortunately, due to misalignment of features of independently trained models, embeddings, cannot be improved with a naive deep ensemble like approach. In this work, we look at the ensembling of representations and propose mean embeddings with test-time augmentation (MeTTA) simple yet well-performing recipe for ensembling representations. Empirically we demonstrate that MeTTA significantly boosts the quality of linear evaluation on ImageNet for both supervised and self-supervised models. Even more exciting, we draw connections between MeTTA, image retrieval, and transformation invariant models. We believe that spreading the success of ensembles to inference higher-quality representations is the important step that will open many new applications of ensembling.
BibTeX
@misc{ashukha2021mean,
  author = {Arsenii Ashukha and Andrei Atanov and Dmitry P. Vetrov},
  title = {{Mean Embeddings with Test-Time Data Augmentation for Ensembling of Representations}},
  year = {2021},
  eprint = {2106.08038},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2106.08038},
}
Figure