Uniform Priors for Data-Efficient Transfer

Samarth Sinha, Karsten Roth, Anirudh Goyal, Marzyeh Ghassemi, Hugo Larochelle, Animesh Garg

Abstract

Deep Neural Networks have shown great promise on a variety of downstream applications; but their ability to adapt and generalize to new data and tasks remains a challenge. However, the ability to perform few or zero-shot adaptation to novel tasks is important for the scalability and deployment of machine learning models. It is therefore crucial to understand what makes for good, transfer-able features in deep networks that best allow for such adaptation. In this paper, we shed light on this by showing that features that are most transferable have high uniformity in the embedding space and propose a uniformity regularization scheme that encourages better transfer and feature reuse. We evaluate the regularization on its ability to facilitate adaptation to unseen tasks and data, for which we conduct a thorough experimental study covering four relevant, and distinct domains: few-shot Meta-Learning, Deep Metric Learning, Zero-Shot Domain Adaptation, as well as Out-of-Distribution classification. Across all experiments, we show that uniformity regularization consistently offers benefits over baseline methods and is able to achieve state-of-the-art performance in Deep Metric Learning and Meta-Learning.

Introduction

While Deep Neural Networks (DNNs) excel at mastering specific tasks, they often struggle when faced with distribution shifts—scenarios where test data differs from training data. This limitation is particularly evident in few-shot learning and zero-shot adaptation. The core issue is often "feature collapse," where models overfit to training-specific biases, creating tight clusters that lack the flexibility needed for novel tasks. This paper investigates the properties of feature spaces that ensure representations are reusable and transferable.

Uniformity Regularization (UR)

The authors propose Uniformity Regularization (UR), a method to force internal representations (embeddings) to follow a uniform distribution, specifically a uniform hypercube $U(-1, 1)$. Because standard divergence measures like KL-divergence are ill-suited for bounded uniform distributions, the authors employ an adversarial framework:

  1. Discriminator: A sub-network trained to distinguish between model features and samples from a true uniform distribution.
  2. Task Model: The primary network is trained to perform its task while simultaneously attempting to "fool" the discriminator.

This creates a uniformity prior that discourages the model from making too many assumptions about the data, effectively maximizing entropy within the bounded feature space.

The Impact of Uniformity on Transfer

A key insight of this work is the relationship between the "flatness" of the prior and generalization performance. By testing Gaussian priors with increasing variance, the authors observed that as the prior approaches a uniform distribution, transfer performance improves significantly.

Performance comparison across different priors for domain adaptation tasks.

As shown in the table above, the uniform prior $U(-1, 1)$ consistently outperforms various Gaussian priors in digit recognition transfer tasks (e.g., MNIST $\to$ USPS), supporting the hypothesis that uniformity is a primary driver of transferability.

Experimental Results

The efficacy of UR was validated across four distinct domains:

Meta-Learning

On the Meta-Dataset benchmark, adding UR to the Universal Representation Transformer (URT) achieved a state-of-the-art average rank of 1.5. The regularization consistently improved performance across diverse datasets like Omniglot and Mini-ImageNet.

Meta-learning performance on standard benchmarks and the Meta-Dataset.

Deep Metric Learning (DML)

UR significantly boosted standard baselines. For instance, on the CUB200-2011 dataset, Recall@1 improved from 61.7% to 65.0% by simply adding UR to a Softmax baseline, allowing it to outperform more complex specialized algorithms.

Domain Adaptation and OOD Robustness

In zero-shot domain adaptation, accuracy in the MNIST $\to$ USPS task rose from 49.0% to 67.2%. Furthermore, models trained with UR showed increased resilience to physical augmentations (rotations, scaling), indicating that uniform features are more physically grounded and less prone to noise-induced failure.

Conclusion

Uniformity Regularization provides a simple yet powerful way to improve the data efficiency and transferability of deep learning models. By encouraging features to spread uniformly across the embedding space, UR preserves essential information and reduces task-specific bias. The empirical success across meta-learning, metric learning, and domain adaptation suggests that uniformity is a fundamental property of robust, reusable representations.

Summary updated

Abstract

Deep Neural Networks have shown great promise on a variety of downstream applications; but their ability to adapt and generalize to new data and tasks remains a challenge. However, the ability to perform few or zero-shot adaptation to novel tasks is important for the scalability and deployment of machine learning models. It is therefore crucial to understand what makes for good, transfer-able features in deep networks that best allow for such adaptation. In this paper, we shed light on this by showing that features that are most transferable have high uniformity in the embedding space and propose a uniformity regularization scheme that encourages better transfer and feature reuse. We evaluate the regularization on its ability to facilitate adaptation to unseen tasks and data, for which we conduct a thorough experimental study covering four relevant, and distinct domains: few-shot Meta-Learning, Deep Metric Learning, Zero-Shot Domain Adaptation, as well as Out-of-Distribution classification. Across all experiments, we show that uniformity regularization consistently offers benefits over baseline methods and is able to achieve state-of-the-art performance in Deep Metric Learning and Meta-Learning.
BibTeX
@misc{sinha2020uniform,
  author = {Samarth Sinha and Karsten Roth and Anirudh Goyal and Marzyeh Ghassemi and Hugo Larochelle and Animesh Garg},
  title = {{Uniform Priors for Data-Efficient Transfer}},
  year = {2020},
  eprint = {2006.16524},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2006.16524},
}
Figure