Quality Diversity for Visual Pre-Training

Ruchika Chavhan, Henry Gouk, Da Li, Timothy M. Hospedales

Introduction: The Invariance Trade-off

Standard visual pre-training paradigms often enforce a "one-size-fits-all" inductive bias: high invariance to image transformations. By using heavy data augmentation during pre-training on datasets like ImageNet, models learn to produce identical feature representations regardless of whether an image is cropped, flipped, or color-distorted. While this robustness is beneficial for object recognition—where a "cat" remains a "cat" despite spatial shifts—it is fundamentally detrimental to tasks requiring sensitivity to spatial orientation or fine-grained appearance details, such as 6D pose estimation or facial landmark detection.

Conceptual trade-off between invariance and quality across different vision tasks.

Currently, no single pre-trained feature set is optimal for all downstream applications. This paper introduces Quality Diversity for Vision (QD4V), a framework designed to learn a small ensemble of high-quality features with diverse sensitivities, allowing downstream tasks to select the features that best suit their specific requirements.

The QD4V Framework

Inspired by Quality-Diversity algorithms in evolutionary robotics, QD4V aims to find a set of solutions that are both high-performing and behaviorally diverse. The framework defines a model's "phenotype" as its degree of invariance to a set of $K$ transformations, measured by the cosine similarity between features of an original image and its augmented version.

The QD4V training objective, incorporating quality, diversity, and distillation losses.

Training Objective

The ensemble of $N$ feature extractors is trained simultaneously using a multi-part loss function:

  1. Quality Loss: A standard task loss (e.g., Cross-Entropy for supervised or Contrastive loss for self-supervised) ensures the features remain useful for general representation.
  2. Diversity Loss: This term explicitly pushes the phenotypes of ensemble members apart, forcing them to develop varying sensitivities to different transformations.
  3. KL-Divergence Loss: A distillation term encourages ensemble members to maintain the high performance of a standard baseline model.

To maintain computational efficiency, the ensemble members share the first three stages of a backbone (e.g., ResNet50) and only diverge in the final stage.

Downstream Fusion

To utilize the ensemble, QD4V employs a "stacking" approach. Rather than simple concatenation, a linear head is trained for each ensemble member on the downstream task. A set of weights is then learned on a validation set to optimally combine their predictions. This mechanism acts as an automated selector: for pose estimation, weights shift toward members sensitive to spatial transforms; for classification, they favor invariant members.

Experimental Results

QD4V was evaluated across 14 diverse tasks, including standard classification (CIFAR, Caltech101), spatially sensitive regression (300W, CelebA, human pose), and dense estimation (object detection, semantic segmentation).

The framework demonstrated superior versatility, achieving the best average rank compared to standard supervised and self-supervised (MoCo) baselines. The gains were particularly pronounced in spatially sensitive regression tasks, with improvements of up to 12.5% on CelebA and 8.5% on Causal3D compared to a standard supervised ResNet50. Notably, QD4V outperformed the massive CLIP model on several pose-sensitive tasks, suggesting that the correct inductive bias can be more impactful than sheer data scale.

Furthermore, the stacking approach provides a theoretical speedup in convergence ($O(m^{-1})$), allowing the model to learn the optimal feature combination rapidly even with limited downstream data.

Discussion and Future Work

While QD4V significantly improves task versatility, it introduces some computational overhead due to the ensemble architecture, although shared layers mitigate this. Currently, diversity is defined by transformation invariances; future research could explore other behavioral metrics, such as sensitivity to frequency components. Additionally, while this work focuses on convolutional architectures like ResNet and ConvNeXt, extending these diversity principles to Vision Transformers (ViTs) remains a promising direction for future exploration.

Summary updated

Abstract

Models pre-trained on large datasets such as ImageNet provide the de-facto standard for transfer learning, with both supervised and self-supervised approaches proving effective. However, emerging evidence suggests that any single pre-trained feature will not perform well on diverse downstream tasks. Each pre-training strategy encodes a certain inductive bias, which may suit some downstream tasks but not others. Notably, the augmentations used in both supervised and self-supervised training lead to features with high invariance to spatial and appearance transformations. This renders them sub-optimal for tasks that demand sensitivity to these factors. In this paper we develop a feature that better supports diverse downstream tasks by providing a diverse set of sensitivities and invariances. In particular, we are inspired by Quality-Diversity in evolution, to define a pre-training objective that requires high quality yet diverse features — where diversity is defined in terms of transformation (in)variances. Our framework plugs in to both supervised and self-supervised pre-training, and produces a small ensemble of features. We further show how downstream tasks can easily and efficiently select their preferred (in)variances. Both empirical and theoretical analysis show the efficacy of our representation and transfer learning approach for diverse downstream tasks. Code available at https://github.com/ruchikachavhan/quality-diversity-pretraining.git
BibTeX
@inproceedings{chavhan2023quality,
  author = {Ruchika Chavhan and Henry Gouk and Da Li and Timothy M. Hospedales},
  title = {{Quality Diversity for Visual Pre-Training}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00496},
}
Figure