Introduction: The Invariance Trade-off
Standard visual pre-training paradigms often enforce a "one-size-fits-all" inductive bias: high invariance to image transformations. By using heavy data augmentation during pre-training on datasets like ImageNet, models learn to produce identical feature representations regardless of whether an image is cropped, flipped, or color-distorted. While this robustness is beneficial for object recognition—where a "cat" remains a "cat" despite spatial shifts—it is fundamentally detrimental to tasks requiring sensitivity to spatial orientation or fine-grained appearance details, such as 6D pose estimation or facial landmark detection.
Currently, no single pre-trained feature set is optimal for all downstream applications. This paper introduces Quality Diversity for Vision (QD4V), a framework designed to learn a small ensemble of high-quality features with diverse sensitivities, allowing downstream tasks to select the features that best suit their specific requirements.
The QD4V Framework
Inspired by Quality-Diversity algorithms in evolutionary robotics, QD4V aims to find a set of solutions that are both high-performing and behaviorally diverse. The framework defines a model's "phenotype" as its degree of invariance to a set of $K$ transformations, measured by the cosine similarity between features of an original image and its augmented version.
Training Objective
The ensemble of $N$ feature extractors is trained simultaneously using a multi-part loss function:
- Quality Loss: A standard task loss (e.g., Cross-Entropy for supervised or Contrastive loss for self-supervised) ensures the features remain useful for general representation.
- Diversity Loss: This term explicitly pushes the phenotypes of ensemble members apart, forcing them to develop varying sensitivities to different transformations.
- KL-Divergence Loss: A distillation term encourages ensemble members to maintain the high performance of a standard baseline model.
To maintain computational efficiency, the ensemble members share the first three stages of a backbone (e.g., ResNet50) and only diverge in the final stage.
Downstream Fusion
To utilize the ensemble, QD4V employs a "stacking" approach. Rather than simple concatenation, a linear head is trained for each ensemble member on the downstream task. A set of weights is then learned on a validation set to optimally combine their predictions. This mechanism acts as an automated selector: for pose estimation, weights shift toward members sensitive to spatial transforms; for classification, they favor invariant members.
Experimental Results
QD4V was evaluated across 14 diverse tasks, including standard classification (CIFAR, Caltech101), spatially sensitive regression (300W, CelebA, human pose), and dense estimation (object detection, semantic segmentation).
The framework demonstrated superior versatility, achieving the best average rank compared to standard supervised and self-supervised (MoCo) baselines. The gains were particularly pronounced in spatially sensitive regression tasks, with improvements of up to 12.5% on CelebA and 8.5% on Causal3D compared to a standard supervised ResNet50. Notably, QD4V outperformed the massive CLIP model on several pose-sensitive tasks, suggesting that the correct inductive bias can be more impactful than sheer data scale.
Furthermore, the stacking approach provides a theoretical speedup in convergence ($O(m^{-1})$), allowing the model to learn the optimal feature combination rapidly even with limited downstream data.
Discussion and Future Work
While QD4V significantly improves task versatility, it introduces some computational overhead due to the ensemble architecture, although shared layers mitigate this. Currently, diversity is defined by transformation invariances; future research could explore other behavioral metrics, such as sensitivity to frequency components. Additionally, while this work focuses on convolutional architectures like ResNet and ConvNeXt, extending these diversity principles to Vision Transformers (ViTs) remains a promising direction for future exploration.
Summary updated