Representation Consolidation for Training Expert Students

Zhizhong Li, Avinash Ravichandran, Charless C. Fowlkes, Marzia Polito, Rahul Bhotika, Stefano SoattoarXiv 2021 · OpenReview

Abstract

Knowledge distillation typically focuses on compressing a large model into a smaller one. However, when distilling specialized "expert" models, student models often suffer from "representation collapse," losing the general-purpose visual features necessary for broad transferability. We propose Representation Consolidation (RC), a multi-task distillation framework that leverages a generalist teacher as an anchor. By training a student on a shared backbone with multiple heads corresponding to various experts and a generalist, we produce a consolidated representation that outperforms individual experts on downstream tasks.

Introduction: The Challenge of Representation Collapse

In computer vision, specialized models (experts) are often preferred for niche domains like medical imaging or satellite photography. While these experts excel in their specific fields, fine-tuning them on narrow datasets often leads to "catastrophic forgetting" of general visual primitives like edges, textures, and basic shapes.

Consequently, an expert model might perform worse than a generic ImageNet-trained model when transferred to a new, even related, task. Representation Consolidation aims to build a single student model that retains both specialized expertise and general-purpose robustness, effectively creating a "library of experts" within a single backbone.

Methodology

The core of Representation Consolidation is a multi-head, multi-task distillation architecture. Instead of a one-to-one teacher-student mapping, the student learns from $N$ task-specific teachers and one "generalist" teacher (typically a model pre-trained on ImageNet).

Comparison of distillation strategies: (a) Traditional, (b) Multi-task, and (bottom) the proposed Representation Consolidation framework.

Key Components

  1. Proxy Dataset ($D_{proxy}$): To facilitate distillation without requiring the original training data of the teachers, a large unlabeled dataset (e.g., ImageNet or Places365) is used as a medium for knowledge transfer.
  2. Multi-Head Architecture: The student model ($\phi_s$) uses a shared backbone with multiple classifier heads ($h_s$). Each head is optimized to match the soft-label outputs of its respective teacher.
  3. Generalist Anchor: One head is dedicated to emulating a generalist teacher. This acts as a critical regularizer, preventing the backbone from over-specializing and ensuring it maintains the broad feature set necessary for transfer learning.
  4. Loss Weighting: A balanced loss function is employed where the generalist teacher is prioritized (typically with a weight $\lambda=1$), while specialized experts share the remaining weight.
Definitions of symbols used in the Representation Consolidation framework.

Experimental Results

The framework was evaluated across diverse domains, including automotive (Cars196), remote sensing (Resisc45), and fine-grained species classification (iNaturalist).

Superior Transferability

The consolidated student consistently outperformed the ImageNet baseline on tasks related to the teachers' expertise while maintaining parity on unrelated tasks. Notably, the student often surpassed its own teachers. For instance, on the Resisc45 satellite imagery task (5-shot), the student achieved 72.6% accuracy, significantly higher than the specialized teacher's 67.9% and the ImageNet baseline's 70.7%.

Multi-Expert Synergy

Consolidating multiple related experts (e.g., five different food-related models) resulted in a student representation stronger than any single expert in that domain. This suggests that RC effectively aggregates complementary features from different specialized sources. Furthermore, the method proved robust to the choice of proxy data; using Places365 (landscapes) as a proxy for learning about Cars still yielded strong results, indicating the method relies on the relationship between teacher outputs rather than specific image content.

Discussion and Conclusion

Representation Consolidation demonstrates that distillation can be repurposed as a powerful tool for feature engineering. By using a generalist anchor, we "rescue" representations from collapse, creating a foundation model that is both specialized and versatile.

Future work includes automating the selection and clustering of teachers to scale consolidation across thousands of expert models. While the method currently relies on a strong generalist baseline, its architecture-agnostic nature and robustness to proxy data make it a promising step toward universal visual representations.

Summary updated

Abstract

Traditionally, distillation has been used to train a student model to emulate the input/output functionality of a teacher. A more useful goal than emulation, yet under-explored, is for the student to learn feature representations that transfer well to future tasks. However, we observe that standard distillation of task-specific teachers actually *reduces* the transferability of student representations to downstream tasks. We show that a multi-head, multi-task distillation method using an unlabeled proxy dataset and a generalist teacher is sufficient to consolidate representations from task-specific teacher(s) and improve downstream performance, outperforming the teacher(s) and the strong baseline of ImageNet pretrained features. Our method can also combine the representational knowledge of multiple teachers trained on one or multiple domains into a single model, whose representation is improved on all teachers' domain(s).
BibTeX
@misc{li2021representation,
  author = {Zhizhong Li and Avinash Ravichandran and Charless C. Fowlkes and Marzia Polito and Rahul Bhotika and Stefano Soatto},
  title = {{Representation Consolidation for Training Expert Students}},
  year = {2021},
  eprint = {2107.08039},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.08039},
}
Figure