Abstract
Knowledge distillation typically focuses on compressing a large model into a smaller one. However, when distilling specialized "expert" models, student models often suffer from "representation collapse," losing the general-purpose visual features necessary for broad transferability. We propose Representation Consolidation (RC), a multi-task distillation framework that leverages a generalist teacher as an anchor. By training a student on a shared backbone with multiple heads corresponding to various experts and a generalist, we produce a consolidated representation that outperforms individual experts on downstream tasks.
Introduction: The Challenge of Representation Collapse
In computer vision, specialized models (experts) are often preferred for niche domains like medical imaging or satellite photography. While these experts excel in their specific fields, fine-tuning them on narrow datasets often leads to "catastrophic forgetting" of general visual primitives like edges, textures, and basic shapes.
Consequently, an expert model might perform worse than a generic ImageNet-trained model when transferred to a new, even related, task. Representation Consolidation aims to build a single student model that retains both specialized expertise and general-purpose robustness, effectively creating a "library of experts" within a single backbone.
Methodology
The core of Representation Consolidation is a multi-head, multi-task distillation architecture. Instead of a one-to-one teacher-student mapping, the student learns from $N$ task-specific teachers and one "generalist" teacher (typically a model pre-trained on ImageNet).
Key Components
- Proxy Dataset ($D_{proxy}$): To facilitate distillation without requiring the original training data of the teachers, a large unlabeled dataset (e.g., ImageNet or Places365) is used as a medium for knowledge transfer.
- Multi-Head Architecture: The student model ($\phi_s$) uses a shared backbone with multiple classifier heads ($h_s$). Each head is optimized to match the soft-label outputs of its respective teacher.
- Generalist Anchor: One head is dedicated to emulating a generalist teacher. This acts as a critical regularizer, preventing the backbone from over-specializing and ensuring it maintains the broad feature set necessary for transfer learning.
- Loss Weighting: A balanced loss function is employed where the generalist teacher is prioritized (typically with a weight $\lambda=1$), while specialized experts share the remaining weight.
Experimental Results
The framework was evaluated across diverse domains, including automotive (Cars196), remote sensing (Resisc45), and fine-grained species classification (iNaturalist).
Superior Transferability
The consolidated student consistently outperformed the ImageNet baseline on tasks related to the teachers' expertise while maintaining parity on unrelated tasks. Notably, the student often surpassed its own teachers. For instance, on the Resisc45 satellite imagery task (5-shot), the student achieved 72.6% accuracy, significantly higher than the specialized teacher's 67.9% and the ImageNet baseline's 70.7%.
Multi-Expert Synergy
Consolidating multiple related experts (e.g., five different food-related models) resulted in a student representation stronger than any single expert in that domain. This suggests that RC effectively aggregates complementary features from different specialized sources. Furthermore, the method proved robust to the choice of proxy data; using Places365 (landscapes) as a proxy for learning about Cars still yielded strong results, indicating the method relies on the relationship between teacher outputs rather than specific image content.
Discussion and Conclusion
Representation Consolidation demonstrates that distillation can be repurposed as a powerful tool for feature engineering. By using a generalist anchor, we "rescue" representations from collapse, creating a foundation model that is both specialized and versatile.
Future work includes automating the selection and clustering of teachers to scale consolidation across thousands of expert models. While the method currently relies on a strong generalist baseline, its architecture-agnostic nature and robustness to proxy data make it a promising step toward universal visual representations.
Summary updated