MHFC

Shuai Shao, Lei Xing, Yan Wang, Rui Xu, Chunyan Zhao, Baodi LiuarXiv 2021 · DOI · arXiv

Abstract

Few-shot learning (FSL) aims to address the data-scarce problem. A standard FSL framework is composed of two components: (1) Pre-train. Employ the base data to generate a CNN-based feature extraction model (FEM). (2) Meta-test. Apply the trained FEM to acquire the novel data's features and recognize them. FSL relies heavily on the design of the FEM. However, various FEMs have distinct emphases. For example, several may focus more attention on the contour information, whereas others may lay particular emphasis on the texture information. The single-head feature is only a one-sided representation of the sample. Besides the negative influence of cross-domain (e.g., the trained FEM can not adapt to the novel class flawlessly), the distribution of novel data may have a certain degree of deviation compared with the ground truth distribution, which is dubbed as distribution-shift-problem (DSP). To address the DSP, we propose Multi-Head Feature Collaboration (MHFC) algorithm, which attempts to project the multi-head features (e.g., multiple features extracted from a variety of FEMs) to a unified space and fuse them to capture more discriminative information. Typically, first, we introduce a subspace learning method to transform the multi-head features to aligned low-dimensional representations. It corrects the DSP via learning the feature with more powerful discrimination and overcomes the problem of inconsistent measurement scales from different head features. Then, we design an attention block to update combination weights for each head feature automatically. It comprehensively considers the contribution of various perspectives and further improves the discrimination of features. We evaluate the proposed method on five benchmark datasets (including cross-domain experiments) and achieve significant improvements of 2.1%-7.8% compared with state-of-the-arts.

Introduction

Few-shot learning (FSL) aims to recognize new objects using only a handful of labeled examples. A major hurdle in FSL is the distribution-shift-problem (DSP). This occurs because the feature extraction models (FEMs) trained on a large "base" dataset often fail to perfectly adapt to "novel" classes.

Furthermore, most FSL methods rely on a single "head" or type of feature. However, different models have different "biases"—one might focus on object contours while another prioritizes texture. Relying on a single perspective provides a one-sided representation that is sensitive to noise and distribution shifts. The Multi-Head Feature Collaboration (MHFC) framework addresses this by fusing diverse perspectives from multiple feature extractors to create a more robust representation.

The MHFC Framework

The proposed MHFC algorithm is a non-parametric fusion strategy that can be applied to features extracted from existing, pre-trained models. It consists of three main stages:

The architecture of the Multi-Head Feature Collaboration (MHFC) framework.
  1. Subspace Learning: Since different FEMs produce features in different mathematical spaces with inconsistent scales, they cannot be simply concatenated. MHFC uses subspace learning (like Laplacian Eigenmaps or PCA) to project these diverse features into a unified, aligned, low-dimensional space. This step also helps filter out redundant information and corrects the DSP by learning more discriminative features.
  2. Multi-Head Feature Attention Block: Not all features are equally useful for every task. The authors designed an attention block that automatically calculates optimal weights for each feature head. It does this by evaluating the classification loss of each head within a specific "episode" (a small sub-task) and assigning higher weights to more discriminative features.
  3. Collaborative Classification: The weighted features are concatenated to form a robust "collaborative representation." This representation is then used to train a simple, regularized linear regression classifier to identify the query samples.

Experimental Evaluation

The authors evaluated MHFC on five widely-used benchmark datasets: mini-ImageNet, tiered-ImageNet, CIFAR-FS, FC100, and CUB (used specifically for cross-domain experiments).

MHFC achieved significant improvements of 2.1% to 7.8% over previous state-of-the-art methods across all standard benchmarks. The gains were particularly pronounced in the most difficult "1-shot" scenarios. For instance, on CIFAR-FS, it improved 1-shot accuracy by nearly 7%. In the challenging mini-ImageNet → CUB task, MHFC outperformed the next best method by a staggering 13.3% in the 1-shot setting, proving its effectiveness at handling extreme distribution shifts.

t-SNE visualization comparing individual feature heads (a, b) vs. the fused multi-head representation (c).

As shown in the t-SNE visualizations above, combining "imperfect" features from different self-supervised tasks (like predicting image rotation vs. image mirroring) creates a representation much stronger than the sum of its parts, leading to significantly clearer class separation.

Key Insights

  • Diversity is Strength: Fusing multiple perspectives provides a more holistic and stable representation of data than any single model can provide.
  • Alignment Matters: You cannot simply "glue" different features together. The use of subspace learning to align measurement scales is critical for effective collaboration between different feature heads.
  • Dynamic Weighting: The importance of a feature (e.g., texture vs. shape) changes depending on the specific classes being compared. The attention block allows the model to adapt its focus per task.

Conclusion and Future Work

MHFC effectively overcomes the distribution-shift problem by looking at data through multiple lenses. While highly effective as a post-processing fusion step, future work could involve integrating this multi-head collaboration directly into the end-to-end training process of the neural networks themselves to reduce computational overhead and further improve performance.

Summary updated

Abstract

Few-shot learning (FSL) aims to address the data-scarce problem. A standard FSL framework is composed of two components: (1) Pre-train. Employ the base data to generate a CNN-based feature extraction model (FEM). (2) Meta-test. Apply the trained FEM to acquire the novel data's features and recognize them. FSL relies heavily on the design of the FEM. However, various FEMs have distinct emphases. For example, several may focus more attention on the contour information, whereas others may lay particular emphasis on the texture information. The single-head feature is only a one-sided representation of the sample. Besides the negative influence of cross-domain (e.g., the trained FEM can not adapt to the novel class flawlessly), the distribution of novel data may have a certain degree of deviation compared with the ground truth distribution, which is dubbed as distribution-shift-problem (DSP). To address the DSP, we propose Multi-Head Feature Collaboration (MHFC) algorithm, which attempts to project the multi-head features (e.g., multiple features extracted from a variety of FEMs) to a unified space and fuse them to capture more discriminative information. Typically, first, we introduce a subspace learning method to transform the multi-head features to aligned low-dimensional representations. It corrects the DSP via learning the feature with more powerful discrimination and overcomes the problem of inconsistent measurement scales from different head features. Then, we design an attention block to update combination weights for each head feature automatically. It comprehensively considers the contribution of various perspectives and further improves the discrimination of features. We evaluate the proposed method on five benchmark datasets (including cross-domain experiments) and achieve significant improvements of 2.1%-7.8% compared with state-of-the-arts.
BibTeX
@misc{shao2021mhfc,
  author = {Shuai Shao and Lei Xing and Yan Wang and Rui Xu and Chunyan Zhao and Yanjiang Wang and Baodi Liu},
  title = {{MHFC}},
  year = {2021},
  eprint = {2109.07785},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.07785},
}
Figure