Revisiting Few-Shot Learning From a Causal Perspective

Guoliang Lin, Yongheng Xu, Hanjiang Lai, Jian Yin

TL;DR

This paper re-examines existing metric-based few-shot learning (FSL) methods through the lens of causal inference, proposing that their success stems from implicitly performing "front-door adjustments" to mitigate spurious correlations. Building on this, the authors introduce two novel FSL methods that explicitly leverage causal principles by considering both inter-example relationships and the diversity of learned representations, achieving superior performance on benchmark datasets.

What Problem Does This Paper Address?

Few-shot learning (FSL) aims to train models that can generalize to new classes with very limited labeled data, a common scenario in real-world applications. While metric-based FSL methods, which rely on learning a distance metric between examples, have shown impressive progress, the underlying reasons for their success remain unclear. The authors argue that a key challenge is the presence of "spurious correlations" in the training data, often introduced by unobserved confounding factors (e.g., how images are captured). These confounders can lead models to learn superficial associations (e.g., "sky" with "bird") rather than true causal relationships, hindering generalization to unseen data distributions. Existing methods, while effective, lack a clear theoretical explanation for how they overcome these spurious correlations.

Methods

The core contribution of this paper is a novel causal interpretation of existing metric-based FSL methods. They frame the problem using a Structural Causal Model (SCM) where an example ($X$) influences a label ($Y$) through a learned representation ($Z$). Unobserved confounders ($U$) can create spurious correlations ($X \leftarrow U \rightarrow Y$). The authors demonstrate that popular FSL methods like Matching Networks and Prototypical Networks can be understood as implicitly performing a "front-door adjustment" (a causal inference technique that doesn't require observing the confounder) to isolate the true causal effect of $X$ on $Y$ via $Z$.

Based on this causal framework, they propose two new FSL methods:

  1. Ensemble Method: This approach leverages multiple visual encoders (e.g., CLIP and BLIP) to generate diverse representations of the same input. By combining predictions from these diverse representations, the method aims to mitigate biases introduced by individual encoders and capture more robust causal relationships.
  2. Stochastic Mapping (SM) Method: This method introduces stochasticity into the representation learning process, for example, by using dropout. The idea is to generate a distribution of representations for a given input, which helps in learning more robust causal associations and preventing overfitting to specific spurious correlations.

Both methods are designed to explicitly consider the diversity of representations, an aspect they argue is overlooked by prior work.

Datasets

The paper evaluates their proposed methods on a comprehensive set of 10 benchmark image classification datasets:

  • ImageNet
  • Caltech101
  • DTD (Describable Textures Dataset)
  • EuroSAT
  • FGVCAircraft
  • Flowers102
  • Food101
  • OxfordPets
  • StanfordCars
  • SUN397

These datasets cover a range of tasks, from general object recognition to fine-grained classification and scene categorization, and are standard benchmarks for FSL research.

Key Insights and Ideas

  • Causal Interpretation of FSL Success: The paper provides a compelling theoretical explanation for why metric-based FSL methods work by linking them to the well-established causal inference technique of front-door adjustment. This offers a unified perspective on existing approaches.
  • The Importance of Representation Diversity: The authors highlight that while existing methods focus on the "interaction" between examples and representations, they often neglect the "diversity" of these representations. Their work suggests that explicitly incorporating representation diversity is crucial for robust FSL.
  • Unobserved Confounders as a Key Challenge: The paper clearly articulates how unobserved confounders can lead to spurious correlations, which are particularly problematic in low-data regimes like FSL.
  • Front-Door Adjustment as a Solution: The front-door adjustment is presented as a powerful tool for deconfounding, especially when confounders are unobservable, making it highly relevant for FSL.

Key Results

The proposed causal methods, Ensemble Method and Stochastic Mapping (SM) Method, demonstrate significant improvements over state-of-the-art baselines on the evaluated datasets.

  • ImageNet Performance: The Ours-Ensemble method achieved 72.73% accuracy on ImageNet with 16 shots, outperforming Tip-Adapter-F (73.68% with ViT/16 encoder) and other baselines. The SM method also showed strong performance.
  • General Improvement: Across the 10 benchmark datasets, the proposed methods consistently improved classification accuracies. For instance, the Ensemble Method showed significant gains, with p-values indicating statistically significant improvements over baselines.
  • Ensemble Benefits: The ensemble method, by combining diverse visual encoders (CLIP and BLIP), showed superior performance compared to single-encoder methods and even other ensemble approaches like CaFo, despite CaFo using more complex models.
  • Stochasticity Benefits: The SM method, by introducing stochasticity (e.g., via dropout), also boosted performance, demonstrating the value of exploring a distribution of representations.

Limitations and Future Work

  • Theoretical Explanation Scope: While the paper provides a causal interpretation for metric-based methods, the authors suggest that future work could extend this theoretical explanation to a broader range of few-shot learning methods.
  • Hyperparameter Tuning: The paper mentions that their proposed methods involve hyperparameter tuning (e.g., $\alpha$ and $\beta$ for the ensemble method, $p$ for the SM method), and ablation studies were conducted to find optimal values. Further research could explore more automated or adaptive hyperparameter selection.
  • Simplicity of Causal Methods: The authors present their two causal methods as simple examples to illustrate their framework. Future work could explore more sophisticated causal inference techniques and their integration into FSL.

Summary updated

Abstract

Few-shot learning with $N$-way $K$-shot scheme is an open challenge in machine learning. Many metric-based approaches have been proposed to tackle this problem, e.g., the Matching Networks and CLIP-Adapter. Despite that these approaches have shown significant progress, the mechanism of why these methods succeed has not been well explored. In this paper, we try to interpret these metric-based few-shot learning methods via causal mechanism. We show that the existing approaches can be viewed as specific forms of front-door adjustment, which can alleviate the effect of spurious correlations and thus learn the causality. This causal interpretation could provide us a new perspective to better understand these existing metric-based methods. Further, based on this causal interpretation, we simply introduce two causal methods for metric-based few-shot learning, which considers not only the relationship between examples but also the diversity of representations. Experimental results demonstrate the superiority of our proposed methods in few-shot classification on various benchmark datasets.
BibTeX
@article{lin2024revisiting,
  author = {Guoliang Lin and Yongheng Xu and Hanjiang Lai and Jian Yin},
  title = {{Revisiting Few-Shot Learning From a Causal Perspective}},
  journal = {IEEE Transactions on Knowledge and Data Engineering},
  year = {2024},
  url = {https://openreview.net/forum?id=P7gx0yUgXH},
}
Figure