How to Trust Unlabeled Data? Instance Credibility Inference for Few-Shot Learning

Yikai Wang, Li Zhang, Yuan Yao, Yanwei Fu

TL;DR

This paper introduces Instance Credibility Inference (ICI), a novel statistical approach for Few-Shot Learning (FSL) that leverages unlabeled data to improve model performance. ICI iteratively selects the most trustworthy pseudo-labeled instances from unlabeled data by analyzing the sparsity of incidental parameters in a generalized linear model, effectively filtering out noisy labels and enhancing classifier training.

What Problem Does This Paper Address?

Deep learning models excel at image recognition but require vast amounts of labeled data, which is often expensive and time-consuming to acquire. This limitation hinders their application in real-world scenarios with "long-tail" distributions of categories, where some classes have very few labeled examples. Few-Shot Learning (FSL) aims to address this by enabling models to learn from a small number of labeled instances. While existing FSL methods often rely on meta-learning or data augmentation, this paper tackles the challenge of effectively utilizing abundant unlabeled data to boost FSL performance, specifically by addressing the inherent noise introduced by pseudo-labeling in self-taught learning paradigms.

Methods

The core of the proposed method is Instance Credibility Inference (ICI). It builds upon the self-taught learning framework, where an initial classifier trained on a small labeled dataset is used to predict pseudo-labels for unlabeled data. The novelty lies in how ICI assesses the credibility of these pseudo-labels.

  1. Generalized Linear Model with Incidental Parameters: ICI models the relationship between features and labels using a generalized linear model that includes "incidental parameters." These parameters are designed to capture instance-specific deviations or noise.
  2. Sparsity and Regularization Path: The credibility of a pseudo-labeled instance is determined by the sparsity of its corresponding incidental parameter. By analyzing the regularization path of these parameters (i.e., how their sparsity changes as a regularization parameter varies), ICI can identify instances that are more reliably labeled. Instances whose incidental parameters become sparse at an earlier stage of regularization are considered more trustworthy.
  3. Iterative Selection and Retraining: The process is iterative. In each step, ICI selects a subset of the most credible pseudo-labeled instances and adds them to the training set. The classifier is then retrained on this augmented dataset. This process continues until all unlabeled data has been considered or the pseudo-labels have converged.
  4. Theoretical Guarantees: The authors provide theoretical analysis showing that under specific conditions (restricted eigenvalue, irrepresentability, and large error), ICI can guarantee the selection of all correctly predicted pseudo-labeled instances, effectively filtering out noise.
  5. Extension to Logistic Regression: While initially formulated for linear regression, ICI is extended to logistic regression, a more common choice for classification tasks, by reformulating it as a generalized linear model with an extended feature space.

Datasets

The proposed method was evaluated on four widely used few-shot visual recognition benchmark datasets:

  • miniImageNet: 100 classes, 600 instances per category.
  • tieredImageNet: A larger dataset with a hierarchical structure, 351 classes in the base set and 160 in the novel set.
  • CIFAR-FS: Derived from CIFAR-100, with lower-resolution images.
  • CUB (Caltech-UCSD Birds-200-2011): A fine-grained dataset of bird categories.

Key Insights and Ideas

  • Trustworthy Pseudo-Labels are Key: The paper highlights that simply using pseudo-labels from an initial classifier can be detrimental if the labels are noisy. ICI's core contribution is a principled way to identify and leverage trustworthy pseudo-labels.
  • Incidental Parameters as Credibility Indicators: The novel idea of using the sparsity of incidental parameters in a generalized linear model as a measure of instance credibility is a significant conceptual contribution.
  • Regularization Path for Robustness: Analyzing the regularization path provides a robust mechanism to rank instance credibility, going beyond simple confidence scores from the classifier.
  • Orthogonality to Classifiers: ICI is designed to be orthogonal to the choice of the base classifier, meaning it can be integrated with various existing FSL methods to improve their performance.
  • Theoretical Foundation for Practical Improvement: The paper provides theoretical guarantees for ICI's ability to select correct instances, lending strong support to its practical effectiveness.

Key Results

  • Improved Accuracy: ICI consistently improved performance across various few-shot learning settings (inductive, transductive, and semi-supervised) on the tested datasets. For instance, in the transductive setting on miniImageNet, ICI achieved accuracies of 71.19% (1-shot) and 82.55% (5-shot) with its logistic regression variant (ICIC), outperforming several strong baselines.
  • Effective Noise Reduction: Ablation studies demonstrated that ICI significantly outperformed other sample selection strategies (random, nearest-neighbor, confidence-based) by effectively filtering out noisy pseudo-labeled instances.
  • Robustness: The method showed robustness to the choice of initial classifier and dimensionality reduction techniques.
  • Iterative Selection Benefits: Experiments indicated that iterative selection of samples with ICI leads to better accuracy than selecting a larger batch of samples in a single iteration.

Limitations and Future Work

  • Hyperparameter Tuning: The method involves hyperparameters (e.g., regularization parameters $\lambda_1, \lambda_2$) that require tuning, although the paper suggests a strategy for finding a good balance.
  • Assumptions for Theoretical Guarantees: The theoretical guarantees rely on specific assumptions about the data distribution and model properties (restricted eigenvalue, irrepresentability, large error). While the paper investigates the practical adherence to these assumptions, they might not always hold perfectly in real-world scenarios.
  • Future Work: The authors suggest exploring how to incorporate ICI with other state-of-the-art methods and investigating the influence of the initial classifier's accuracy on the overall performance. They also mention that the theoretical analysis of ICI for generalized linear models is an ongoing area of research.

Summary updated

Abstract

Deep learning based models have excelled in many computer vision tasks and appear to surpass humans' performance. However, these models require an avalanche of expensive human labeled training data and many iterations to train their large number of parameters. This severely limits their scalability to the real-world long-tail distributed categories, some of which are with a large number of instances, but with only a few manually annotated. Learning from such extremely limited labeled examples is known as Few-shot learning (FSL). Different to prior arts that leverage meta-learning or data augmentation strategies to alleviate this extremely data-scarce problem, this paper presents a statistical approach, dubbed Instance Credibility Inference (ICI) to exploit the support of unlabeled instances for few-shot visual recognition. Typically, we repurpose the self-taught learning paradigm to predict pseudo-labels of unlabeled instances with an initial classifier trained from the few shot and then select the most confident ones to augment the training set to re-train the classifier. This is achieved by constructing a (Generalized) Linear Model (LM/GLM) with incidental parameters to model the mapping from (un-)labeled features to their (pseudo-)labels, in which the sparsity of the incidental parameters indicates the credibility of the corresponding pseudo-labeled instance. We rank the credibility of pseudo-labeled instances along the regularization path of their corresponding incidental parameters, and the most trustworthy pseudo-labeled examples are preserved as the augmented labeled instances. Theoretically, under mild conditions of restricted eigenvalue, irrepresentability, and large error, our approach is guaranteed to collect all the correctly-predicted instances from the noisy pseudo-labeled set.
BibTeX
@article{wang2022how,
  author = {Yikai Wang and Li Zhang and Yuan Yao and Yanwei Fu},
  title = {{How to Trust Unlabeled Data? Instance Credibility Inference for Few-Shot Learning}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  doi = {10.1109/tpami.2021.3086140},
}
Figure