Multidimensional Belief Quantification for Label-Efficient Meta-Learning

Deep Pandey, Qi Yu

Abstract

Optimization-based meta-learning often struggles with overconfident incorrect predictions and high labeling costs due to random task sampling. We present Units-ML, a framework that utilizes Subjective Logic to decompose uncertainty into vacuous (lack of evidence) and conflicting (contradictory evidence) components. A key theoretical contribution is the proof that conflicting belief lower-bounds the probability of incorrect predictions, enabling label-free estimation of task difficulty. By implementing a multi-query task selection strategy based on these belief measures, Units-ML significantly improves label efficiency and model reliability across various benchmarks, including Omniglot and mini-ImageNet.

Introduction

Optimization-based meta-learning, exemplified by algorithms like MAML, enables models to adapt rapidly to new tasks with minimal data. However, two primary challenges persist: the tendency for models to be overconfident in their errors (unknown uncertainty) and the computational/labeling inefficiency of random task sampling. Standard approaches often require vast amounts of labeled data, much of which provides redundant information. There is a critical need for a principled method to quantify uncertainty and select the most informative tasks for training, rather than relying on random selection.

Methods

Units-ML addresses these challenges by applying Subjective Logic (SL) to the meta-learning objective. Instead of standard softmax probabilities, the model outputs evidence for each class, which is then mapped to a multidimensional belief distribution:

  • Vacuous Belief ($vb$): Represents uncertainty due to a lack of evidence (e.g., encountering a novel class).
  • Conflicting Belief ($cb$): Represents uncertainty due to contradictory evidence across multiple classes.
  • Incorrect Belief ($ib$): Represents the likelihood of a confident but wrong prediction ("unknown uncertainty").

A central theoretical result of this work is the proof that $ib \geq \frac{1}{2}cb$. This allows the framework to identify potentially incorrect tasks using only unlabeled query sets by measuring conflict.

The Units-ML workflow employs a multi-query task formulation. During training, the model adapts to a support set and evaluates several unlabeled query sets. It then selects the query set with the highest combined vacuous and conflicting uncertainty for labeling and the subsequent global update, creating an automated curriculum for the meta-learner.

Comparison of task selection strategies: No selection, Simple selection, and the proposed Units-ML selection.

Results

The framework was evaluated on standard few-shot learning benchmarks including Omniglot, mini-ImageNet, and CifarFS. Units-ML demonstrates superior label efficiency, reaching high accuracy levels with significantly fewer labeled tasks compared to baselines like MAML and VERSA. For instance, on 20-way 5-shot Omniglot, it achieves over 90% accuracy within 1,000 iterations.

The evolution of belief components during training reveals that the model effectively learns to reduce both vacuity and conflict over time. As shown in the training and validation trends, the "Correct" belief increases while "Vacuous," "Incorrect," and "Conflicting" beliefs diminish, indicating a more reliable and certain model.

Training and validation trends showing the evolution of multidimensional belief components over 100 epochs.

Furthermore, the vacuity measure serves as a robust indicator for out-of-distribution (OOD) detection. By filtering predictions based on a vacuity threshold, the model can significantly boost its accuracy on remaining samples. This reliable confidence estimation makes Units-ML safer for real-world deployment than standard overconfident deep learning models.

Conclusion

Units-ML provides a principled approach to uncertainty quantification in meta-learning. By distinguishing between types of uncertainty and leveraging the relationship between conflict and error, it enables highly efficient active learning. While currently focused on optimization-based approaches, future research will explore extending these belief measures to metric-based meta-learning architectures and optimizing the computational overhead of multi-query evaluation.

Summary updated

Abstract

Optimization-based meta-learning offers a promising direction for few-shot learning that is essential for many real-world computer vision applications. However, learning from few samples introduces uncertainty, and quantifying model confidence for few-shot predictions is essential for many critical domains. Furthermore, few-shot tasks used in meta training are usually sampled randomly from a task distribution for an iterative model update, leading to high labeling costs and computational overhead in meta-training. We propose a novel uncertainty-aware task selection model for label efficient meta-learning. The proposed model formulates a multidimensional belief measure, which can quantify the known uncertainty and lower bound the unknown uncertainty of any given task. Our theoretical result establishes an important relationship between the conflicting belief and the incorrect belief. The theoretical result allows us to estimate the total uncertainty of a task, which provides a principled criterion for task selection. A novel multi-query task formulation is further developed to improve both the computational and labeling efficiency of meta-learning. Experiments conducted over multiple real-world few-shot image classification tasks demonstrate the effectiveness of the proposed model.
BibTeX
@inproceedings{pandey2022multidimensional,
  author = {Deep Pandey and Qi Yu},
  title = {{Multidimensional Belief Quantification for Label-Efficient Meta-Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  url = {https://openreview.net/forum?id=3iBtpUYDTN7s},
}
Figure