Deep Active Learning with Manifold-preserving Trajectory Sampling

Yingrui Ji, Vijaya Sindhoori Kaza, Nishanth Artham, Tianyang Wang

TL;DR

Manifold-Preserving Trajectory Sampling (MPTS) addresses a fundamental flaw in active learning: the "vicious cycle" where repeatedly selecting only the most uncertain examples biases the model’s perception of the data distribution. By using Maximum Mean Discrepancy (MMD) to align labeled features with the broader unlabeled pool and sampling model parameters directly from the training trajectory, MPTS achieves state-of-the-art performance across vision and tabular datasets without requiring complex auxiliary models.

What Problem Does This Paper Address?

Active Learning (AL) aims to build high-performing models with minimal labeled data by strategically choosing which samples to annotate. However, the authors identify two major "bottlenecks" in current deep AL methods:

  1. Data Bias: Most AL methods select "hard" or "uncertain" examples. Over multiple cycles, the labeled training set becomes a skewed, non-representative slice of the actual data distribution. This makes the model's estimate of the posterior distribution $P(\theta|D)$—and thus its uncertainty—increasingly unreliable.
  2. Rigid Assumptions: Existing Bayesian approaches often rely on explicit (and sometimes inaccurate) assumptions, like assuming the posterior follows a Gaussian distribution or requiring specific architectural changes (e.g., adding Dropout layers).

The core challenge is: How can we keep the model grounded in the "true" data manifold while still focusing on the most informative samples?

Methods

The authors propose a two-part solution called Manifold-Preserving Trajectory Sampling (MPTS):

1. Manifold-Preserving Regularization

To fix the data bias, the authors argue that the feature space ($Z$) learned from the small labeled set should still resemble the feature space of the entire dataset (labeled + unlabeled).

  • The Technique: They use Maximum Mean Discrepancy (MMD), a kernel-based method to measure the distance between two distributions.
  • The Loss: During training, the model minimizes a joint loss: standard Cross-Entropy on labeled data plus an MMD penalty that forces the distribution of labeled features to match the distribution of unlabeled features. This ensures the model "remembers" the global structure of the data.

2. Sampling from Optimization Trajectory

Instead of assuming a mathematical shape for the model's parameter uncertainty, they look at the training path itself.

  • The Technique: Inspired by Stochastic Weight Averaging (SWA), they use a cyclic learning rate. Once the model begins to converge, they save multiple "checkpoints" (parameter sets) along the optimization trajectory near local minima.
  • Uncertainty Estimation: To decide which data to label next, they pass unlabeled samples through this "committee" of saved checkpoints. The variance/entropy across these different versions of the model provides a robust, unbiased measure of uncertainty.

Datasets

The authors evaluated MPTS on a diverse range of benchmarks to prove its generalizability:

  • Vision Benchmarks: MNIST (digits), CIFAR-10 (objects), SVHN (street numbers), and Mini-ImageNet (complex objects).
  • Non-Vision (Tabular) Benchmarks: OpenML-6 and OpenML-155. These are structured datasets with mixed feature types, often used to test if a method works beyond just pixels.

Key Insights and Ideas

  • Correcting the Posterior: This is the first paper to explicitly propose data bias correction for estimating the posterior distribution in AL. By regularizing the feature space, they prevent the model from "tunnel visioning" on just the hard examples.
  • Model Agnostic: Unlike methods that require specific architectures (like GCNs or Variational Autoencoders), MPTS is a plug-and-play loss function and sampling strategy. It worked effectively across MLPs, ResNet-18, and Vision Transformers (ViT).
  • The "Free" Ensemble: Trajectory sampling effectively gives you the benefits of an ensemble (multiple viewpoints) without the massive computational cost of training several independent models from scratch.

Key Results

  • Superior Accuracy: MPTS consistently outperformed baselines like BALD, CoreSet, and BADGE.
  • Complex Scenes: The performance gap was most noticeable on complex datasets like CIFAR-10 and SVHN, suggesting that manifold preservation is especially vital when the data structure is intricate.
  • Tabular Success: Surprisingly, MPTS showed a significant lead on non-vision data (OpenML). The authors noted that standard MLPs are highly susceptible to data bias, making the manifold-preserving MMD term particularly effective here.
  • Efficiency: Even with a very small labeling budget (e.g., only 100 samples per round), MPTS maintained a steady performance lead, demonstrating high "data efficiency."

Limitations and Future Work

  • Hyperparameter Sensitivity: The method introduces a weighting parameter ($\lambda$) for the MMD loss. The paper doesn't deeply explore how sensitive the results are to different values of $\lambda$ across different domains.
  • Computational Overhead: While more efficient than training an ensemble, calculating MMD on large batches and maintaining multiple checkpoints adds some memory and compute overhead compared to simple entropy-based sampling.
  • Scalability to Massive Datasets: While tested on Mini-ImageNet, the performance on "Industrial-scale" datasets (like full ImageNet or massive web-scale tabular data) remains an open question for future research.

Summary updated

Abstract

Active learning (AL) is for optimizing the selection of unlabeled data for annotation (labeling), aiming to enhance model performance while minimizing labeling effort. The key question in AL is which unlabeled data should be selected for annotation. Existing deep AL methods arguably suffer from bias incurred by clabeled data, which takes a much lower percentage than unlabeled data in AL context. We observe that such an issue is severe in different types of data, such as vision and non-vision data. To address this issue, we propose a novel method, namely Manifold-Preserving Trajectory Sampling (MPTS), aiming to enforce the feature space learned from labeled data to represent a more accurate manifold. By doing so, we expect to effectively correct the bias incurred by labeled data, which can cause a biased selection of unlabeled data. Despite its focus on manifold, the proposed method can be conveniently implemented by performing distribution mapping with MMD (Maximum Mean Discrepancies). Extensive experiments on various vision and non-vision benchmark datasets demonstrate the superiority of our method. Our source code can be found here.
BibTeX
@misc{ji2024deep,
  author = {Yingrui Ji and Vijaya Sindhoori Kaza and Nishanth Artham and Tianyang Wang},
  title = {{Deep Active Learning with Manifold-preserving Trajectory Sampling}},
  year = {2024},
  eprint = {2410.15605},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.15605},
}
Figure