Introduction
In modern machine learning, the efficiency of data acquisition is often as critical as model architecture. When operating under constraints such as limited labeling budgets, compute, or bandwidth, selecting the most informative subset of data becomes paramount. Traditional approaches generally fall into two categories: diversity-based methods (e.g., Determinantal Point Processes or DPP), which aim for geometric coverage of the feature space, and uncertainty-based methods (Active Learning), which target samples the current model finds ambiguous.
However, both paradigms have limitations. Diversity-based methods are often "task-blind," failing to prioritize features relevant to specific classification boundaries. Conversely, uncertainty-based methods can be highly biased in early training stages when the model lacks a foundational understanding of the data distribution. This paper introduces RD-DPP, a framework that bridges Information Theory and geometry to implement a "bi-modal" data selection strategy that adapts to the model's learning state.
Methodology: Bridging RD Theory and DPP
The core of RD-DPP lies in the mathematical connection between Rate-Distortion (RD) theory and DPP. The authors demonstrate that maximizing the "coding rate" of a data subset—a concept from signal processing used to measure the minimum bits required for lossy compression—serves as a robust proxy for maximizing geometric diversity.
Semantic Diversity and the Bi-modal Strategy
To make diversity "task-aware," the researchers propose a semantic diversity (sdiv) kernel. This kernel prioritizes samples that maximize inter-class separation while maintaining intra-class variety, effectively performing "sphere packing" in high-dimensional latent spaces.
A key theoretical contribution is the identification of a phase transition in the utility of diversity. Early in training, diversity is essential for mapping the overall data geometry (Regime I). Once the distribution is sufficiently represented, the marginal utility of further diversity diminishes, and the model benefits more from "hard" samples near decision boundaries (Regime II).
RD-DPP utilizes this insight by employing a bi-modal scheduler: it uses diversity-based selection until the phase transition is reached, then automatically switches to uncertainty-based sampling.
Experimental Results
The authors evaluated RD-DPP across various architectures (EfficientNet, ResNet, ResNeXt) and datasets (MNIST, CIFAR10, UCI benchmarks).
Performance and Generalization
RD-DPP consistently outperformed standard active learning and coreset methods like K-Center. On CIFAR10, the method showed significant accuracy gains across all labeling budgets, maintaining a clear lead over random and uncertainty-based baselines.
Beyond raw accuracy, RD-DPP demonstrated superior robustness to task shift. In scenarios where samples selected for one task were deployed to train a model for a different task (label-shift), RD-DPP outperformed uncertainty-based methods by up to 20% in F1 score. This suggests that diversity-based samples capture fundamental data structures that generalize better than samples tailored to a specific model's temporary weaknesses.
Discussion and Future Work
While RD-DPP offers a powerful framework for data selection, it faces computational challenges. The reliance on Singular Value Decomposition (SVD) introduces a complexity of $O(d^3)$, which the authors mitigate via bootstrapping. Additionally, determining the optimal transition point across diverse datasets remains an empirical challenge. Future research will focus on automating this threshold detection and extending the framework to non-Gaussian data distributions and real-world streaming scenarios.
Summary updated