The Power of Few: Accelerating and Enhancing Data Reweighting with Coreset Selection

Mohammad Jafari, Yimeng Zhang, Yihua Zhang, Sijia Liu

Abstract

Modern machine learning faces a significant challenge in balancing computational efficiency with model performance. While data reweighting improves generalization by emphasizing critical samples, its computational cost on large datasets is often prohibitive. Conversely, coreset selection reduces training time but frequently sacrifices the nuanced diversity required for peak accuracy. This paper introduces CW-ERM (Coreset Reweighting for Empirical Risk Minimization), a novel framework that synergizes these two approaches. By performing reweighting on a representative coreset and broadcasting those weights to the full dataset, CW-ERM achieves state-of-the-art results with significantly reduced overhead.

Introduction

As datasets and model architectures continue to scale, the computational burden of training has reached a sustainability crisis. Researchers typically employ one of two strategies to manage this: coreset selection or data reweighting. Coreset selection identifies a small subset of data for training, which accelerates the process but often leads to a drop in final accuracy. Data reweighting (e.g., MetaWeightNet) assigns importance values to individual samples to improve robustness, but calculating these weights for millions of instances is computationally grueling.

CW-ERM addresses this gap by asking: Can we use the efficiency of coresets to make the precision of data reweighting scalable? The authors demonstrate that calculating weights on a tiny fraction of the data (less than 1%) and extending those weights to the rest of the dataset provides a rare "win-win" for both speed and accuracy.

Methodology: The CW-ERM Framework

The CW-ERM pipeline consists of three distinct stages designed to optimize the training process:

1. Moderate Coreset Selection

Rather than selecting random samples or focusing solely on "hard" examples, the authors utilize a "moderate coreset" strategy. Using a pre-trained model (such as a Vision Transformer) to extract features, they calculate the median feature vector for each class. Samples closest to these medians are selected. This approach effectively summarizes the dataset's core distribution while filtering out extreme outliers that might otherwise skew the reweighting process.

2. Coreset Reweighting

In this stage, a sophisticated meta-learning algorithm, such as MetaWeightNet, is applied exclusively to the selected coreset. Because the coreset is extremely small—often representing less than 1% of the total data—this step, which is usually the primary bottleneck in reweighting schemes, becomes exceptionally fast.

3. Weight Broadcasting

The final stage involves "broadcasting" the learned weights to the entire dataset. Using a Nearest Neighbors approach, every sample in the full dataset inherits the weight of its closest neighbor within the coreset. The model is then trained on the full dataset using these inherited weights, ensuring that the entire data distribution is utilized while benefiting from the precision of the meta-learned importance values.

Experimental Results

The researchers evaluated CW-ERM on standard computer vision benchmarks, including CIFAR-10 and CIFAR-100, using architectures such as PreactResNet18 and ResNet20.

Accuracy and Generalization

On CIFAR-10, CW-ERM achieved an accuracy of 94.9%, outperforming both standard training (94.1%) and full-dataset reweighting (93.8%). The lower performance of full-dataset reweighting in some cases suggests that calculating weights on the entire dataset can sometimes lead to overfitting on noise—a problem mitigated by the moderate coreset selection in CW-ERM.

Computational Efficiency

The speed gains were most evident on the CIFAR-100 dataset. Standard data reweighting (W-ERM) required over 300 minutes to complete. In contrast, CW-ERM achieved superior results in just 57 minutes, representing a roughly 5x speedup in the reweighting phase.

Key Insights

  • The "1% Rule": The study reveals that a coreset representing less than 1% of the total data is sufficient to learn high-quality weights that generalize to the remaining 99%.
  • Algorithmic Synergy: Coreset selection and reweighting are complementary. The coreset provides the efficiency needed for reweighting, while reweighting provides the precision that standard coreset training lacks.
  • Outlier Mitigation: By focusing on samples near the class median, the method naturally avoids assigning high importance to noisy outliers, which often plague standard reweighting algorithms.

Conclusion and Future Work

CW-ERM offers a compelling blueprint for making data-centric AI techniques practical for resource-constrained environments. While currently validated on image classification tasks, the framework's reliance on feature extraction suggests potential applicability in Natural Language Processing (NLP) and other domains. Future research will focus on scaling this method to "web-scale" datasets like ImageNet-21k and exploring its performance in non-vision modalities. Ultimately, CW-ERM proves that in the quest for better models, there is immense power in focusing on the right few.

Summary updated

BibTeX
@inproceedings{jafari2024power,
  author = {Mohammad Jafari and Yimeng Zhang and Yihua Zhang and Sijia Liu},
  title = {{The Power of Few: Accelerating and Enhancing Data Reweighting with Coreset Selection}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=o8x3rGm6No},
}
Figure