Improving Generalization via Meta-Learning on Hard Samples

Nishant Jain, Arun Suggala, Pradeep Shenoy

Abstract

Standard Learned Reweighting (LRW) improves model generalization by weighting training samples to perform well on a representative validation set. This paper demonstrates that intentionally selecting hard-to-classify samples for the validation set leads to significantly better generalization and larger decision margins. We introduce MOLERE (Meta-Optimized LEarned REweighting) to optimize validation data selection and show consistent performance gains across a wide range of benchmarks, including ImageNet-1K and various out-of-distribution tasks.

Introduction

Deep learning models are prone to overfitting the training distribution. Learned Reweighting (LRW) has emerged as a powerful meta-learning technique where a "reweighter" assigns importance to training instances based on their utility in minimizing loss on a separate validation set. However, the choice of the validation set itself is a critical hyperparameter that is rarely optimized. Most existing methods assume a random, clean split. This work addresses the question: Can we optimize the selection of validation data to maximize the final model's generalization?

Methodology

The Tri-Level Objective

The authors formalize validation set selection as a nested optimization problem:

  1. Level 1: Identify an optimal data split (Training vs. Validation).
  2. Level 2: Learn training weights (the reweighter) to minimize loss on the selected validation set.
  3. Level 3: Train the final classifier using these learned weights.

MOLERE and LRWOpt

To make this tri-level problem tractable, the authors propose MOLERE (Meta-Optimized LEarned REweighting). Its core algorithm, LRWOpt, collapses the problem into a bi-level min-max game using two auxiliary networks: a "splitter" that identifies hard samples for the validation set and a "reweighter" that learns to minimize loss on those specific samples.

Train-Twice Heuristic

For practical applications, a simpler "train-twice" heuristic is introduced. A standard model is trained first to calculate the probabilistic margin (the difference between the correct class probability and the highest incorrect one). The samples with the lowest margins (the hardest samples) are then used as the validation set for a second, meta-weighted training run.

Key Insights: The Power of Hard Samples

The most significant finding is a consistent performance hierarchy: LRW-Hard > LRW-Random > LRW-Easy. Intuitively, forcing the model to perform well on "borderline" cases during the meta-learning phase makes the final classifier more robust.

Comparison of LRW variants (left) and LRWOpt against baselines (right) on standard classification benchmarks.
  • Margin Maximization: Using hard validation samples acts as a margin maximizer. By focusing the meta-loss on samples near the decision boundary, the model learns a boundary that is further from the data points, enhancing test-time performance.
  • Theoretical Connection to DRO: The authors prove that their objective is asymptotically the dual of Distributionally Robust Optimization (DRO). While DRO targets the "worst-case" distribution, MOLERE achieves robustness by optimizing for the hardest samples in the existing dataset.

Experimental Results

The authors evaluated MOLERE across a massive battery of benchmarks, including general classification (ImageNet-1K), fine-grained tasks (Stanford Cars), and noisy label datasets (Clothing-1M).

  • ImageNet-1K: Achieved a ~1% accuracy gain using a ViT-B backbone, a significant improvement on a highly optimized benchmark.
  • Noisy Labels: On Clothing-1M, MOLERE outperformed specialized baselines, gaining 4.2% over standard Empirical Risk Minimization (ERM).
  • Out-of-Distribution (OOD): The method proved highly effective under domain shift. On the iWildCam dataset, it provided a 1.36% gain, suggesting that hard validation sets help the model ignore spurious correlations.
Performance on out-of-distribution (OOD) benchmarks, comparing LRW variants (left) and LRWOpt against baselines (right).

Limitations and Future Work

While effective, the LRWOpt algorithm increases computational complexity due to the training of auxiliary networks. Additionally, the "train-twice" heuristic depends on the quality of the initial ERM model; if the first model is poorly trained, it may misidentify outliers or mislabeled data as "hard" samples. Future research will explore deeper theoretical links to DRO and the application of meta-optimization to other stages of the machine learning pipeline.

Conclusion

This work demonstrates that the composition of the validation set is a powerful lever for improving generalization. By intentionally meta-learning on hard samples, models achieve better margins and robustness across diverse and challenging data distributions.

Summary updated

Abstract

Learned reweighting (LRW) approaches to supervised learning use an optimization criterion to assign weights for training instances, in order to maximize performance on a representative validation dataset. We pose and formalize the problem of optimized selection of the validation set used in LRW training, to improve classifier generalization. In particular, we show that using hard-to-classify instances in the validation set has both a theoretical connection to, and strong empirical evidence of generalization. We provide an efficient algorithm for training this meta-optimized model, as well as a simple train-twice heuristic for careful comparative study. We demonstrate that LRW with easy validation data performs consistently worse than LRW with hard validation data, establishing the validity of our meta-optimization problem. Our proposed algorithm outperforms a wide range of baselines on a range of datasets and domain shift challenges (Imagenet-1K, CIFAR-100, Clothing-1M, CAMELYON, WILDS, etc.), with ~1% gains using VIT-B on Imagenet. We also show that using naturally hard examples for validation (Imagenet-R / Imagenet-A) in LRW training for Imagenet improves performance on both clean and naturally hard test instances by 1-2%. Secondary analyses show that using hard validation data in an LRW framework improves margins on test data, hinting at the mechanism underlying our empirical gains. We believe this work opens up new research directions for the meta-optimization of meta-learning in a supervised learning context
BibTeX
@misc{jain2024improving,
  author = {Nishant Jain and Arun Suggala and Pradeep Shenoy},
  title = {{Improving Generalization via Meta-Learning on Hard Samples}},
  year = {2024},
  eprint = {2403.12236},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.12236},
}
Figure