Learning Data Manipulation for Augmentation and Weighting

Zhiting Hu, Bowen Tan, Ruslan Salakhutdinov, Tom M. Mitchell, Eric P. Xing

This paper offers an elegant mathematical bridge between how we train models and how we prepare the data they learn from.

TL;DR

The authors propose a unified framework that treats data manipulation—such as augmentation and sample weighting—as a "reward learning" problem borrowed from reinforcement learning. By parameterizing the data itself as a reward function, they create a single gradient-based algorithm that automatically learns the best way to transform or prioritize training examples, significantly boosting performance in low-data and imbalanced scenarios.

What Problem Does This Paper Address?

In modern machine learning, the quality of your data often matters more than the complexity of your architecture. To deal with small datasets or "messy" data (like class imbalances), researchers use data manipulation:

  1. Data Augmentation: Creating new examples (e.g., flipping an image or replacing words with synonyms).
  2. Data Weighting: Deciding which examples are more important to focus on during training.

The problem is that most manipulation techniques are either manual heuristics (like "always flip the image") or specialized algorithms designed for only one task. There hasn't been a general-purpose way to learn how to manipulate data across different domains using a single, consistent algorithm.

Methods

The core insight of this paper is a mathematical connection: Supervised learning is a special case of Reinforcement Learning (RL).

In RL, an agent learns a policy to maximize a reward. In supervised learning, the "policy" is the model, and the "reward" is simply matching the training data. The authors show that standard training is equivalent to a "delta-function reward"—you get a reward of 1 if you match the data exactly, and negative infinity otherwise.

By "relaxing" this reward, they introduce a parameterized data reward ($\phi$):

Bi-level optimization loop
  • The Algorithm: It uses a bi-level optimization loop.
  • Inner Loop: The target model ($\theta$) is updated to maximize the current "data reward" (the manipulated data).
  • Outer Loop: The manipulation parameters ($\phi$) are updated based on how well the resulting model performs on a held-out validation set.
  • Instantiations:
    • Text Augmentation: They use a BERT-based language model as the "reward." Instead of fixed synonyms, the system learns to fine-tune this BERT model to generate substitutions that specifically help the classifier improve.
  • Data Weighting: They assign a learnable weight to every training instance. The algorithm learns to "turn up the volume" on difficult or rare examples that help the model generalize better to the validation set.

Datasets

The authors evaluated their approach on both text and image classification tasks, specifically focusing on "low-data" regimes (only 40 training examples per class):

  • Text: SST-5 (5-class sentiment), IMDB (binary sentiment), and TREC (6-class question types).
  • Images: CIFAR-10 (subsampled to create low-data and imbalanced versions).
  • Imbalance Tests: Binary versions of SST-2 and CIFAR-10 where one class significantly outnumbered the other (e.g., a 20:1000 ratio).

Key Insights and Ideas

  • Unified Framework: The most powerful idea is that augmentation, weighting, and even data synthesis are just different ways of parameterizing the same "data reward" function.
  • Dynamic Evolution: Unlike traditional augmentation (which is static), their fine-tuned augmentation model evolves alongside the classifier. As the classifier gets smarter, the augmentation model learns to provide more challenging or relevant examples.
  • Parametric Weighting vs. Meta-Learning: Previous "meta-learning" weighting methods often re-estimate weights from scratch at every step. This paper treats weights as persistent parameters ($\phi$) that are updated iteratively, leading to smoother training and better historical "memory" of which samples are useful.
  • Task-Specific Strengths: The authors observed that augmentation is superior for very small datasets, while weighting is more effective at solving class imbalance problems.

Key Results

The method consistently outperformed strong baselines, even when using powerful pretrained models like BERT and ResNet-34.

Text classification results
  • Text Classification: On the SST-5 dataset (with only 40 examples per class), their fine-tuned augmentation reached 37.03% accuracy, compared to 33.32% for a standard BERT model and 34.84% for fixed augmentation.
  • Class Imbalance: In a highly imbalanced CIFAR-10 scenario (20 vs 1000 examples), their weighting method achieved 75.32% accuracy, outperforming the base model (72.20%) and the previous state-of-the-art weighting method (74.35%).
  • Qualitative Improvement: They showed that their augmentation model learned to move from simple word replacements to more "semantically coherent" substitutions as training progressed.

Limitations and Future Work

  • The Validation Set Requirement: The method relies on a small, clean validation set to guide the learning of $\phi$. If the validation set is too small or biased, the data manipulation might overfit to those specific examples.
  • Computational Cost: Because this is a bi-level optimization (training a model to train a model), it is computationally more expensive than standard training.
  • Data Synthesis: While the paper mentions data synthesis (generating entirely new artificial examples) as a potential use case, they did not implement or test it in this specific work, leaving it for future exploration.

Summary updated

Abstract

Manipulating data, such as weighting data examples or augmenting with new instances, has been increasingly used to improve model training. Previous work has studied various rule- or learning-based approaches designed for specific types of data manipulation. In this work, we propose a new method that supports learning different manipulation schemes with the same gradient-based algorithm. Our approach builds upon a recent connection of supervised learning and reinforcement learning (RL), and adapts an off-the-shelf reward learning algorithm from RL for joint data manipulation learning and model training. Different parameterization of the "data reward" function instantiates different manipulation schemes. We showcase data augmentation that learns a text transformation network, and data weighting that dynamically adapts the data sample importance. Experiments show the resulting algorithms significantly improve the image and text classification performance in low data regime and class-imbalance problems.
BibTeX
@inproceedings{hu2019learning,
  author = {Zhiting Hu and Bowen Tan and Ruslan Salakhutdinov and Tom M. Mitchell and Eric P. Xing},
  title = {{Learning Data Manipulation for Augmentation and Weighting}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure