Machine learning models are notoriously sensitive to the quality of their training data. In an era where datasets are increasingly scaled using automated pipelines—such as distant supervision, web scraping, and generative models—training sets are frequently riddled with noise, false positives, and irrelevant samples.
Traditional solutions to this problem rely on static heuristics, such as filtering out data points below a certain model confidence threshold. However, these heuristics are brittle, require manual tuning, and fail to account for how data utility changes as a model learns.
Reinforcement learning (RL) based instance selection reframe data curation. Instead of treating data filtering as a static preprocessing step or a fixed architectural constraint, it treats data selection as a sequential decision-making process. By training an RL agent to act as a gatekeeper, the system learns an optimal data-filtering policy guided directly by the downstream model’s actual performance.
The Mechanics of the Selection Loop
To understand why this approach works, consider how a data selection policy is optimized. The process is modeled as a Markov Decision Process (MDP) operating alongside a primary target model:
- State: The agent observes a representation of the current data instance (e.g., text embeddings, image features, or prediction confidence) combined with the current state of the target model.
- Action: The agent makes a discrete or continuous decision. In a binary formulation, the action is either $1$ (retain the sample for training) or $0$ (discard it).
- Reward: The target model is updated using the selected data. The agent then receives a reward proportional to the target model's performance gains (such as accuracy increase or loss reduction) evaluated on a small, clean, gold-standard validation set.
┌────────────────────────────────────────────────────────┐
│ │
▼ │
[Data Stream] ──► [RL Selection Agent] ──► [Selected Data] │ (Reward: Validation Gain)
│ │ │
▼ ▼ │
(Action: Keep/Drop) [Target Model] ──┘
By tying the reward directly to validation performance, the selection agent learns to identify which samples genuinely aid generalization. For instance, Yang et al. utilized this mechanism to denoise distantly supervised text for Named Entity Recognition (NER), training a selector to maximize performance on a clean validation set without relying on manual labels for the noisy training text.
Similarly, in computer vision, Chen et al. applied this loop to human segmentation when transitioning across domains (e.g., RGB to Infrared). Their policy learned to filter out noisy, automatically generated optical flow masks by receiving rewards tied directly to the target model's segmentation accuracy gains.
Why Hard Selection Beats Soft Attention
A common alternative for handling noisy data is the use of attention mechanisms or soft weighting, which allow a model to down-weight suspicious samples during backpropagation. Why go through the complexity of training an RL agent when one could use attention?
The answer lies in how gradients handle pure noise. Attention mechanisms are forced to distribute a continuous budget of weights across a batch or sequence. If an entire batch consists of irrelevant or corrupted data, attention must still assign weights to those samples, leaking noise into the model's gradients.
An RL agent, by contrast, can execute hard binary selection. If a batch contains nothing but noise, the policy can choose an action of $0$ for every single instance.
As Feng et al. demonstrated in relation classification, this binary gating mechanism provides a fundamental robustness advantage over aggregate models. Their agent evaluates each instance based on its specific features and context, completely isolating the target model from false positives generated by automated labeling.
Zeng et al. leveraged a similar principle for large-scale relation extraction, showing that treating sentence selection as discrete agent actions allows a system to autonomously discover informative sentences while assigning zero probability to sentences that do not support the target relationship.
Preventing Negative Transfer
Data selection is not just about removing random noise; it is also highly effective at preventing negative transfer—a phenomenon where a model degrades because it learns patterns from a source domain that conflict with the target domain.
In partial domain adaptation, for example, the source dataset often contains outlier classes that do not exist in the target domain. Training on these outlier classes corrupts the model's target representations.
To solve this, Hua introduced an actor-critic reinforcement learning framework (RTNet) that acts as a binary selector to filter source domain samples before they are used for adaptation. By using image reconstruction errors as a reward signal, the agent learns to block outlier classes entirely.
Wu et al. similarly framed this as a plug-and-play sequential decision task, demonstrating that an RL gatekeeper effectively isolates the target model from irrelevant source classes, stabilizing the alignment of domain-invariant features.
Curation Across Diverse Paradigms
Because the RL selection framework treats the target model as a black box, it generalizes across highly diverse machine learning paradigms:
- Synthetic Data Curation: When training models on automatically generated tasks, Pei et al. implemented an RL selector to filter synthetic data. The selector receives a reward signal tied to the learner's actual performance improvements on a validation set, preventing noisy or low-quality generated tasks from degrading model accuracy.
- Ensemble Learning: For imbalanced datasets, Liu et al. developed a meta-sampler framework (MESA). Instead of relying on fixed down-sampling heuristics, they framed sample selection as a sequential decision problem optimized via RL, adaptively determining which data points are most beneficial for the next model in an ensemble sequence.
- Alignment Buffers (RLHF): In Reinforcement Learning from Human Feedback, reward models can be highly volatile in the middle of their distribution. Zhang et al. demonstrated that filtering training buffers to include only samples where the reward model is highly reliable (at the extreme ends of the distribution) prevents policy networks from optimizing for inaccurate, noisy reward signals.
Decoupling for Training Stability
Co-training an RL agent alongside a primary model can introduce optimization instability, as the agent's environment (the target model's state) changes at every training step.
To mitigate this, successful architectures often decouple the selection policy from the target model's internal optimization loop. The authors of a reinforced denoising study found that using reinforcement learning as a discrete data-filtering preprocessing step, rather than training it jointly in an end-to-end fashion, significantly improves training stability.
By separating the selection policy's updates from the primary model's gradient steps, the agent can cleanly isolate and remove low-quality, automatically labeled samples before they ever corrupt the downstream model's representations.
Key Foundational Work
Reinforcement Learning for Relation Classification From Noisy Data (2018) — contributed 4 concept elements to this direction.
Large Scaled Relation Extraction With Reinforcement Learning (2018) — contributed 3 concept elements to this direction.
Reinforcement Learning for Relation Classification from Noisy Data (2018) — contributed 3 concept elements to this direction.