Reinforced Training Data Selection for Domain Adaptation

Miaofeng Liu, Yan Song, Hongbin Zou, Tong ZhangACL 2019 · DOI

Introduction

Supervised machine learning models frequently encounter "domain shift," where the distribution of training data (source domain) differs significantly from real-world application data (target domain). For instance, a sentiment analysis model trained on formal movie reviews may struggle with the informal language of social media.

Traditional approaches to mitigate this involve Training Data Selection (TDS)—filtering the source domain for examples that resemble the target. However, standard TDS methods suffer from two primary limitations: they rely on manual, often arbitrary selection thresholds (e.g., "top 10%") and are typically static preprocessing steps that lack a feedback loop from the actual task performance. This paper introduces a reinforcement learning (RL) framework to automate and optimize this selection process dynamically.

The Reinforced TDS Framework

The proposed framework treats data selection as a sequential decision-making problem, jointly optimizing selection and representation learning. The system is composed of two primary modules:

  1. Selection Distribution Generator (SDG): Acting as the RL agent, the SDG evaluates "bags" of source data and learns a policy to select the most relevant instances.
  2. The Predictor: This module contains a Feature Extractor and a Classifier. It performs the target task using the data selected by the SDG.
Architecture of the Reinforced Training Data Selection framework, showing the interaction between the RL agent (SDG) and the Predictor.

The Reward Mechanism

The RL agent is guided by a reward function based on Distribution Discrepancy. It calculates the distance between the mathematical representations of the selected source data and a small "guidance set" of unlabeled target data. The authors explored several metrics for this reward, including Jensen-Shannon (JS) Divergence, Maximum Mean Discrepancy (MMD), and Rényi Divergence. Through policy gradient methods, the SDG learns to minimize this discrepancy while the Predictor simultaneously adapts its features to the selected subset.

Experimental Evaluation

The framework was tested across three NLP tasks using the SANCL 2012 benchmark (for POS tagging and dependency parsing) and a Product Review dataset (for sentiment analysis). The experiments utilized a multi-source setup, where one domain served as the target and all others formed the source pool.

Key Results

  • Superior Accuracy: In POS tagging for the Weblogs domain, the SDG (using MMD) achieved 96.95% accuracy, outperforming the "ALL" baseline (95.79%) which utilized the entire source dataset.
  • Competitive Sentiment Analysis: On the Kitchen appliances domain, the SDG (using Rényi divergence) reached 86.18% accuracy, surpassing specialized domain adaptation algorithms like SDAMS-SVM.
  • Efficiency: For complex tasks like dependency parsing, the model maintained competitive performance while using only approximately 50% of the available source data.

Insights and Analysis

The "Less is More" Paradox

A significant finding is that training on a carefully selected 40-50% of the data often yields better results than using 100% of the source pool. This suggests that irrelevant or "noisy" source data can cause negative transfer, actively harming model performance on the target domain.

Representation Alignment

Visualizing the feature space using t-SNE confirms that the framework does more than just filter data; it aligns representations. After training, selected source data points move significantly closer to target data points in the vector space compared to baseline methods.

t-SNE visualizations showing how different selection methods, particularly SDG (Rényi), align source and target representations in the feature space.

Conclusion and Future Work

By eliminating manual thresholds and introducing a dynamic feedback loop, the Reinforced TDS framework provides a robust solution for domain adaptation. While the method shows immense promise, the authors note that in highly complex structured prediction tasks, the sheer volume of data can sometimes outweigh the benefits of selection. Future research will focus on more sophisticated reward functions and applying the framework to extreme low-resource scenarios where target data is nearly non-existent.

Summary updated

BibTeX
@inproceedings{liu2019reinforced,
  author = {Miaofeng Liu and Yan Song and Hongbin Zou and Tong Zhang},
  title = {{Reinforced Training Data Selection for Domain Adaptation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1189},
}
Figure