Robust Data Sampling in Machine Learning: A Game-Theoretic Framework for Training and Validation Data Selection

Zhaobin Mo, Xuan Di, Rongye ShiGames 2023 · OpenReview

TL;DR

This paper introduces a novel game-theoretic framework for robustly selecting training and validation data for machine learning models, especially in the face of heterogeneous and skewed datasets. By modeling the data selection process as a two-player game between a "trainer" and a "validator," they achieve a more balanced and representative dataset, leading to improved model generalization and reduced overfitting.

What Problem Does This Paper Address?

Machine learning models thrive on data, but the quality and representativeness of that data are critical. Traditional approaches often randomly sample training and validation sets, which can be problematic when dealing with complex, real-world datasets that are heterogeneous, skewed, or contain "long-tail" phenomena (rare but important events). This can lead to models that overfit to common scenarios and perform poorly on less frequent but crucial situations. The paper highlights that not only the training data but also the validation data's completeness is vital for accurate model evaluation. The core problem is how to strategically select both training and validation data to ensure a model is robust and generalizes well to unseen data, especially when dealing with limited or challenging datasets.

Methods

The authors propose a game-theoretic framework for data sampling. This framework models the selection of training and validation data as a two-player game:

  • Trainer: Aims to select training data that minimizes the model's test error.
  • Validator: Adversarially selects validation data that maximizes the model's test error.

This adversarial process is designed to create a balanced dataset that covers critical scenarios. To efficiently find the optimal data selection strategy, they combine:

  • Monte Carlo Tree Search (MCTS): A search algorithm used to explore the vast space of possible data selections.
  • Reinforcement Learning (RL): Used to train the "players" (trainer and validator) to develop optimal strategies for selecting data. Specifically, they employ Reinforcement-Learning-Aided MCTS (RL-MCTS), where value networks are used to guide the MCTS search, replacing random playouts with learned evaluations.

The game proceeds in turns, with each player selecting or skipping data points. The game's outcome (reward) is based on the model's performance (test MSE) on the selected data, with penalties for selecting too much data to encourage efficiency.

Datasets

The paper uses the Next Generation Simulation (NGSIM) project dataset for their case study. This dataset contains vehicle trajectories from two test sites (I-80 and US101) and includes both free-flow and congestion scenarios, with a particular focus on the challenging "oscillation area" which represents rare but important traffic dynamics. The data was extracted from video recordings.

Key Insights and Ideas

  • Adversarial Validation is Key: The core insight is that a strategically adversarial validation set, designed to maximize test error, is crucial for forcing the training process to learn robust representations. This is a departure from traditional validation, which typically aims to be representative.
  • Game Theory for Data Selection: Framing data selection as a competitive game between a trainer and validator provides a principled way to discover optimal, balanced datasets, especially for long-tail distributions.
  • RL-MCTS for Efficient Exploration: Combining RL with MCTS allows for efficient exploration of the complex data selection space, enabling the discovery of optimal sampling strategies in a computationally feasible manner.
  • Mitigating Overfitting: The proposed method demonstrably reduces overfitting by ensuring that the model is exposed to data that challenges its performance, rather than just data that confirms its existing knowledge.

Key Results

The experiments were conducted on a car-following (CF) model using the NGSIM dataset. The results show:

  • Reduced Overfitting: Models trained with RL-MCTS-sampled data exhibited significantly less overfitting compared to models trained with randomly sampled data. The prediction MSE for the RL-MCTS method converged smoothly, while the random sampling method showed a spike in prediction MSE during training.
  • More Balanced Data Distribution: The velocity distributions of the data sampled by RL-MCTS were more balanced, reflecting a better representation of various driving behaviors, including less frequent ones. This contrasts with random sampling, which tended to overrepresent common velocities.
  • Improved Robustness: Over 100 experimental runs, the RL-MCTS sampled data consistently led to more robust models, as evidenced by lower and more stable test MSE scores and rewards compared to random sampling.

Limitations and Future Work

The authors acknowledge the following:

  • Dataset Size: The algorithm was verified on a relatively small dataset. They suggest applying it to larger datasets with more diverse characteristics.
  • Sequential vs. One-Shot: The current framework treats data selection as a one-shot process. They propose that transforming it into a sequential problem where players can retract moves could lead to suboptimal solutions and suggest addressing this by allowing players to retract false moves.

Future work could involve applying this framework to other domains with long-tail data distributions and exploring more complex game structures.

Summary updated

Abstract

How to sample training/validation data is an important question for machine learning models, especially when the dataset is heterogeneous and skewed. In this paper, we propose a data sampling method that robustly selects training/validation data. We formulate the training/validation data sampling process as a two-player game: a trainer aims to sample training data so as to minimize the test error, while a validator adversarially samples validation data that can increase the test error. Robust sampling is achieved at the game equilibrium. To accelerate the searching process, we adopt reinforcement learning aided Monte Carlo trees search (MCTS). We apply our method to a car-following modeling problem, a complicated scenario with heterogeneous and random human driving behavior. Real-world data, the Next Generation SIMulation (NGSIM), is used to validate this method, and experiment results demonstrate the sampling robustness and thereby the model out-of-sample performance.
BibTeX
@inproceedings{mo2023robust,
  author = {Zhaobin Mo and Xuan Di and Rongye Shi},
  title = {{Robust Data Sampling in Machine Learning: A Game-Theoretic Framework for Training and Validation Data Selection}},
  booktitle = {Games},
  year = {2023},
  url = {https://openreview.net/forum?id=CeS1aDOriBP},
}
Figure