Proximal Validation Protocol

MingFeng Ou, Yiming Zhang, Sai Wu, Gang Chen, Junbo ZhaoICLR 2023 · OpenReview

TL;DR

This paper introduces the Proximal Validation Protocol (PVP), a novel method for constructing validation sets without sacrificing training data. By leveraging data augmentation and a distributional-consistent sampling algorithm, PVP aims to provide reliable model evaluation while maximizing the data available for training, addressing a critical trade-off in real-world ML development.

What Problem Does This Paper Address?

In machine learning, the standard practice involves splitting a labeled dataset into training, validation, and testing sets. However, in many real-world ML applications, especially in production environments, a dedicated testing set is often unavailable until deployment. This makes the validation set crucial for model selection, hyperparameter tuning, and overall evaluation. The core challenge lies in determining the optimal split ratio between training and validation data. A larger validation set leads to more reliable evaluation but reduces the data available for training, potentially hindering model performance. Conversely, a smaller validation set maximizes training data but risks unreliable evaluation. This creates a vexing trade-off between performance enhancement and trustworthy model assessment. The authors argue that this problem is often overlooked in academic research, where pre-defined splits are common, contributing to "technical debt" in ML production.

Methods

The proposed Proximal Validation Protocol (PVP) tackles the train/validation split problem by creating a validation set without using any of the original labeled data for validation itself. The core idea is to keep all original data for training. PVP consists of two main components:

  1. Data Generator (G): This component synthetically generates a large pool of candidate data points, termed the "auxiliary set." The authors primarily use data augmentation techniques for this purpose, as they can generate arbitrarily large datasets and are adaptable across various data modalities.
  2. Distributional-Consistent Sampling Algorithm (A): This algorithm selects the most representative samples from the auxiliary set to form the "proximal validation set." The challenge here is to ensure this synthetic validation set mimics the distribution of the true underlying data. PVP addresses this by first estimating the empirical distribution of the source (original) dataset using intra-class angular distributions on feature space. Then, it samples angles from this estimated distribution and searches for corresponding samples in the auxiliary set that best match these angles. This aims to create a validation set that is distributionally similar to the true data distribution, thereby reducing the test-validation gap.

Datasets

The authors evaluate PVP on three data modalities:

  • Tabular Data: Four publicly available datasets from the UCI Machine Learning Repository (BankMarket, PageBlocks, Diabetes, MushRoom).
  • Image Data: Long-tailed versions of CIFAR10 and CIFAR100 (CIFAR10-LT, CIFAR100-LT), constructed to mimic imbalanced real-world scenarios.
  • Natural Language Processing: The Reuters-21578 dataset, using a specific subset as per prior work.

No new benchmarks were introduced; instead, existing datasets were used to demonstrate PVP's effectiveness across diverse data types and challenges like class imbalance.

Key Insights and Ideas

  • Breaking the Split Trade-off: The most significant insight is that it's possible to have a robust validation set without sacrificing training data. PVP offers a way to achieve both high model performance and reliable evaluation simultaneously.
  • Synthetic Validation is Viable: The paper demonstrates that a validation set constructed entirely from synthetic data (via augmentation) can be effective, challenging the notion that validation must rely on real, held-out data points.
  • Distributional Consistency is Key: The success of PVP hinges on its ability to generate a synthetic validation set that closely mirrors the distribution of the original data. The intra-class angular distribution estimation and sampling strategy are crucial for achieving this.
  • PVP as a "Plug-and-Play" Solution: The authors highlight that PVP is designed to be model-agnostic and adaptable to various data modalities, making it a potentially versatile tool for ML practitioners.

Key Results

PVP consistently outperformed traditional validation methods (holdout, k-fold CV, and a recent method called SPlit) across all tested datasets and modalities.

  • Performance Improvement: PVP achieved superior test scores (F1 score, AUC, Accuracy) compared to baselines, with significant improvements on several datasets. This is attributed to the ability to use all original data for training.
  • Reduced Test-Validation (T-V) Gap: PVP significantly reduced the T-V gap, indicating more accurate and reliable performance estimates. On average, the T-V gap was reduced by 32.3% over the best baselines.
  • Competitive Variance: PVP maintained variance comparable to or better than traditional methods, suggesting stable evaluation results across different random seeds.
  • Smaller Distribution Gap: Quantitative analysis (Wasserstein distance) showed that PVP's validation set had a smaller distribution gap compared to the global distribution of the source set, reinforcing its representativeness.
  • Broader Coverage: Visualizations (t-SNE) demonstrated that PVP's proximal validation set covered a wider region of the data space, including central and marginal areas, compared to split-based methods.

Limitations and Future Work

  • Theoretical Understanding: The authors acknowledge that the theoretical underpinnings of PVP are not fully explored in this work, as the focus was on empirical validation. They suggest that further theoretical study is needed.
  • Data Generator Flexibility: While data augmentation is used, the paper mentions that other data generation methods (e.g., generative modeling, dataset distillation) could also be explored for creating the auxiliary set.
  • Computational Cost: The initial generation of the auxiliary set can be time-consuming, although this is a one-time cost. The authors suggest that engineering efforts like multithreading could mitigate this.
  • Future Work: The authors hope PVP will inspire more research into validation set construction, particularly for ML production scenarios where data is scarce or testing sets are unavailable. They believe this line of research is vital for practical ML deployment.

Summary updated

Abstract

Modern machine learning algorithms are generally built upon a train/validation/test split protocol. In particular, with the absence of accessible testing set in real-world ML development, how to split out a validation set becomes crucial for reliable model evaluation, selection and etc. Concretely, under a randomized splitting setup, the split ratio of the validation set generally acts as a vital meta-parameter; that is, with more data picked and used for validation, it would cost model performance due to the less training data, and vice versa. Unfortunately, this implies a vexing trade-off between performance enhancement against trustful model evaluation. However, to date, the research conducted on this line remains very few. We reason this could be due to a workflow gap between the academic and ML production which we may attribute to a form of technical debt of ML. In this article, we propose a novel scheme --- dubbed Proximal Validation Protocol (PVP) --- which is targeted to resolve this problem of validation set construction. Core to PVP is to assemble a \emph{proximal set} as a substitution for the traditional validation set while avoiding the valuable data wasted by the training procedure. The construction of the proximal validation set is established with dense data augmentation followed by a novel distributional-consistent sampling algorithm. With extensive empirical findings, we prove that PVP works (much) better than all the other existing validation protocols on three data modalities (images, text, and tabular data), demonstrating its feasibility towards ML production.
BibTeX
@inproceedings{ou2023proximal,
  author = {MingFeng Ou and Yiming Zhang and Sai Wu and Gang Chen and Junbo Zhao},
  title = {{Proximal Validation Protocol}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=HlRfoQDDj-V},
}
Figure