Contingency Training

Danilo Vasconcellos Vargas, Hirotaka Takano, Junichi MurataarXiv 2018 · arXiv

Introduction

In modern machine learning, the "curse of dimensionality" remains a persistent hurdle. Real-world datasets often arrive with hundreds or thousands of features, many of which are irrelevant noise. While feature selection techniques aim to prune this data, they frequently err on the side of caution, leaving behind "junk" variables that can degrade model performance. Contingency Training (CT) offers a novel solution: rather than attempting to perfectly clean the data beforehand, it "hardens" the classifier during training to prioritize signal over noise.

The Challenge of Irrelevant Variables

Machine learning models, particularly high-capacity ones like neural networks, are prone to overfitting on irrelevant features if they appear to provide a shortcut to minimizing training error. Traditional feature selection is often a binary "all-or-nothing" process. If a selection algorithm fails to remove a noisy variable, the classifier is left to deal with it. CT shifts this paradigm toward implicit feature weighting, forcing the model to learn which attributes are truly reliable across various subsets of available information.

Methodology: Contingency Training

The core of Contingency Training is a simple, classifier-agnostic data augmentation process. It can be applied to any supervised learning algorithm—from decision trees to Multi-Layer Perceptrons (MLPs)—without modifying the underlying architecture.

The Contingency Training process, illustrating value removal and the subsequent change in data representation with a binary mask.

The CT process involves two primary steps:

  1. Data Perturbation: For each training sample, feature values are randomly "hidden" (replaced with zero) based on a uniform probability ($prob$).
  2. Masking for Representation: To prevent the model from confusing a "missing" value with a "true zero," the original feature vector is concatenated with a binary mask. A mask bit of 1 indicates the feature is present, while 0 indicates it was removed.

By presenting the model with these "broken" samples, CT introduces a set of constraints. The model must learn to predict the correct label even when critical features are missing. This forces the learning algorithm to assign higher importance to features that consistently provide signal, effectively performing continuous feature weighting.

Experimental Evaluation

The researchers tested CT across several benchmarks, including the UCI Machine Learning Repository (Wine, Zoo, Glass, Iris, and Pima Indians Diabetes) and the Az-5000 character recognition dataset. To specifically test robustness, they created "noisy" versions of the Iris and Diabetes datasets by adding 20 irrelevant random variables.

Results and Discussion

The results demonstrate that Contingency Training consistently outperforms standard training methods, particularly in the presence of noise.

Accuracy comparison between standard ("Usual") training and Contingency Training across the Az-5000, Iris, and Diabetes datasets.

Key performance highlights include:

  • Noisy Data Robustness: On datasets with 20 added irrelevant variables, CT improved median accuracy by 5.2% to 7.7%.
  • Large-Scale Performance: In the Az-5000 character recognition task, accuracy saw a significant jump of 6.5%.
  • Generalization: CT did not penalize performance on "clean" datasets, showing results comparable to or slightly better than standard training.

One of the most surprising findings was that models trained entirely on degraded samples often outperformed those trained on the original clean data. This suggests that the increased difficulty of the augmented training set acts as a powerful regularizer, preventing the model from relying on brittle, non-generalizable features.

Conclusion and Future Work

Contingency Training provides a robust, easy-to-implement framework for improving classifier performance in high-dimensional, noisy environments. While this study focused on MLPs, the method's classifier-agnostic nature suggests broad applicability. Future research could explore alternative missing-value representations or the integration of CT into ensemble methods to further enhance model stability and accuracy.

Summary updated

BibTeX
@misc{vargas2018contingency,
  author = {Danilo Vasconcellos Vargas and Hirotaka Takano and Junichi Murata},
  title = {{Contingency Training}},
  year = {2018},
  eprint = {1811.08214},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.08214},
}
Figure