Leakage-Robust Classifier via Mask-Enhanced Training (Student Abstract)

Damian Stachura, Christopher Galias, Konrad ZolnaAAAI 2020 · DOI

Abstract

Machine learning models frequently exploit data leakage—irrelevant signals in a dataset that correlate with target labels—leading to poor generalization in real-world scenarios. We introduce Mask-Enhanced Training (MET), an adversarial framework designed to automatically identify and mask these leakages during the training phase. By hiding the most predictive features, MET forces the classifier to learn robust, meaningful representations. Our experiments on CIFAR-10 and Tiny ImageNet demonstrate that MET effectively recovers performance even when the entire training set is corrupted with synthetic leakage, performing comparably to an "Oracle" masker that has prior knowledge of the leakage location.

Introduction

Deep neural networks are highly effective at finding patterns, but they are often "lazy," prioritizing simple shortcuts over complex, robust features. This phenomenon, known as data leakage, occurs when training data contains information about the target label that will not be available during inference. For instance, a model might learn to identify a specific hospital's watermark instead of a medical condition. Identifying and mitigating such leakage typically requires manual inspection of saliency maps, which is unscalable.

Mask-Enhanced Training (MET)

The proposed MET framework addresses this by framing leakage mitigation as an adversarial game between two components:

  1. The Masker (Saliency Extractor): This component identifies the pixels most critical to the classifier's current prediction and generates a mask to hide them.
  2. The Classifier: A standard architecture (e.g., ResNet18) trained to maintain accuracy even when its most informative features are masked.

Because leaked signals are often the most predictive, the masker naturally targets them first. This pressure forces the classifier to ignore these "easy" shortcuts and instead develop redundant, robust features from the remaining image data.

Experimental Setup

We evaluated MET using CIFAR-10 and Tiny ImageNet. To simulate a worst-case scenario, we "spoiled" every training image by overwriting a few pixels in the top-left corner with a binary representation of the class label.

Illustration of the spoiling and masking process. Left: original image. Center: spoiled image with synthetic leakage. Right: masked image.

A baseline CNN trained on this spoiled data achieves near-perfect training accuracy but fails completely on clean validation data, as it relies entirely on the synthetic leakage.

Results and Discussion

MET-trained models demonstrated significant robustness to the synthetic leakage. On CIFAR-10, the classifier achieved 74.39% accuracy on clean (unmasked) validation images, despite never seeing a clean image during training.

Comparison of MET performance against an Oracle masker on CIFAR-10 and Tiny ImageNet.

As shown in the table above, MET performs nearly as well as an "Oracle" masker—a theoretical upper bound that knows exactly where the leakage is located. This indicates that the adversarial masker successfully "discovered" the location of the synthetic leakage without human intervention. Furthermore, the method is robust to hyperparameters; optimal performance was observed when the mask size was kept small (under 1% of the image), ensuring the model still has enough information to learn the primary task.

Conclusion and Future Work

MET provides an automated way to build classifiers that are robust to data leakage. By forcing the model to learn multiple ways to solve a task, it promotes the development of generalized features. Future research will explore the effectiveness of MET against more subtle, semantic leakages, such as background correlations, and investigate the computational efficiency of the adversarial training process.

Summary updated

BibTeX
@inproceedings{stachura2020leakage,
  author = {Damian Stachura and Christopher Galias and Konrad Zolna},
  title = {{Leakage-Robust Classifier via Mask-Enhanced Training (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i10.7234},
}
Figure