Abstract
Data augmentation techniques like Mixup have revolutionized deep learning performance, yet their application to regression tasks remains challenging, particularly in the presence of label noise. We present RC-Mixup (Robust C-Mixup), a data-centric framework that integrates selective data mixing with iterative robust training. By alternating between noise identification and selective augmentation, RC-Mixup creates a "virtuous cycle" where cleaner data enables more effective augmentation, which in turn improves the model's ability to identify noise. Our experiments across synthetic and real-world datasets demonstrate that RC-Mixup significantly outperforms existing robust training and augmentation methods.
1. Introduction
Deep learning models are notoriously data-hungry. Data augmentation, specifically the "Mixup" strategy, has become a standard approach to improve generalization by creating synthetic training samples through the interpolation of existing data points. While highly effective for classification, standard Mixup often struggles with regression tasks where the continuous nature of labels makes arbitrary interpolation risky.
The recently proposed C-Mixup addresses this by only mixing samples with similar labels. However, C-Mixup is highly sensitive to label noise—a common occurrence in real-world datasets due to sensor errors or human labeling mistakes. When noise is present, C-Mixup inadvertently propagates errors into the synthetic data. Conversely, traditional robust training methods focus on removing or down-weighting noisy samples, which appears to be at odds with the goal of data augmentation. RC-Mixup bridges this gap.
2. The RC-Mixup Framework
RC-Mixup (Robust C-Mixup) is a three-step iterative framework designed to refine the dataset and the model simultaneously through a synergy of cleaning and mixing.
2.1 Iterative Cleaning and Mixing
The framework operates in three distinct phases:
- Cleaning (Selection): The algorithm identifies a subset of potentially clean samples using robust training techniques, such as Iterative Trimmed Loss Minimization. It effectively prioritizes samples with the lowest loss.
- Selective Mixing (C-Mixup): Data augmentation is performed exclusively on the identified clean subset. Using a Gaussian kernel, the system determines the probability of mixing pairs based on the distance between their labels, ensuring that only semantically similar samples are interpolated.
- Model Updating: The model is retrained on the augmented clean set. The resulting improved accuracy allows the model to better distinguish between complex clean data and actual noise in subsequent rounds.
2.2 Dynamic Bandwidth Tuning
A critical component of RC-Mixup is the management of the "bandwidth" parameter, which controls the label distance threshold for mixing. The authors identified a "Bandwidth Paradox": in early stages with high noise, a larger bandwidth can help dilute the impact of outliers. However, as the dataset becomes cleaner through successive rounds, a smaller, more precise bandwidth yields better results. RC-Mixup periodically re-tunes this parameter to optimize performance throughout the training process.
3. Experimental Evaluation
The effectiveness of RC-Mixup was evaluated using a mix of synthetic and real-world regression benchmarks:
- Spectrum: A synthetic dataset for semiconductor manufacturing thickness prediction.
- NO2: Real-world nitrogen dioxide concentration prediction based on traffic and weather.
- Airfoil: Aerodynamic and acoustic test results from wind tunnel trials.
- Exchange-Rate: A time-series dataset of daily exchange rates from eight countries.
The models were subjected to both Gaussian noise (random jitters) and Label Flipping (extreme errors) to test their resilience.
4. Key Insights and Ideas
4.1 The Virtuous Cycle
The core contribution of RC-Mixup is the synergy between cleaning and mixing. By cleaning the data before mixing, the framework prevents the "poisoning" of synthetic samples. The resulting high-quality augmented data then helps the model generalize better, making it more proficient at identifying subtle noise in the next iteration.
4.2 Data-Centricity
RC-Mixup is a data-centric solution. It does not require modifications to the underlying neural network architecture, making it a "plug-and-play" module compatible with various robust training backbones like O2U-Net or SELFIE.
5. Results
RC-Mixup consistently outperformed standalone robust training and augmentation strategies across all tested datasets.
- Accuracy: On the Spectrum dataset with 30% noise, RC-Mixup achieved a Root Mean Squared Error (RMSE) of 7.47, significantly outperforming C-Mixup alone (12.12) or robust training alone (9.75).
- Noise Detection: The framework improved the accuracy of identifying noisy samples by up to 5% compared to standard robust training.
- Near-Optimal Performance: In several benchmarks, RC-Mixup performed almost as well as models trained on perfectly clean versions of the datasets.
6. Limitations and Future Work
Despite its performance, RC-Mixup introduces computational overhead due to its multi-round training and bandwidth tuning requirements. While a "bandwidth decaying" shortcut can mitigate this, it involves a slight performance trade-off. Additionally, the method ideally benefits from a small, clean validation set, which may not always be available. Future research may focus on more efficient, incremental updates to sampling probabilities to reduce computational costs while maintaining high accuracy.
Summary updated