Introduction
Deep learning models often struggle when deployed in environments that differ from their training data. Unsupervised Domain Adaptation (UDA) addresses this by aligning a labeled "source" domain (e.g., synthetic images) with an unlabeled "target" domain (e.g., real-world photos). While UDA has seen significant success, standard models remain incredibly brittle: their accuracy often collapses to 0% when faced with adversarial attacks—small, human-imperceptible perturbations designed to fool neural networks.
Traditional solutions like Adversarial Training (AT) require ground-truth labels to generate effective perturbations, which are unavailable in the target domain of UDA. Naive self-training using "pseudo-labels" (the model's own predictions) often leads to error propagation, where the model learns from its own mistakes, resulting in poor accuracy and robustness. SRoUDA (Meta Self-Training for Robust UDA) breaks this cycle by using a meta-learning feedback loop to refine these labels.
Methodology
SRoUDA introduces a two-phase pipeline to inject robustness into UDA without the typical "robustness tax" that degrades performance on clean data.
1. Pre-training with Random Masked Augmentation (RMA)
To generate high-quality initial pseudo-labels, the authors introduce Random Masked Augmentation (RMA). During pre-training, the model is forced to classify target images where a staggering 85% of the patches have been masked out. This extreme regularization forces the network to focus on global object structures rather than domain-specific textures or local noise, leading to more generalized features and more accurate initial guesses for the target data.
2. The Meta Self-Training Loop
The core innovation of SRoUDA is a bi-level optimization process that alternates between two steps:
- Target Model Training: A target model ($G_t$) is trained using Adversarial Training on target data, guided by pseudo-labels generated by a source model ($G_s$).
- The Meta Step: To prevent the target model from learning from noisy labels, the framework uses the performance of $G_t$ on the labeled source data as a feedback signal (meta-loss). The intuition is that if the target model (trained on pseudo-labels) generalizes well back to the source domain, the pseudo-labels must have been accurate. This feedback is used to fine-tune the source model $G_s$, which in turn produces "cleaner" labels for the next iteration.
Experimental Results
SRoUDA was evaluated on standard benchmarks including Office-31, Digits (MNIST/USPS/SVHN), and CIFAR-10 to STL-10.
Key findings include:
- Eliminating the Robustness Tax: In many cases, SRoUDA actually improved clean accuracy while simultaneously boosting robustness. For example, on the Office-31 (A → W) task, SRoUDA achieved 95.97% clean accuracy and 84.68% robustness, whereas standard UDA achieved 94.5% clean accuracy but 0% robustness.
- State-of-the-Art Performance: SRoUDA consistently outperformed existing robust UDA methods like RFA and simple combinations of UDA with AT across all tested datasets.
- High-Fidelity Robustness: On the Digits (U → M) task, the model reached a robustness of 96.44%, nearly matching its clean accuracy of 98.50%.
Conclusion
SRoUDA demonstrates that adversarial robustness and high clean accuracy are not mutually exclusive in domain adaptation. By treating pseudo-label generation as a meta-learning problem and using the labeled source domain as a validation set, SRoUDA provides a robust framework for deploying reliable AI in unlabeled target environments. Future work may explore applying this meta-feedback mechanism to other label-scarce scenarios like semi-supervised or open-set learning.
Summary updated