TL;DR
This paper introduces CAM+SPLR, a novel self-training method for Unsupervised Domain Adaptation (UDA) that simultaneously enhances model robustness against adversarial attacks and improves classification accuracy. By aligning attention maps of clean and adversarial target images and refining pseudo-labels through a two-step gradient update, CAM+SPLR significantly boosts performance on challenging UDA benchmarks.
What Problem Does This Paper Address?
Unsupervised Domain Adaptation (UDA) aims to transfer knowledge from a labeled source domain to an unlabeled target domain. While successful, UDA models are highly vulnerable to adversarial attacks, where small perturbations can drastically degrade performance. Existing methods for making models robust (Adversarial Training) often rely on ground-truth labels, making them unsuitable for UDA. While self-training can leverage unlabeled target data, it struggles with noisy pseudo-labels, leading to models that focus on irrelevant features and compromise both accuracy and robustness. This paper addresses the critical gap of achieving both adversarial robustness and high accuracy in UDA settings, especially for safety-critical applications.
Methods
The proposed method, Consistent Attention Mapping with Self Pseudo Label Refinement (CAM+SPLR), builds upon the self-training paradigm. It involves two key components:
- Consistent Attention Mapping (CAM): This technique encourages the attention maps of clean target images and their corresponding adversarial counterparts to be similar. By using a frozen "Anchor" model to guide the attention of the main "TargetNet" model, CAM ensures that the model focuses on semantically relevant features, preventing adversarial perturbations from diverting attention to irrelevant regions.
- Self Pseudo Label Refinement (SPLR): This component refines the pseudo-labels generated for the target domain. It employs a novel two-step Stochastic Gradient Descent (SGD) process within each training epoch. The first step updates the TargetNet model using a combined loss of cross-entropy and attention loss. The second step further updates the model using a combination of Unsupervised Data Augmentation Loss (UDA) and Semi-Supervised Loss (SSL), which progressively refines the pseudo-labels without requiring additional models.
The overall training pipeline begins with a pre-trained UDA model (e.g., DANN) for initial pseudo-label generation. This pre-trained model is then split into the Anchor and TargetNet models, with the Anchor model being frozen.
Datasets
The authors evaluate their method on three widely used multi-domain UDA datasets:
- OfficeHome: Contains four domains (Art, Clipart, Product, RealWorld) with 65 categories.
- PACS: Features four domains (Photo, Art Painting, Cartoon, Sketch) with seven categories.
- VisDA: A large dataset with two domains (Synthetic, Real) and 12 categories.
Key Insights and Ideas
- Attention Shift is a Major Vulnerability: The paper visually demonstrates how adversarial attacks can cause significant shifts in model attention, leading to misclassifications even for semantically clear objects.
- Pseudo-label Noise is a Double-Edged Sword: While essential for UDA self-training, noisy pseudo-labels can lead to overfitting on irrelevant features, degrading both accuracy and robustness.
- Jointly Optimizing Attention and Pseudo-Labels is Crucial: The core insight is that improving robustness and accuracy in UDA requires addressing both the feature representation (via attention) and the label quality (via refinement) simultaneously.
- Two-Step SGD for Refinement: The novel two-step SGD update allows for progressive refinement of pseudo-labels and robust feature learning within a single training epoch, leveraging feedback from both clean and adversarial examples.
- Multi-Level Attention is Beneficial: Using attention maps from various feature levels (low, mid, and high) in ResNet-50 provides a more comprehensive understanding of the image and leads to better performance than using single-level attention.
Key Results
- Significant Robustness and Accuracy Gains: CAM+SPLR achieves substantial improvements over existing UDA baselines and state-of-the-art methods.
- On PACS and VisDA datasets, it shows an average accuracy improvement of 6% and 8.1%, respectively.
- Average robustness improvement of 10.2% and 4.9% on PACS and VisDA datasets, respectively.
- Outperforms Component-wise Methods: Ablation studies show that CAM+SPLR performs better than using CAM or SPLR individually. For instance, using only CAM can lead to misclassifications due to label noise, while only SPLR can suffer from attention shifts.
- Scalability with Attack Budget: The method demonstrates consistent outperformance across varying perturbation sizes and attack iterations, indicating good scalability.
Limitations and Future Work
- Dependence on UDA Baseline: The performance of CAM+SPLR can be influenced by the quality of the initial UDA baseline model used for pre-training. A better UDA baseline can further enhance the robustness and accuracy of CAM+SPLR.
- Computational Cost: The two-step SGD process and attention mapping might introduce additional computational overhead compared to simpler UDA methods.
- Future Work: The paper suggests exploring the integration of CAM+SPLR with other UDA techniques and investigating its application to more complex domains and tasks. Further research into optimizing the attention map selection and the balance between CAM and SPLR components could also be beneficial.
Summary updated