Abstract
Real-world machine learning models often fail when faced with both domain shifts and label noise. We present $A^3W$ (Anchor Alignment and Adaptive Weighting), a novel method that leverages text embeddings from vision-language models like CLIP as "semantic anchors" to guide image feature learning. By aligning visual features to these stable text anchors and dynamically down-weighting misaligned (noisy) samples, $A^3W$ significantly improves generalization and robustness on unseen domains.
Introduction
Domain Generalization (DG) aims to train models that perform well on unseen environments without target-domain training data. A significant challenge in DG is the model's tendency to latch onto spurious correlations—domain-specific artifacts like backgrounds or lighting that correlate with labels in training but fail in the real world.
This issue is exacerbated when training data contains label noise (incorrectly tagged examples). Overparameterized deep networks easily memorize this noise, amplifying errors and degrading out-of-distribution performance. Existing DG methods often lack a reliable, domain-invariant "ground truth" to anchor their representations when label noise is introduced.
Methodology
The proposed Anchor Alignment and Adaptive Weighting ($A^3W$) framework injects external, domain-invariant knowledge into the training loop using large vision-language models.
Core Components
- NLP Anchors: For each class, a text prompt (e.g., "a photo of a dog") is encoded using a frozen CLIP text encoder. These normalized embeddings serve as fixed "semantic anchors."
- Feature Alignment: Class-specific trainable mapping layers project extracted image features into the same semantic space as the NLP anchors. An alignment loss (cosine similarity) forces visual features toward their corresponding text anchors.
- Adaptive Weighting: $A^3W$ dynamically weights each sample's contribution to the loss using a softmax function over alignment scores. Samples aligning well with the text anchor are trusted as clean/invariant, while poorly aligned samples are down-weighted as potential noise.
- Iterative Optimization: Training alternates between updating the featurizer/classifier and the mapping layers. An Exponential Moving Average (EMA) network is utilized for robust inference.
Experimental Results
The method was evaluated on five standard DG benchmarks (PACS, VLCS, Office-Home, SVIRO, and DomainNet) injected with synthetic symmetric label noise at 10% and 25% levels.
As shown in the results table, $A^3W$ consistently outperformed baselines such as ERM, GroupDRO, and IRM. At a 25% label noise level, $A^3W$ achieved 82.1% accuracy on PACS (compared to 74.5% for ERM) and 93.9% on SVIRO (compared to 77.9% for ERM). On average, the method provided a 6–8% margin of improvement over baseline methods.
Key Insights
- Text as a Noise Filter: Semantic anchors act as a natural filter; noisy or out-of-distribution samples align poorly with the true class's text anchor and are automatically down-weighted.
- Semantic Richness: The most dramatic improvements occurred on the SVIRO dataset, which uses highly descriptive class names (e.g., "child in convertible car seat"), providing richer anchors than single-word labels.
- Ablation Studies: Removing NLP anchor alignment caused an ~0.86% accuracy drop, while removing adaptive weighting caused an ~1.16% drop, confirming both components are essential.
Discussion and Future Work
While $A^3W$ demonstrates significant robustness, it introduces approximately a 10% increase in training time due to class-specific mapping layers. Future work may explore dynamic anchor refinement, where text embeddings evolve based on learned feature distributions, or domain-aware prompts (e.g., "a sketch of a dog") to further enhance adaptation. Currently, the method is optimized for single-label classification and could be extended to multi-label or hierarchical tasks in the future.
Summary updated