Abstract
The topic of generalizing machine learning models learned on a collection of source domains to unknown target domains is challenging. While many domain generalization (DG) methods have achieved promising results, they primarily rely on the source domains at train-time without manipulating the target domains at test-time. Thus, it is still possible that those methods can overfit to source domains and perform poorly on target domains. Driven by the observation that domains are strongly related to styles, we argue that reducing the gap between source and target styles can boost models' generalizability. To solve the dilemma of having no access to the target domain during training, we introduce Test-time Fourier Style Calibration (TF-Cal) for calibrating the target domain style on the fly during testing. To access styles, we utilize Fourier transformation to decompose features into amplitude (style) features and phase (semantic) features. Furthermore, we present an effective technique to Augment Amplitude Features (AAF) to complement TF-Cal. Extensive experiments on several popular DG benchmarks and a segmentation dataset for medical images demonstrate that our method outperforms state-of-the-art methods.
1. Introduction
Deep learning models often exhibit brittleness when encountering data from unseen domains, a phenomenon central to the challenge of Domain Generalization (DG). For instance, a model trained on photographic images may fail to generalize to sketches or artistic renderings of the same objects. Traditional DG approaches typically focus on training-time data augmentation to increase model robustness. However, these models remain static during inference and often struggle with "out-of-distribution" samples that fall outside the learned distribution.
This research posits that the primary discrepancy between domains lies in "style" (textures, colors, and intensities), whereas "semantics" (shapes and structures) remain largely invariant. To address this, we propose a dynamic approach that adjusts for target styles during inference without requiring retraining.
2. Methodology
We introduce TAF-Cal (Test-time Augmented Fourier Style Calibration), a framework operating in the frequency domain to bridge the style gap between source and target domains.
2.1 Fourier Decomposition and Style Calibration
Using the Fast Fourier Transform (FFT), latent features from early CNN layers are decomposed into amplitude and phase components. The amplitude encodes low-level style and intensity, while the phase preserves spatial positions and semantic structure.
Test-time Fourier Style Calibration (TF-Cal) leverages this decomposition. During inference, the model maintains a "Source Prototype"—the average style representation of the training data. The unknown target's amplitude is linearly interpolated with this source prototype, effectively "nudging" the novel style toward the known source distribution while preserving the original semantic content (phase).
2.2 Augmenting Amplitude Features (AAF)
To enhance robustness during training, we introduce Augmenting Amplitude Features (AAF). Unlike standard pixel-level augmentation, AAF swaps and mixes amplitude features between different source samples. This expands the "convex hull" of styles the model encounters during training, facilitating better generalization.
3. Experiments and Results
The proposed method was evaluated on several benchmarks, including PACS, Office-Home, and the MICCAI WMH medical imaging challenge, using a "leave-one-domain-out" protocol.
3.1 Performance on Benchmarks
On the PACS dataset using a ResNet-18 backbone, TAF-Cal achieved an average accuracy of 86.73%, significantly outperforming the "DeepAll" baseline (80.28%) and existing state-of-the-art methods like MixStyle and FACT. In medical imaging (WMH Challenge), the method demonstrated remarkable gains, particularly in the "Utrecht" domain, where the Dice Similarity Coefficient improved from 0.183 to 0.736. Ablation studies indicate that applying this calibration to early layers (specifically the second residual block) yields the best results.
3.2 Qualitative Analysis
t-SNE visualizations demonstrate that TAF-Cal effectively aligns feature distributions across disparate domains (Art, Sketch, Cartoon, Photo), creating more cohesive clusters compared to baseline methods.
4. Discussion and Conclusion
TAF-Cal provides a lightweight, "plug-and-play" solution for DG by performing calibration on the fly. Its effectiveness in early CNN layers aligns with the intuition that these layers process low-level textures. While highly effective, the method's reliance on hyperparameters suggests that future work could explore automated parameter estimation or application to Transformer-based architectures. By focusing on the frequency domain, TAF-Cal successfully decouples style from semantics, offering a robust path forward for generalizing models to truly novel environments.
Summary updated