Domain-Specific Batch Normalization for Unsupervised Domain Adaptation

Tackgeun You

TL;DR

This paper introduces Domain-Specific Batch Normalization (DSBN), a novel technique that enhances unsupervised domain adaptation by allowing batch normalization layers to specialize for source and target domains. By integrating DSBN into a two-stage learning framework, the authors achieve state-of-the-art results on several benchmarks, demonstrating improved domain invariance and adaptation capabilities.

What Problem Does This Paper Address?

Unsupervised Domain Adaptation (UDA) aims to transfer knowledge learned from a labeled source domain to an unlabeled target domain. A major hurdle in this process is "domain shift," where the data distributions of the source and target domains differ, leading to performance degradation. Existing UDA methods often share the entire network architecture across domains, which can be suboptimal as it forces a single model to accommodate distinct domain characteristics. The authors argue that separating domain-specific information from domain-invariant features is crucial for better generalization. They propose a method to achieve this by specializing a key component of deep neural networks: Batch Normalization (BN).

Methods

The core contribution is Domain-Specific Batch Normalization (DSBN). Unlike standard BN, which uses a single set of learned statistics (mean and variance) and affine parameters (gamma and beta) for all data within a mini-batch, DSBN creates separate BN branches for each domain (source and target). This means that for a given input, the network uses domain-specific affine parameters and statistics for normalization. This allows the network to capture and leverage domain-specific information effectively.

The proposed framework utilizes DSBN within a two-stage learning process:

  1. Pseudo-label Estimation: An existing UDA network, augmented with DSBN layers, is trained to generate initial pseudo-labels for the unlabeled target domain data. This stage leverages the improved domain-invariant representations provided by DSBN.
  2. Multi-task Classification: A second network, also equipped with DSBN, is trained using ground-truth labels for the source domain and the pseudo-labels generated in the first stage for the target domain. This stage acts as a self-training mechanism, refining the model with more reliable labels.

The authors also highlight that DSBN is a generic module that can be easily plugged into existing UDA architectures that use BN layers. Furthermore, it can be extended to multi-source domain adaptation by simply adding more domain-specific branches.

Datasets

The paper evaluates its approach on three standard UDA benchmarks:

  • VisDA-C: A large-scale dataset for visual domain adaptation, consisting of synthetic and real images from 12 object categories.
  • Office-31: A widely used benchmark with three domains (Amazon, Webcam, DSLR) and 31 object categories.
  • Office-Home: A more challenging dataset with four domains (Art, Clipart, Product, Real-World) and 65 object categories.

The authors use a fully transductive protocol for evaluation, meaning target domain labels are not used during training.

Key Insights and Ideas

  • Batch Normalization as a Domain-Specific Component: The paper successfully demonstrates that BN, often treated as a domain-agnostic component, can be a powerful lever for capturing domain-specific characteristics. By specializing BN, the network can better disentangle domain-invariant features.
  • DSBN Enhances Representation Consistency: Visualizations (t-SNE plots) show that DSBN leads to more consistent representations for samples of the same class across different domains, indicating improved domain invariance.
  • Two-Stage Learning with DSBN is Synergistic: The two-stage approach, where DSBN is used first for better pseudo-label generation and then for refined classification, significantly boosts performance. The pseudo-labels generated with DSBN are more reliable, leading to better self-training.
  • Generality and Extensibility: DSBN is presented as a plug-and-play module, easily integrated into existing UDA frameworks. Its extension to multi-source UDA is also straightforward and effective.

Key Results

The proposed DSBN-based framework achieves state-of-the-art results on the evaluated datasets:

  • On VisDA-C, the method using MSTN as a baseline with DSBN (Stage 1 and 2) achieved an accuracy of 80.2%, outperforming previous methods.
  • On Office-31, the DSBN-enhanced MSTN and CPUA models achieved accuracies of 88.3% and 88.3% respectively, surpassing other approaches.
  • In multi-source domain adaptation scenarios on Office-31 and Office-Home, DSBN consistently improved performance, especially in "separate" multi-source settings where domain-specific branches were used for each source domain. For instance, in the "Separate" A, D→W task on Office-31, DSBN achieved 99.9% accuracy compared to 99.6% for standard BN.

Limitations and Future Work

  • Iterative Refinement: While the paper shows that iterative refinement of pseudo-labels in the second stage can further improve accuracy (as seen in Table 6), the current implementation focuses on a fixed number of iterations. Exploring more sophisticated iterative strategies could be a future direction.
  • Computational Cost: While DSBN is presented as a generic module, using separate BN layers for each domain might introduce a slight increase in model complexity and computational cost, especially in multi-source scenarios with many domains. The paper does not explicitly quantify this overhead.
  • Domain-Specific Learning for Other Components: The paper focuses solely on batch normalization. Future work could explore applying domain-specific learning to other network components to further enhance adaptation.

Summary updated

Abstract

We propose a novel unsupervised domain adaptation framework based on domain-specific batch normalization in deep neural networks. We aim to adapt to both domains by specializing batch normalization layers in convolutional neural networks while allowing them to share all other model parameters, which is realized by a two-stage algorithm. In the first stage, we estimate pseudo-labels for the examples in the target domain using an external unsupervised domain adaptation algorithm---for example, MSTN or CPUA---integrating the proposed domain-specific batch normalization. The second stage learns the final models using a multi-task classification loss for the source and target domains. Note that the two domains have separate batch normalization layers in both stages. Our framework can be easily incorporated into the domain adaptation techniques based on deep neural networks with batch normalization layers. We also present that our approach can be extended to the problem with multiple source domains. The proposed algorithm is evaluated on multiple benchmark datasets and achieves the state-of-the-art accuracy in the standard setting and the multi-source domain adaption scenario.
BibTeX
@inproceedings{you2019domain,
  author = {Tackgeun You},
  title = {{Domain-Specific Batch Normalization for Unsupervised Domain Adaptation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=siq27TjgNe},
}
Figure