Energy-based Self-Training and Normalization for Unsupervised Domain Adaptation

Samitha Herath, Basura Fernando, Ehsan Abbasnejad, Munawar Hayat, Shahram Khadivi, Mehrtash Harandi, Hamid Rezatofighi, Gholamreza Haffari

Unsupervised Domain Adaptation (UDA) remains a critical challenge in computer vision, aiming to transfer knowledge from a labeled source domain to an unlabeled target domain. Traditional approaches typically rely on self-training via pseudo-labeling or domain alignment of feature distributions. However, these methods often suffer from confirmation bias—where models learn from their own incorrect predictions—and instability when aligning unbounded output scores. This paper, presented at ICCV 2023, introduces a novel framework that leverages Energy-Based Learning (EBL) to address these limitations through robust instance selection and stabilized score normalization.

The Energy-Based Learning Framework

The core innovation of this work lies in treating a discriminative classifier as an energy-based model. By defining "Energy" as the negated output of the model's final logit layer, the authors can model the joint distribution of data and labels, $P(x, y)$. This perspective allows the model to assess not just the conditional probability of a label given an image, but the overall compatibility of a target image with the learned source distribution.

1. Energy-Based Instance Selection (SEEBS)

To mitigate confirmation bias in self-training, the authors propose SEEBS (Energy-Based Instance Selection). Unlike standard methods that select target samples based solely on softmax confidence, SEEBS evaluates the reliability of an instance by looking at its "free energy."

The proposed framework: (a) SEEBS instance selection using ViT attention maps, (b) the SCON normalization module, and (c) the overall SCAL+SCON alignment architecture.

As illustrated in Figure 1(a), the method utilizes the inherent attention mechanism of Vision Transformers (ViT). It compares the free energy calculated using the standard attention map ($A$) against a "non-attention" mask ($A'$). A target instance is deemed reliable if its free energy is significantly lower (indicating higher density/probability) when the model focuses on its learned attention regions compared to non-important regions. This ensures the model only trains on target samples that structurally align with its internal representation of the task.

2. Score Normalization (SCON)

For domain alignment, the authors employ Score ALignment (SCAL), which minimizes the free energy gap between domains. However, energy scores are naturally unbounded, which can lead to training instability or representation collapse. To solve this, the authors introduce SCON (Score Normalization), shown in Figure 1(b). SCON acts as a specialized normalization layer for energy scores, ensuring they remain within a stable, comparable range across domains during the alignment process. This module is the "glue" that enables stable energy-based alignment for deep architectures like ViTs.

Experimental Results

The proposed method, SEEBS+, was evaluated on three major UDA benchmarks: DomainNet, OfficeHome, and VISDA2017.

  • DomainNet: The framework achieved an average accuracy of 83.3%, a 1.8% improvement over the previous state-of-the-art (PACMAC). Notably, on the difficult "Clipart to Painting" task, the method saw a 6.8% performance jump.
  • VISDA2017: SEEBS+ reached 79.5% accuracy, outperforming PACMAC by 2.4%.

Ablation Studies and Stability

The importance of the SCON module cannot be overstated. In ablation experiments, the authors found that attempting energy alignment without proper normalization leads to a catastrophic failure in performance.

Ablation studies showing the impact of instance selection strategies (left) and the critical role of SCON in stabilizing alignment (right).

As shown in the right table of Figure 5, removing the SCON normalization (the "OnlySCON" or "OnlySCAL" rows) results in significantly lower performance compared to the combined SCAL+SCON approach. In one specific test on DomainNet, the accuracy dropped from 82.1% to 46.4% when SCON was not properly integrated, highlighting its necessity for stable energy-based UDA.

Conclusion

By integrating Energy-Based Learning into the UDA pipeline, this research provides a more principled way to select target instances and align domains. The SEEBS mechanism allows for cleaner self-training by modeling data density, while SCON provides the necessary stability for energy-based alignment. While currently optimized for Vision Transformers due to the attention-based selection, future work may extend these energy-based principles to broader architectures and more complex adaptation scenarios like Open-Set Domain Adaptation.

Summary updated

Abstract

We propose an Unsupervised Domain Adaptation (UDA) method by making use of Energy-Based Learning (EBL) and demonstrate 1. EBL can be used to improve the instance selection for a self-training task on the unlabelled target domain, and 2. alignment and normalizing energy scores can learn domain-invariant representations. For the former, we show that an energy-based selection criterion can be used to model instance selections by mimicking the joint distribution between data and predictions in the target domain. As per learning domain invariant representations, we show that stable domain alignment can be achieved by a combined energy alignment and an energy normalization process. We implement our method in consistent with the vision-transformer (ViT) backbone and show that our proposed method can outperform state-of-the-art ViT based UDA methods on diverse benchmarks (DomainNet, Office-Home, and VISDA2017).
BibTeX
@inproceedings{herath2023energy,
  author = {Samitha Herath and Basura Fernando and Ehsan Abbasnejad and Munawar Hayat and Shahram Khadivi and Mehrtash Harandi and Hamid Rezatofighi and Gholamreza Haffari},
  title = {{Energy-based Self-Training and Normalization for Unsupervised Domain Adaptation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01070},
}
Figure