Learning CNN on ViT: A Hybrid Model to Explicitly Class-Specific Boundaries for Domain Adaptation

Ba Hung Ngo, Nhat-Tuong Do-Tran, Tuan-Ngoc Nguyen, Hae-Gon Jeon, Tae Jong ChoiCVPR 2024

Abstract

Domain Adaptation (DA) remains a pivotal challenge in computer vision, requiring models to generalize from a labeled source domain to an unlabeled or partially labeled target domain. Traditional approaches often rely on either Convolutional Neural Networks (CNNs) or Vision Transformers (ViTs), each possessing inherent limitations regarding local texture bias or global context overfitting. This paper introduces the Explicitly Class-specific Boundaries (ECB) model, a hybrid framework that leverages the complementary strengths of ViTs and CNNs. Through a "Finding to Conquering" (FTC) strategy and mutual co-training, ECB establishes robust decision boundaries that significantly improve performance on standard benchmarks, including Office-Home, DomainNet, and Office-31.

1. Introduction

The core difficulty in Domain Adaptation (DA) lies in the distribution shift between source and target data. While CNNs excel at capturing local spatial features, they often struggle with global context. Conversely, ViTs capture long-range dependencies but are susceptible to overfitting when target data is limited. Furthermore, many DA models suffer from a bias toward source-domain decision boundaries, leading to poor generalization on "hard" target samples that fall outside these narrow margins.

The Explicitly Class-specific Boundaries (ECB) model addresses these issues by integrating ViT and CNN architectures into a unified learning framework. By explicitly defining and then refining class-specific boundaries, ECB ensures that target features are correctly clustered even in the presence of significant domain shifts.

2. Methodology: The Finding to Conquering (FTC) Strategy

The ECB framework is built upon a two-stage strategy termed "Finding to Conquering" (FTC), which utilizes the distinct inductive biases of ViTs and CNNs to identify and resolve boundary uncertainties.

2.1 The Finding Stage (ViT as the Scout)

In the "Finding" stage, a fixed ViT encoder serves as a global scout. The model trains two separate classifiers to maximize their prediction discrepancy on unlabeled target data. This adversarial approach intentionally "stretches" the decision boundaries, identifying the most uncertain regions of the feature space—the "worst-case" scenarios where the model's confidence is lowest and the domain gap is most apparent.

2.2 The Conquering Stage (CNN as the Settler)

Once the boundaries are identified, the CNN encoder takes the lead in the "Conquering" stage. The CNN works to minimize the discrepancy established by the ViT. By doing so, it pulls target features away from the uncertain boundary regions and into well-defined clusters, leveraging the CNN’s local precision and spatial invariance to stabilize the feature space.

2.3 Mutual Co-training and Pseudo-labeling

To prevent architectural divergence, ECB employs a mutual co-training mechanism. The ViT and CNN models exchange knowledge via pseudo-labels. To maintain high label quality, the system uses a dual-threshold approach ($\tau_{vit}$ and $\tau_{cnn}$). Only predictions exceeding these confidence thresholds are used as training signals for the peer model, effectively filtering out noise and reducing the propagation of errors common in semi-supervised learning.

3. Experimental Evaluation

The researchers evaluated ECB across three major Domain Adaptation benchmarks: Office-Home, DomainNet, and Office-31. The experiments covered both Unsupervised Domain Adaptation (UDA) and Semi-Supervised Domain Adaptation (SSDA) settings (1-shot and 3-shot).

3.1 Quantitative Results

ECB demonstrated superior performance compared to existing state-of-the-art (SOTA) methods:

  • Office-Home (UDA): The model achieved an average accuracy of 81.2%, representing a +5.4% improvement over the previous leader, EIDCo.
  • DomainNet (SSDA): In 3-shot scenarios, ECB reached 85.5% accuracy. It showed particular strength in difficult tasks, such as "Sketch to Painting," where it achieved a +9.3% boost.
  • Office-31 (SSDA): ECB outperformed the G-ABC model by a substantial +12.6%, reaching an average accuracy of 84.6%.

3.2 Qualitative Analysis

Grad-CAM visualizations indicate that the ECB-enhanced ViT focuses more accurately on relevant objects compared to standard ViT baselines. While standard models may be distracted by background elements, ECB directs the attention of the network toward the definitive features of the object, such as the specific anatomy of a bird or the structure of a cannon.

4. Discussion and Conclusion

The success of ECB highlights the power of architectural synergy. By using ViT to map the global landscape and CNN to navigate local clusters, the model overcomes the individual weaknesses of each backbone. The "Finding to Conquering" strategy provides a robust method for handling hard target samples that typically fall outside narrow source-biased boundaries.

However, the model's reliance on manually tuned confidence thresholds remains a limitation. Future work will explore dynamic thresholding techniques to automate this process, allowing the model to adapt more fluidly to varying levels of domain noise and dataset complexity without manual intervention.

Summary updated

BibTeX
@inproceedings{ngo2024learning,
  author = {Ba Hung Ngo and Nhat-Tuong Do-Tran and Tuan-Ngoc Nguyen and Hae-Gon Jeon and Tae Jong Choi},
  title = {{Learning CNN on ViT: A Hybrid Model to Explicitly Class-Specific Boundaries for Domain Adaptation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
}
Figure