Attention Bridging Network for Knowledge Transfer

Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, Yun Fu

Abstract

The Attention Bridging Network (AttnBN) addresses the limitations of weakly-supervised semantic segmentation (WSSS) by leveraging knowledge from simple, single-label datasets. Traditional WSSS models often focus only on discriminative object parts or get confused by background co-occurrences. AttnBN introduces an "attention bridge" that regularizes a model's attention maps using foreground priors from a source domain of simple scenes. By forcing the network to learn domain-invariant features and holistic object patterns, AttnBN significantly improves object localization and achieves state-of-the-art performance in WSSS benchmarks.

Introduction

In computer vision, training models for semantic segmentation usually requires expensive pixel-level annotations. Weakly-supervised semantic segmentation (WSSS) attempts to solve this by training on simple image-level labels. Most WSSS methods rely on "attention maps" (like Grad-CAM) to find objects. However, standard classification networks are often "lazy": they focus primarily on the most discriminative regions (e.g., a bird’s beak or a dog’s face) rather than the whole object. Furthermore, in complex scenes, networks often get confused by background elements that consistently co-occur with an object, such as water appearing with boats.

Conceptual overview of knowledge transfer from simple source domains to complex target domains using foreground prior guidance.

The authors identify a missed opportunity: simple datasets (single-label, centered objects) have very clear foreground/background boundaries. They aim to transfer this "complete object" knowledge to complex datasets to produce more holistic and accurate attention maps. This process, illustrated in Figure 1, uses foreground priors from the simple domain to guide the attention of the model in the complex domain.

Methodology

The proposed Attention Bridging Network (AttnBN) consists of two parallel classification streams—Source and Target—that share weights. The core idea is that objects of the same category share similar visual patterns across domains. By forcing the network to have similar attention responses in both the simple source and complex target domains, knowledge is successfully transferred.

The Attention Bridging Network architecture, showing shared weights between source and target streams and the integration of attention constraint and adversarial losses.

As shown in Figure 3, the model integrates the gradient-based attention mechanism directly into the training objective using four combined losses:

  • Classification Loss ($L_{cl}$): Standard cross-entropy for both domains.
  • Attention Constraint Loss ($L_{ac}$): Uses pre-computed saliency maps from the simple source domain to "teach" the network to cover the entire object.
  • Adversarial Loss ($L_{adv}$): A discriminator network tries to distinguish between source and target features. This forces the classification network to learn domain-invariant features, making the attention bridge more effective.

Key Insights

  • Attention as a Regularizer: Instead of just using attention to explain a model, this paper treats it as a target for optimization. By "shaping" the attention, the underlying features the model learns are also shaped.
  • Simple Data as Advantageous Knowledge: Simple datasets contain "clean" knowledge about what an object looks like in its entirety. This is "advantageous knowledge" that can be harvested to fix the biases of models trained on messy, complex data.
  • Breaking Background Bias: AttnBN helps the model ignore co-occurring backgrounds. Because the simple source domain doesn't always feature the same background biases (e.g., boats aren't always on water in the source), the network learns to focus strictly on the object itself.

Experimental Results

The model was evaluated using PASCAL VOC 2012 as the target domain, with source domains derived from Caltech-256 and ImageNet CLS-LOC.

  • Better Attention Maps: In direct evaluations on PASCAL VOC 2012, AttnBN achieved a 50.9% mIoU (mean Intersection over Union), a massive jump from the 30.2% mIoU achieved by the standard Grad-CAM baseline.
  • State-of-the-Art WSSS: When integrated into a full weakly-supervised segmentation framework (DSRG), the method achieved 63.0% mIoU on the VOC test set, outperforming previous state-of-the-art methods like DilConv and GAIN.
  • Improved Generalization: In domain adaptation tasks, adding the attention bridge improved the Mean Average Precision (mAP) from 61.3% to 71.0%, proving that learning "integral" object patterns makes the model more robust to new environments.

Conclusion

The Attention Bridging Network demonstrates that attention maps can serve as an effective medium for knowledge transfer between domains of varying complexity. While the method currently relies on pre-computed saliency maps for the source domain, future work aims to explore unsupervised saliency detection to automate the generation of foreground priors. This approach opens new avenues for leveraging large-scale, simple datasets to improve performance on complex, real-world vision tasks.

Summary updated

Abstract

The attention of a deep neural network obtained by back-propagating gradients can effectively explain the decision of the network. They can further be used to explicitly access to the network response to a specific pattern. Considering objects of the same category but from different domains share similar visual patterns, we propose to treat the network attention as a bridge to connect objects across domains. In this paper, we use knowledge from the source domain to guide the network's response to categories shared with the target domain. With weights sharing and domain adversary training, this knowledge can be successfully transferred by regularizing the network's response to the same category in the target domain. Specifically, we transfer the foreground prior from a simple single-label dataset to another complex multi-label dataset, leading to improvement of attention maps. Experiments about the weakly-supervised semantic segmentation task show the effectiveness of our method. Besides, we further explore and validate that the proposed method is able to improve the generalization ability of a classification network in domain adaptation and domain generalization settings.
BibTeX
@inproceedings{li2019attention,
  author = {Kunpeng Li and Yulun Zhang and Kai Li and Yuanyuan Li and Yun Fu},
  title = {{Attention Bridging Network for Knowledge Transfer}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00530},
}
Figure