Explainable Deep Classification Models for Domain Generalization

Andrea Zunino, Sarah Adel Bargal, Riccardo Volpi, Mehrnoosh Sameki, Jianming Zhang, Stan Sclaroff, Vittorio Murino, Kate Saenko

Abstract

Conventionally, AI models are thought to trade off explainability for lower accuracy. We develop a training strategy that not only leads to a more explainable AI system for object classification, but as a consequence, suffers no perceptible accuracy degradation. Explanations are defined as regions of visual evidence upon which a deep classification network makes a decision. Our training strategy enforces a periodic saliency-based feedback to encourage the model to focus on the image regions that directly correspond to the ground-truth object. We demonstrate that this leads to improved generalization to new domains without hindering performance on the original domain.

Introduction: The Challenge of Shortcut Learning

Deep learning models often act as "shortcut learners," relying on background context or domain-specific artifacts rather than the actual object of interest. For instance, a model might identify a boat primarily by the presence of water. This leads to significant performance drops when the model encounters a new domain, such as a boat in a dry dock or a sketch of a boat. This phenomenon is known as domain bias.

Visualizing the gap between domains and the role of XAI in focusing on relevant features.

The authors address single-source domain generalization, where a model is trained on one dataset (e.g., real photos) and must generalize to vastly different styles like sketches, clipart, or 3D renders. The core idea is to use explainable AI (XAI) to force the model to be "right for the right reasons."

Methodology: XAI as a Training Regularizer

The proposed strategy introduces a periodic feedback loop during training. Every few epochs, the model generates a saliency map (using Grad-CAM) for its current prediction. The algorithm then performs a "Pointing Game" check: if the peak of the saliency map falls within the ground-truth object mask, the model is looking at the correct features. If not, the model is being distracted by context.

To correct this, the model applies Activation Masking. The ground-truth mask (or the correct saliency map) is used to element-wise multiply the feature activations in the final spatial layers of the network. This process "mutes" irrelevant background noise and amplifies the object's features. Crucially, this masking only occurs during training; at test time, the model remains a standard, efficient classifier.

Key Results and Generalization

The XAI training strategy was evaluated on source domains like MSCOCO and PASCAL VOC, then tested on unseen target domains from DomainNet and Syn2Real.

Accuracy comparison across six unseen target domains using different training strategies.

The results demonstrate a clear advantage:

  • Improved Generalization: The XAI strategy achieved a 2.56% relative accuracy improvement across all six unseen domains compared to standard training.
  • Complementary to Augmentation: As shown in the results, combining XAI feedback with standard data augmentation (XAI+aug) consistently yielded the highest performance.
  • No Accuracy Trade-off: Performance on the original training domain (MSCOCO) remained stable, debunking the myth that explainability necessitates a drop in accuracy.
  • Human Alignment: A study showed that humans preferred the XAI-trained model's explanations 67% of the time, indicating that the model's internal logic better aligns with human intuition.

Conclusion

By treating explainability as a proactive training signal rather than a post-hoc analysis tool, this research shows that forcing models to focus on semantically relevant features significantly boosts their ability to generalize. This approach effectively bridges the visual-semantic gap, allowing models trained on real images to perform better on stylized or abstract representations without additional training data from those domains.

Summary updated

Abstract

Conventionally, AI models are thought to trade off explainability for lower accuracy. We develop a training strategy that not only leads to a more explainable AI system for object classification, but as a consequence, suffers no perceptible accuracy degradation. Explanations are defined as regions of visual evidence upon which a deep classification network makes a decision. This is represented in the form of a saliency map conveying how much each pixel contributed to the network's decision. Our training strategy enforces a periodic saliency-based feedback to encourage the model to focus on the image regions that directly correspond to the ground-truth object. We quantify explainability using an automated metric, and using human judgement. We propose explainability as a means for bridging the visual-semantic gap between different domains where model explanations are used as a means of disentagling domain specific information from otherwise relevant features. We demonstrate that this leads to improved generalization to new domains without hindering performance on the original domain.
BibTeX
@inproceedings{zunino2021explainable,
  author = {Andrea Zunino and Sarah Adel Bargal and Riccardo Volpi and Mehrnoosh Sameki and Jianming Zhang and Stan Sclaroff and Vittorio Murino and Kate Saenko},
  title = {{Explainable Deep Classification Models for Domain Generalization}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=m4mRS8SzOe},
}
Figure