Abstract
Conventionally, AI models are thought to trade off explainability for lower accuracy. We develop a training strategy that not only leads to a more explainable AI system for object classification, but as a consequence, suffers no perceptible accuracy degradation. Explanations are defined as regions of visual evidence upon which a deep classification network makes a decision. Our training strategy enforces a periodic saliency-based feedback to encourage the model to focus on the image regions that directly correspond to the ground-truth object. We demonstrate that this leads to improved generalization to new domains without hindering performance on the original domain.
Introduction: The Challenge of Shortcut Learning
Deep learning models often act as "shortcut learners," relying on background context or domain-specific artifacts rather than the actual object of interest. For instance, a model might identify a boat primarily by the presence of water. This leads to significant performance drops when the model encounters a new domain, such as a boat in a dry dock or a sketch of a boat. This phenomenon is known as domain bias.
The authors address single-source domain generalization, where a model is trained on one dataset (e.g., real photos) and must generalize to vastly different styles like sketches, clipart, or 3D renders. The core idea is to use explainable AI (XAI) to force the model to be "right for the right reasons."
Methodology: XAI as a Training Regularizer
The proposed strategy introduces a periodic feedback loop during training. Every few epochs, the model generates a saliency map (using Grad-CAM) for its current prediction. The algorithm then performs a "Pointing Game" check: if the peak of the saliency map falls within the ground-truth object mask, the model is looking at the correct features. If not, the model is being distracted by context.
To correct this, the model applies Activation Masking. The ground-truth mask (or the correct saliency map) is used to element-wise multiply the feature activations in the final spatial layers of the network. This process "mutes" irrelevant background noise and amplifies the object's features. Crucially, this masking only occurs during training; at test time, the model remains a standard, efficient classifier.
Key Results and Generalization
The XAI training strategy was evaluated on source domains like MSCOCO and PASCAL VOC, then tested on unseen target domains from DomainNet and Syn2Real.
The results demonstrate a clear advantage:
- Improved Generalization: The XAI strategy achieved a 2.56% relative accuracy improvement across all six unseen domains compared to standard training.
- Complementary to Augmentation: As shown in the results, combining XAI feedback with standard data augmentation (XAI+aug) consistently yielded the highest performance.
- No Accuracy Trade-off: Performance on the original training domain (MSCOCO) remained stable, debunking the myth that explainability necessitates a drop in accuracy.
- Human Alignment: A study showed that humans preferred the XAI-trained model's explanations 67% of the time, indicating that the model's internal logic better aligns with human intuition.
Conclusion
By treating explainability as a proactive training signal rather than a post-hoc analysis tool, this research shows that forcing models to focus on semantically relevant features significantly boosts their ability to generalize. This approach effectively bridges the visual-semantic gap, allowing models trained on real images to perform better on stylized or abstract representations without additional training data from those domains.
Summary updated