Generating Part-Based Global Explanations Via Correspondence

Kunal Rathore, Prasad TadepalliarXiv 2025 · DOI · arXiv

Abstract

Deep Neural Networks (DNNs) often function as "black boxes," making their decision-making processes difficult to interpret. This paper introduces GEPC (Global Explanations via Part Correspondence), a framework that generates global, part-based explanations by leveraging deep feature correspondence. By labeling only a small fraction of a dataset and using Hyperpixel Flow (HPF) to transfer these labels, GEPC identifies the minimal sufficient parts required for a model's prediction. These local insights are then aggregated into broad, human-readable rules. Experimental results on datasets such as PartImageNet and CUB-200 show that GEPC achieves high label transfer accuracy and provides succinct global explanations that reveal architectural differences in model reasoning.

Introduction

While local explanation methods like saliency maps highlight influential pixels for individual images, they fail to describe a model's general behavior across an entire category. Conversely, existing global explanation methods are often bottlenecked by the need for exhaustive human-annotated part labels. GEPC addresses this challenge by providing a pipeline to generate succinct, global explanations using human-understandable concepts (e.g., "beak," "wing") without requiring massive manual annotation.

Methodology

The GEPC framework consists of a four-stage pipeline designed to transition from raw image data to symbolic global rules.

The GEPC pipeline: from gallery image sets with part-labels to global symbolic explanation visualization.
  1. Label Transfer via Correspondence: The system uses a small "gallery" of labeled images. For any "query" image, visually similar gallery images are identified. Hyperpixel Flow (HPF) is then employed to match deep features across CNN layers, effectively transferring part labels (e.g., "Bird-Head") from the gallery to the query.
  2. Unsupervised Segmentation: The SLIC algorithm partitions images into "superpixels," which act as the fundamental units for building explanations.
  3. Minimal Sufficient Explanations (MSXs): A beam search algorithm identifies the smallest set of parts (e.g., just the head and beak) that allows the model to maintain high confidence in its original prediction.
  4. Global Aggregation: A Greedy Set Cover algorithm finds a minimal list of symbolic rules (e.g., {Head, Wing, Beak}) that explains the model's decisions for the largest possible portion of the dataset.

Key Insights

A central finding of this research is that pre-trained classification models inherently function as part detectors. GEPC simply maps human-readable labels to these existing internal feature representations. Furthermore, the authors discovered that relational rules—describing spatial orientations like "Head above Body"—significantly increase the "coverage" of global explanations compared to simple part lists.

Experimental Results

The framework was evaluated on PartImageNet, Stanford Cars, and the Caltech-UCSD Birds (CUB-200) datasets.

Visualization of the part-labeling and segmentation process on a query image.
  • Label Accuracy: HPF-based label transfer achieved an average accuracy of 84.28% across 158 categories in PartImageNet.
  • Rule Generation: For the "Bee-Eater" class, the top-performing rule covered 15% of the test data using only {Bird-Head, Bird-Body}.
  • Architectural Nuance: GEPC revealed that different architectures, such as ResNet50 and VGG19, rely on distinct part combinations to reach the same classification, highlighting the method's utility in model auditing.

Conclusion and Future Work

GEPC offers a scalable path toward interpretable AI by reducing the reliance on manual labeling. While the method is currently sensitive to extreme variations in object pose, it successfully generates readable decision lists for complex visual categories. Future research will explore extending this logic to video activity recognition and visual question answering.

Summary updated

BibTeX
@misc{rathore2025generating,
  author = {Kunal Rathore and Prasad Tadepalli},
  title = {{Generating Part-Based Global Explanations Via Correspondence}},
  year = {2025},
  eprint = {2509.15393},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.15393},
}
Figure