Causal Inference over Visual-Semantic-Aligned Graph for Image Classification

Lei Meng, Xiangxian Li, Xiaoshuo Yan, Haokai Ma, Zhuang Qi, Wei Wu, Xiangxu Meng

Introduction

Standard image classification models often struggle with visual diversity, where the appearance of an object varies significantly due to environmental factors like lighting, background, or camera angle. While semantic tags (textual descriptions) are frequently used to guide models, two primary challenges persist: modality heterogeneity and spurious associations. Modality heterogeneity refers to the difficulty of mapping pixel-level data to textual concepts, while spurious associations occur when models learn "shortcuts"—such as identifying a cow based on the presence of grass rather than the animal itself.

To address these issues, this paper introduces VSCNet, a framework that aligns image regions with semantic tags and organizes them into a heterogeneous graph. By leveraging causal inference, VSCNet filters out non-causal background noise, resulting in more robust visual representations.

Methods

The VSCNet architecture is divided into three primary stages designed to refine the relationship between visual features and semantic labels.

Overview of the VSCNet architecture, including the Cascaded Spatial Network (CSNet) and the causal graph inference stage.

Fine-grained Visual-Semantic Alignment (FVA)

The first stage utilizes a Cascaded Spatial Net (CSNet) to adaptively locate "semantic-intensive" regions within an image. Unlike traditional methods that require manual bounding box annotations, FVA automatically identifies regions corresponding to specific tags. This process ensures that the model focuses on the most relevant parts of the image for a given concept.

Visual-semantic alignment comparison showing how the proposed loss functions improve region-tag localization.

Heterogeneous Association Refinement (HAR)

To mitigate the impact of noisy or incorrect tag predictions, the model employs a hierarchy of visual prototypes—pre-learned exemplars that represent typical visual characteristics of a class. By comparing image regions against these prototypes, the model can self-correct its semantic labels. These refined elements are then organized into a heterogeneous graph where nodes represent image regions, tags, and prototypes.

Causal Inference with Graphical Masking (CIM)

The core innovation of VSCNet lies in its use of causal inference. The model applies learnable masks to the visual-semantic graph to distinguish between causal factors (features essential to the class) and non-causal factors (incidental background features). Using a "backdoor adjustment" strategy, the system ensures that the final classification is based on robust causal links, effectively ignoring statistical coincidences that lead to misclassification in novel environments.

Experimental Results

VSCNet was evaluated on two major benchmarks: Ingredient-101 (food ingredient identification) and NUS-WIDE (web image concept classification).

  • Ingredient-101: Using a ViT backbone, VSCNet achieved a Top-1 Accuracy of 88.6%, outperforming previous state-of-the-art methods such as MGCC (87.1%) and CHAN (84.6%).
  • NUS-WIDE: The model reached a Precision at 1 (P@1) of 82.6%, a 3% improvement over the baseline ViT model.

Ablation studies confirmed that the Causal Inference (+C) and Prototype Refinement (+K) modules were critical to these performance gains, demonstrating that filtering associations for causality is as important as the initial alignment.

Discussion and Future Work

While VSCNet shows significant promise, the authors note that the computational cost of maintaining the Visual-Semantic Exemplar (VSE) hierarchy may pose scalability challenges for datasets containing millions of images. Future research may explore integrating this causal alignment approach into large-scale foundation models like CLIP or GPT-4V to reduce visual bias. Additionally, the application of causal graphs to fields like federated learning remains an intriguing avenue for addressing domain shifts and data privacy.

Summary updated

Abstract

Incorporating tagging information to regularize the representation learning of images usually leads to improved performance in image classification by aligning the visual features with the textual ones of higher discriminative power. Existing methods typically follow the predictive approach, which uses tags as the semantic labels for visual input to make predictions. However, they typically face the problem of handling the heterogeneity between modalities. In order to learn accurate visual-semantic mapping, this paper presents a visual-semantic causal association modeling framework termed VSCNet. It aligns visual regions with tags, uses a pre-learned hierarchy of visual and semantic exemplars to refine tag predictions and constructs an augmented heterogeneous graph to perform causal intervention. Specifically, the fine-grained visual-semantic alignment (FVA) module adaptively locates the semantic-intensive regions corresponding to tags. The heterogeneous association refinement (HAR) module associates the visual regions, semantic elements and pre-learned visual prototypes in a heterogeneous graph to filter the error predictions and enrich the information. The causal inference with graphical masking (CIM) module applies self-learned masks to discover the causal nodes and edges in the heterogeneous graph to address the spurious association, forming robust causal representations. Experimental results from two benchmarking datasets show that VSCNet effectively builds the visual-semantic associations from images and leads to better performance than the state-of-the-art methods with enriched predictive information.
BibTeX
@inproceedings{meng2025causal,
  author = {Lei Meng and Xiangxian Li and Xiaoshuo Yan and Haokai Ma and Zhuang Qi and Wei Wu and Xiangxu Meng},
  title = {{Causal Inference over Visual-Semantic-Aligned Graph for Image Classification}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i18.34141},
}
Figure