Semantic-Space-Intervened Diffusive Alignment for Visual Classification

Zixuan Li, Lei Meng, Guoqing Chao, Wei Wu, Xiaoshuo Yan, Yimeng Yang, Zhuang Qi, Xiangxu Meng

Abstract

Cross-modal alignment typically relies on direct, one-step mappings between visual and textual features. However, the inherent structural differences between these modalities often lead to suboptimal alignment and semantic confusion. This paper introduces Semantic-Space-Intervened Diffusive Alignment (SeDA), a novel framework that leverages a bi-stage diffusion process to progressively align visual features with textual distributions through an intermediate "semantic space." By decomposing the alignment into a multi-step denoising and translation process, SeDA significantly enhances the discriminative power of visual backbones. Experimental results on image and video datasets demonstrate that SeDA consistently outperforms existing alignment methods.

Introduction

In the realm of visual classification, the Learning Using Privileged Information (LUPI) paradigm has shown that incorporating textual descriptions during training can substantially improve model performance. The core challenge lies in cross-modal alignment: forcing visual features to match the distribution of textual features.

Existing approaches, such as distance metric minimization or knowledge distillation, typically attempt this bridge in a single, abrupt step. Because vision and text are highly heterogeneous—differing in data structure, feature ranges, and class distributions—this "one-shot" projection often fails to capture the nuances of the target distribution. This leads to "semantic ambiguities," where visually similar but semantically distinct classes remain indistinguishable to the model.

Methodology: SeDA Framework

To address the limitations of one-step mapping, SeDA utilizes the multi-step Markov process characteristic of Diffusion Models. Instead of generating images from noise, SeDA progressively denoises and translates visual features into the textual feature space.

Bi-Stage Diffusion Process

The framework operates through two primary modules that utilize a modality-shared semantic space as an intermediate bridge:

  1. Diffusion-controlled Semantic Learner (DSL): During the initial stages of the diffusion process, the DSL strips away irrelevant, low-level visual noise. It maps visual features into the intermediate semantic space by enforcing tight clustering around category centers.
  2. Diffusion-controlled Semantic Translator (DST): In the subsequent stages, the DST translates these refined semantic features into the final target textual distribution.
Overview of the SeDA architecture, featuring the Progressive Feature Interaction Network (PFIN) and the bi-stage diffusion modules (DSL and DST).

Progressive Feature Interaction Network (PFIN)

The standard UNet architecture used in generative diffusion is replaced by the Progressive Feature Interaction Network (PFIN). The PFIN employs cross-attention mechanisms to fuse visual and textual information at each step of the diffusion process, ensuring a smooth transition across the modality gap. Notably, since SeDA follows the LUPI paradigm, the text data and diffusion process are only required during training; the visual backbone operates independently during inference with no added computational overhead.

Experimental Setup

The effectiveness of SeDA was evaluated across three diverse benchmarks:

  • VIREO Food-172: A domain-specific image dataset containing 172 food categories with corresponding text.
  • NUS-WIDE: A multi-label web image dataset with 81 categories and textual tags.
  • MSRVTT: A video dataset used for single-class video prediction based on captions.

Results and Discussion

SeDA consistently outperformed nine competing alignment frameworks, including CLIP and C2KD, across all tested datasets.

Quantitative Performance

On the VIREO Food-172 dataset using a ViT-B/16 backbone, SeDA achieved a Top-1 Accuracy of 89.19%. This represents a significant improvement over the baseline ViT (85.37%) and the previous state-of-the-art alignment method (87.97%).

Qualitative Analysis and Visualization

T-SNE visualizations confirm that SeDA effectively merges visual and textual distributions, whereas traditional L2-norm alignment leaves a distinct gap. Furthermore, SeDA excels at disambiguating confusing classes. For instance, in the VIREO dataset, baseline models often confuse "Braised beef with brown sauce" with "noodles" due to background noise like chopsticks. SeDA successfully extracts core semantic features, leading to correct classification.

Visualization of feature alignment via t-SNE (a), confusion matrices (b), and qualitative examples (c) showing SeDA's ability to disambiguate visually similar classes.

Conclusion

SeDA demonstrates that the progressive, multi-step nature of diffusion models is highly effective for cross-modal alignment in discriminative tasks. By introducing an intermediate semantic space, the framework smooths the transition between heterogeneous modalities. Future work will focus on fine-grained alignment to further improve accuracy in complex visual scenarios by mapping specific image regions to individual words.

Summary updated

BibTeX
@inproceedings{li2025semantic,
  author = {Zixuan Li and Lei Meng and Guoqing Chao and Wei Wu and Xiaoshuo Yan and Yimeng Yang and Zhuang Qi and Xiangxu Meng},
  title = {{Semantic-Space-Intervened Diffusive Alignment for Visual Classification}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/169},
}
Figure