Semantic-Space-Intervened Diffusive Alignment for Visual Classification

Zixuan Li, Lei Meng, Guoqing Chao, Wei Wu, Yimeng Yang, Xiaoshuo Yan, Zhuang Qi, Xiangxu Meng

Abstract

Cross-modal learning often struggles with the "modality gap"—the fundamental difference in distribution and structure between visual and textual data. Traditional one-step projection methods frequently result in ambiguous feature alignment, leading to significant inter-class confusion. We present SeDA (Semantic-Space-Intervened Diffusive Alignment), a novel framework that utilizes a bi-stage diffusion process to bridge this gap. By introducing an intermediate semantic space and a Progressive Feature Interaction Network (PFIN), SeDA iteratively refines visual features into the textual domain. Experimental results across multiple benchmarks, including VIREO Food-172 and NUS-WIDE, demonstrate that SeDA consistently outperforms state-of-the-art alignment methods, achieving a top-1 accuracy of 89.19% on food classification.

Introduction

In visual classification, leveraging textual category descriptions can significantly enhance model performance. However, the standard approach of projecting visual features into a textual space via a single-step mapping (e.g., MLP or Transformer layers) often fails due to modality heterogeneity. This "one-step" jump can produce "blurry" aligned features where visually similar categories, such as "fried chicken wings" and "fried chicken drumsticks," become indistinguishable in the target space.

Comparison between traditional one-step alignment and the proposed SeDA method.

As illustrated in Figure 1, traditional methods (a) often fail to resolve overlaps in the feature space. In contrast, SeDA (b) introduces a multi-step diffusion process that gradually migrates features through a semantic buffer, ensuring a more precise alignment and reducing inter-class ambiguity.

Methodology: The SeDA Framework

The core of SeDA is the replacement of abrupt projections with a gradual, multi-step diffusion process. The framework is optimized through a bi-stage strategy:

1. Diffusion-Controlled Semantic Learner (DSL)

In the initial stages of the reverse diffusion process, the DSL maps visual features to a shared Semantic Space. This stage utilizes a structural consistency loss to cluster features around category centers, effectively filtering out low-level visual noise and focusing on high-level semantic relationships.

2. Diffusion-Controlled Semantic Translator (DST)

In the subsequent stages, the DST transforms these refined semantic representations into the specific distribution of the textual modality. This two-stage approach allows the model to find "neutral ground" before committing to the final textual representation.

3. Progressive Feature Interaction Network (PFIN)

The PFIN serves as the architectural backbone, as shown in Figure 2. Unlike standard generative diffusion models that remove Gaussian noise, PFIN employs a cross-attention mechanism to progressively inject textual information into the features at every step of the reverse process.

Detailed architecture of the SeDA framework, highlighting the Progressive Feature Interaction Network (PFIN) and the bi-stage optimization.

Experimental Evaluation

The authors evaluated SeDA on three diverse benchmarks: VIREO Food-172 (110,241 food images), NUS-WIDE (over 200,000 web images), and MSRVTT (10,000 video clips).

Key Results

SeDA demonstrated superior performance compared to existing state-of-the-art alignment methods like CLIP and MoMKE. On the VIREO Food-172 dataset, SeDA achieved a Top-1 accuracy of 89.19%, a substantial improvement over the baseline ViT (85.37%).

Qualitative analysis confirms that SeDA is particularly effective at "semantic disambiguation," successfully separating categories that traditional methods conflate due to visual similarity. Furthermore, the framework proved robust across various backbones (ResNet-50, ViT, RepVGG) and relatively insensitive to hyperparameter variations in loss weights and diffusion steps.

Discussion and Future Work

The primary innovation of SeDA lies in using the Markovian reverse process of diffusion models as a "bridge" for feature alignment rather than for image generation. This iterative nature acts as a stabilizer for high-variance cross-modal mapping.

While SeDA significantly reduces inter-class confusion, challenges remain in extremely fine-grained classification tasks. Additionally, the iterative nature of the diffusion process introduces higher computational overhead during inference compared to single-step projections. Future research will focus on optimizing these iterative steps and exploring even more fine-grained semantic modeling to improve accuracy in highly complex scenarios.

Summary updated

Abstract

Cross-modal alignment is an effective approach to improving visual classification. Existing studies typically enforce a one-step mapping that uses deep neural networks to project the visual features to mimic the distribution of textual features. However, they typically face difficulties in finding such a projection due to the two modalities in both the distribution of class-wise samples and the range of their feature values. To address this issue, this paper proposes a novel Semantic-Space-Intervened Diffusive Alignment method, termed SeDA, models a semantic space as a bridge in the visual-to-textual projection, considering both types of features share the same class-level information in classification. More importantly, a bi-stage diffusion framework is developed to enable the progressive alignment between the two modalities. Specifically, SeDA first employs a Diffusion-Controlled Semantic Learner to model the semantic features space of visual features by constraining the interactive features of the diffusion model and the category centers of visual features. In the later stage of SeDA, the Diffusion-Controlled Semantic Translator focuses on learning the distribution of textual features from the semantic space. Meanwhile, the Progressive Feature Interaction Network introduces stepwise feature interactions at each alignment step, progressively integrating textual information into mapped features. Experimental results show that SeDA achieves stronger cross-modal feature alignment, leading to superior performance over existing methods across multiple scenarios.
BibTeX
@inproceedings{li2025semantic,
  author = {Zixuan Li and Lei Meng and Guoqing Chao and Wei Wu and Yimeng Yang and Xiaoshuo Yan and Zhuang Qi and Xiangxu Meng},
  title = {{Semantic-Space-Intervened Diffusive Alignment for Visual Classification}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/169},
}
Figure