Abstract
Cross-modal learning often struggles with the "modality gap"—the fundamental difference in distribution and structure between visual and textual data. Traditional one-step projection methods frequently result in ambiguous feature alignment, leading to significant inter-class confusion. We present SeDA (Semantic-Space-Intervened Diffusive Alignment), a novel framework that utilizes a bi-stage diffusion process to bridge this gap. By introducing an intermediate semantic space and a Progressive Feature Interaction Network (PFIN), SeDA iteratively refines visual features into the textual domain. Experimental results across multiple benchmarks, including VIREO Food-172 and NUS-WIDE, demonstrate that SeDA consistently outperforms state-of-the-art alignment methods, achieving a top-1 accuracy of 89.19% on food classification.
Introduction
In visual classification, leveraging textual category descriptions can significantly enhance model performance. However, the standard approach of projecting visual features into a textual space via a single-step mapping (e.g., MLP or Transformer layers) often fails due to modality heterogeneity. This "one-step" jump can produce "blurry" aligned features where visually similar categories, such as "fried chicken wings" and "fried chicken drumsticks," become indistinguishable in the target space.
As illustrated in Figure 1, traditional methods (a) often fail to resolve overlaps in the feature space. In contrast, SeDA (b) introduces a multi-step diffusion process that gradually migrates features through a semantic buffer, ensuring a more precise alignment and reducing inter-class ambiguity.
Methodology: The SeDA Framework
The core of SeDA is the replacement of abrupt projections with a gradual, multi-step diffusion process. The framework is optimized through a bi-stage strategy:
1. Diffusion-Controlled Semantic Learner (DSL)
In the initial stages of the reverse diffusion process, the DSL maps visual features to a shared Semantic Space. This stage utilizes a structural consistency loss to cluster features around category centers, effectively filtering out low-level visual noise and focusing on high-level semantic relationships.
2. Diffusion-Controlled Semantic Translator (DST)
In the subsequent stages, the DST transforms these refined semantic representations into the specific distribution of the textual modality. This two-stage approach allows the model to find "neutral ground" before committing to the final textual representation.
3. Progressive Feature Interaction Network (PFIN)
The PFIN serves as the architectural backbone, as shown in Figure 2. Unlike standard generative diffusion models that remove Gaussian noise, PFIN employs a cross-attention mechanism to progressively inject textual information into the features at every step of the reverse process.
Experimental Evaluation
The authors evaluated SeDA on three diverse benchmarks: VIREO Food-172 (110,241 food images), NUS-WIDE (over 200,000 web images), and MSRVTT (10,000 video clips).
Key Results
SeDA demonstrated superior performance compared to existing state-of-the-art alignment methods like CLIP and MoMKE. On the VIREO Food-172 dataset, SeDA achieved a Top-1 accuracy of 89.19%, a substantial improvement over the baseline ViT (85.37%).
Qualitative analysis confirms that SeDA is particularly effective at "semantic disambiguation," successfully separating categories that traditional methods conflate due to visual similarity. Furthermore, the framework proved robust across various backbones (ResNet-50, ViT, RepVGG) and relatively insensitive to hyperparameter variations in loss weights and diffusion steps.
Discussion and Future Work
The primary innovation of SeDA lies in using the Markovian reverse process of diffusion models as a "bridge" for feature alignment rather than for image generation. This iterative nature acts as a stabilizer for high-variance cross-modal mapping.
While SeDA significantly reduces inter-class confusion, challenges remain in extremely fine-grained classification tasks. Additionally, the iterative nature of the diffusion process introduces higher computational overhead during inference compared to single-step projections. Future research will focus on optimizing these iterative steps and exploring even more fine-grained semantic modeling to improve accuracy in highly complex scenarios.
Summary updated