Abstract
Cross-modal alignment typically relies on direct, one-step mappings between visual and textual features. However, the inherent structural differences between these modalities often lead to suboptimal alignment and semantic confusion. This paper introduces Semantic-Space-Intervened Diffusive Alignment (SeDA), a novel framework that leverages a bi-stage diffusion process to progressively align visual features with textual distributions through an intermediate "semantic space." By decomposing the alignment into a multi-step denoising and translation process, SeDA significantly enhances the discriminative power of visual backbones. Experimental results on image and video datasets demonstrate that SeDA consistently outperforms existing alignment methods.
Introduction
In the realm of visual classification, the Learning Using Privileged Information (LUPI) paradigm has shown that incorporating textual descriptions during training can substantially improve model performance. The core challenge lies in cross-modal alignment: forcing visual features to match the distribution of textual features.
Existing approaches, such as distance metric minimization or knowledge distillation, typically attempt this bridge in a single, abrupt step. Because vision and text are highly heterogeneous—differing in data structure, feature ranges, and class distributions—this "one-shot" projection often fails to capture the nuances of the target distribution. This leads to "semantic ambiguities," where visually similar but semantically distinct classes remain indistinguishable to the model.
Methodology: SeDA Framework
To address the limitations of one-step mapping, SeDA utilizes the multi-step Markov process characteristic of Diffusion Models. Instead of generating images from noise, SeDA progressively denoises and translates visual features into the textual feature space.
Bi-Stage Diffusion Process
The framework operates through two primary modules that utilize a modality-shared semantic space as an intermediate bridge:
- Diffusion-controlled Semantic Learner (DSL): During the initial stages of the diffusion process, the DSL strips away irrelevant, low-level visual noise. It maps visual features into the intermediate semantic space by enforcing tight clustering around category centers.
- Diffusion-controlled Semantic Translator (DST): In the subsequent stages, the DST translates these refined semantic features into the final target textual distribution.
Progressive Feature Interaction Network (PFIN)
The standard UNet architecture used in generative diffusion is replaced by the Progressive Feature Interaction Network (PFIN). The PFIN employs cross-attention mechanisms to fuse visual and textual information at each step of the diffusion process, ensuring a smooth transition across the modality gap. Notably, since SeDA follows the LUPI paradigm, the text data and diffusion process are only required during training; the visual backbone operates independently during inference with no added computational overhead.
Experimental Setup
The effectiveness of SeDA was evaluated across three diverse benchmarks:
- VIREO Food-172: A domain-specific image dataset containing 172 food categories with corresponding text.
- NUS-WIDE: A multi-label web image dataset with 81 categories and textual tags.
- MSRVTT: A video dataset used for single-class video prediction based on captions.
Results and Discussion
SeDA consistently outperformed nine competing alignment frameworks, including CLIP and C2KD, across all tested datasets.
Quantitative Performance
On the VIREO Food-172 dataset using a ViT-B/16 backbone, SeDA achieved a Top-1 Accuracy of 89.19%. This represents a significant improvement over the baseline ViT (85.37%) and the previous state-of-the-art alignment method (87.97%).
Qualitative Analysis and Visualization
T-SNE visualizations confirm that SeDA effectively merges visual and textual distributions, whereas traditional L2-norm alignment leaves a distinct gap. Furthermore, SeDA excels at disambiguating confusing classes. For instance, in the VIREO dataset, baseline models often confuse "Braised beef with brown sauce" with "noodles" due to background noise like chopsticks. SeDA successfully extracts core semantic features, leading to correct classification.
Conclusion
SeDA demonstrates that the progressive, multi-step nature of diffusion models is highly effective for cross-modal alignment in discriminative tasks. By introducing an intermediate semantic space, the framework smooths the transition between heterogeneous modalities. Future work will focus on fine-grained alignment to further improve accuracy in complex visual scenarios by mapping specific image regions to individual words.
Summary updated