Unifying Visual and Semantic Feature Spaces with Diffusion Models for Enhanced Cross-Modal Alignment

Yuze Zheng, Zixuan Li, Xiangxian Li, Jinxing Liu, Yuqing Wang, Xiangxu Meng, Lei Meng

Introduction

Image classification in real-world environments is frequently hampered by "visual noise," such as lighting variations, cluttered backgrounds, and diverse camera angles. While multimodal learning—leveraging textual descriptions to regularize visual models—offers a solution, the fundamental differences between pixel-based visual features and word-based semantic features create a significant "heterogeneity bottleneck." Traditional alignment methods often struggle to bridge this gap effectively using simple distance metrics.

Conceptual overview of MARNet's alignment and diffusion-based reconstruction process.

The MARNet Framework

To address these challenges, the authors propose the Multimodal Alignment and Reconstruction Network (MARNet). MARNet utilizes a dual-module approach to unify visual and semantic feature spaces, treating semantic information as "privileged information" available during training to guide the learning of robust visual representations.

Detailed architecture of MARNet, showcasing the overall flow and the specific Embedding Matching Alignment (EMA) and Cross-Modal Diffusion Reconstruction (CDR) modules.

Embedding Matching Alignment (EMA)

The EMA module focuses on instance-wise alignment using a contrastive learning strategy. It calculates the cosine similarity between image-text pairs, pulling positive pairs (matching image and text) closer together in the latent space while pushing negative pairs apart. This establishes a baseline for cross-modal correspondence.

Cross-Modal Diffusion Reconstruction (CDR)

The core innovation of MARNet is the CDR module, which leverages a conditional diffusion model to reconstruct semantic features from visual inputs.

  • Forward Process: Gaussian noise is incrementally added to the semantic (text) embeddings ($x_s$).
  • Reverse Process: The model performs denoising to reconstruct the original semantic features, using the visual features ($x_v$) as a guiding condition.

This mechanism forces the network to identify the "core" shared information between modalities. By reconstructing text from noise using the image as a map, the model effectively filters out visual noise that does not contribute to semantic meaning, resulting in a "smoothed" and more discriminative feature space.

Experimental Evaluation

MARNet was evaluated on two large-scale food classification datasets, consistently outperforming both standalone visual models and existing cross-modal alignment frameworks:

  • Vireo-Food172: Achieved a Top-1 Accuracy of 88.1%, surpassing the Vision Transformer (ViT-B/16) baseline of 85.4% and the previous best alignment method (ITA) at 87.8%.
  • Ingredient-101: Reached 89.0% Top-1 Accuracy, outperforming the baseline (88.3%) and other state-of-the-art methods like TEAM and SDM.

Ablation studies confirmed that MARNet is model-agnostic, providing immediate performance boosts when integrated into various backbones, including ResNet and Swin-Transformer. Furthermore, t-SNE visualizations indicate that MARNet creates significantly tighter and more distinct category clusters compared to standard visual encoders.

Conclusion and Future Work

By repurposing diffusion models for feature engineering rather than image generation, MARNet provides a novel approach to cross-modal alignment. The diffusion process acts as a powerful regularizer, smoothing visual features and aligning them more closely with semantic distributions. While the authors acknowledge that sampling noise in the diffusion process can occasionally lead to unstable samples, the overall impact on classification accuracy is positive. Future work will focus on reducing this noise interference to further preserve the integrity of the original visual information.

Summary updated

Abstract

Image classification models often demonstrate unstable performance in real-world applications due to variations in image information, driven by differing visual perspectives of subject objects and lighting discrepancies. To mitigate these challenges, existing studies commonly incorporate additional modal information matching the visual data to regularize the model’s learning process, enabling the extraction of high-quality visual features from complex image regions. Specifically, in the realm of multimodal learning, cross-modal alignment is recognized as an effective strategy, harmonizing different modal information by learning a domain-consistent latent feature space for visual and semantic features. However, this approach may face limitations due to the heterogeneity between multimodal information, such as differences in feature distribution and structure. To address this issue, we introduce a Multimodal Alignment and Reconstruction Network (MARNet), designed to enhance the model’s resistance to visual noise. Importantly, MARNet includes a cross-modal diffusion reconstruction module for smoothly and stably blending information across different domains. Experiments conducted on two benchmark datasets, Vireo-Food172 and Ingredient-101, demonstrate that MARNet effectively improves the quality of image information extracted by the model. It is a plug-and-play framework that can be rapidly integrated into various image classification frameworks, boosting model performance.
BibTeX
@inproceedings{zheng2024unifying,
  author = {Yuze Zheng and Zixuan Li and Xiangxian Li and Jinxing Liu and Yuqing Wang and Xiangxu Meng and Lei Meng},
  title = {{Unifying Visual and Semantic Feature Spaces with Diffusion Models for Enhanced Cross-Modal Alignment}},
  booktitle = {ICANN},
  year = {2024},
  url = {https://openreview.net/forum?id=7wGWjN9I5x},
}
Figure