Unifying Visual and Semantic Feature Spaces with Diffusion Models for Enhanced Cross-Modal Alignment

Yuze Zheng, Zixuan Li, Xiangxian Li, Jinxing Liu, Yuqing Wang, Xiangxu Meng, Lei Meng

Introduction

Image classification in real-world environments is frequently hampered by "visual noise," such as lighting variations, cluttered backgrounds, and diverse camera angles. While multimodal learning—leveraging textual descriptions to regularize visual models—offers a solution, the fundamental differences between pixel-based visual features and word-based semantic features create a significant "heterogeneity bottleneck." Traditional alignment methods often struggle to bridge this gap effectively using simple distance metrics.

Conceptual overview of MARNet's alignment and diffusion-based reconstruction process.

The MARNet Framework

To address these challenges, the authors propose the Multimodal Alignment and Reconstruction Network (MARNet). MARNet utilizes a dual-module approach to unify visual and semantic feature spaces, treating semantic information as "privileged information" available during training to guide the learning of robust visual representations.

Detailed architecture of MARNet, showcasing the overall flow and the specific Embedding Matching Alignment (EMA) and Cross-Modal Diffusion Reconstruction (CDR) modules.

Embedding Matching Alignment (EMA)

The EMA module focuses on instance-wise alignment using a contrastive learning strategy. It calculates the cosine similarity between image-text pairs, pulling positive pairs (matching image and text) closer together in the latent space while pushing negative pairs apart. This establishes a baseline for cross-modal correspondence.

Cross-Modal Diffusion Reconstruction (CDR)

The core innovation of MARNet is the CDR module, which leverages a conditional diffusion model to reconstruct semantic features from visual inputs.

  • Forward Process: Gaussian noise is incrementally added to the semantic (text) embeddings ($x_s$).
  • Reverse Process: The model performs denoising to reconstruct the original semantic features, using the visual features ($x_v$) as a guiding condition.

This mechanism forces the network to identify the "core" shared information between modalities. By reconstructing text from noise using the image as a map, the model effectively filters out visual noise that does not contribute to semantic meaning, resulting in a "smoothed" and more discriminative feature space.

Experimental Evaluation

MARNet was evaluated on two large-scale food classification datasets, consistently outperforming both standalone visual models and existing cross-modal alignment frameworks:

  • Vireo-Food172: Achieved a Top-1 Accuracy of 88.1%, surpassing the Vision Transformer (ViT-B/16) baseline of 85.4% and the previous best alignment method (ITA) at 87.8%.
  • Ingredient-101: Reached 89.0% Top-1 Accuracy, outperforming the baseline (88.3%) and other state-of-the-art methods like TEAM and SDM.

Ablation studies confirmed that MARNet is model-agnostic, providing immediate performance boosts when integrated into various backbones, including ResNet and Swin-Transformer. Furthermore, t-SNE visualizations indicate that MARNet creates significantly tighter and more distinct category clusters compared to standard visual encoders.

Conclusion and Future Work

By repurposing diffusion models for feature engineering rather than image generation, MARNet provides a novel approach to cross-modal alignment. The diffusion process acts as a powerful regularizer, smoothing visual features and aligning them more closely with semantic distributions. While the authors acknowledge that sampling noise in the diffusion process can occasionally lead to unstable samples, the overall impact on classification accuracy is positive. Future work will focus on reducing this noise interference to further preserve the integrity of the original visual information.

Summary updated

BibTeX
@inproceedings{zheng2024unifying,
  author = {Yuze Zheng and Zixuan Li and Xiangxian Li and Jinxing Liu and Yuqing Wang and Xiangxu Meng and Lei Meng},
  title = {{Unifying Visual and Semantic Feature Spaces with Diffusion Models for Enhanced Cross-Modal Alignment}},
  booktitle = {ICANN},
  year = {2024},
  url = {https://openreview.net/forum?id=7wGWjN9I5x},
}
Figure