Rebalanced Vision-Language Retrieval Considering Structure-Aware Distillation

Yang Yang, Wenjuan Xi, Luping Zhou, Jinhui Tang

Abstract

Vision-Language Retrieval (VLR) typically aims to align images and text within a shared latent space. However, standard alignment techniques often suffer from "modal imbalance," where the higher information density of one modality (usually text) is degraded to match the lower density of the other (images). This paper introduces a Structure-Aware Distillation framework that employs independent single-modal teachers to regularize a cross-modal student. By preserving the internal geometric relationships of each modality, the proposed method ensures a more robust and balanced common space, significantly improving performance in cross-modal, single-modal, and mixed retrieval tasks.

Introduction

In traditional Vision-Language Retrieval, models are trained to map paired instances (e.g., an image and its caption) to the same point in a latent space. This approach assumes "modal balance"—that both modalities provide equivalent representation sufficiency. In reality, modalities are often imbalanced; a detailed caption may contain far more structured information than a noisy or low-resolution image. Forcing a "strong" modality to align perfectly with a "weak" one can destroy the high-quality internal structure of the former, leading to sub-optimal retrieval performance, especially for single-modal searches (like text-to-text) within the cross-modal model.

Methods

The authors propose a Multi-Granularity Distillation approach to mitigate this imbalance. The architecture consists of a Cross-Modal Student Model (based on X-VLM) and two specialized Single-Modal Teacher Models (one for vision, one for text).

The proposed Multi-Granularity Distillation framework, featuring single-modal teachers and a cross-modal student with structure-aware regularization.

The framework utilizes two primary distillation objectives:

  1. Representation-Level Distillation: A contrastive loss ensures the student's individual encoders produce representations aligned with their respective high-performing single-modal teachers.
  2. Structure-Aware Distillation: This objective regularizes the geometric consistency of the latent space. It calculates relational similarity matrices—mapping how every item in a batch relates to every other item—for both teachers and the student. The student is penalized if its internal "relational map" deviates from a fused version of the teachers' maps, weighted by a learnable parameter $\lambda$ that adaptively balances visual vs. textual structural importance.

Datasets and Evaluation

The model was rigorously evaluated on four major benchmarks:

  • MS-COCO (1K & 5K splits): Large-scale image-captioning dataset.
  • FLICKR30K: Descriptive image-text pairs from Flickr.
  • Vizwiz: A challenging dataset of images taken by people who are blind, characterized by significant visual noise and "weak" visual information.

Key Results

The proposed method, X-VLM*++, consistently outperformed state-of-the-art baselines across all evaluated benchmarks.

Performance comparison across MS-COCO, FLICKR30K, and Vizwiz datasets, demonstrating the superiority of the proposed method (X-VLM*++) over state-of-the-art baselines.

Key performance highlights include:

  • Cross-Modal Retrieval: On FLICKR30K, the model achieved 87.3% R@1 for Image-to-Text (I2T) and 73.4% for Text-to-Image (T2I).
  • Single-Modal Retrieval: Significant improvements were noted in Text-to-Text (T2T) retrieval (NDCG@10), proving that distillation prevents the degradation of the "strong" text modality during training.
  • Mixed Retrieval: The method showed superior ranking performance compared to models like BLIP and CYCLIP in scenarios requiring simultaneous retrieval of images and text.

Discussion and Insights

  • Preserving Structure over Point-Alignment: The core insight is that preserving the structure of data is more beneficial than forcing noisy point-to-point alignment when an information gap exists between modalities.
  • Geometric Symmetry: Structure-aware distillation leads to a "geometrically symmetric" latent space where relative distances between images mirror the relative distances between their corresponding texts.
  • Architectural Agnostic: The distillation module is "plug-and-play," showing performance gains when applied to various dual-encoder (e.g., SCAN) and transformer-based (e.g., ALBEF) architectures.

Limitations and Future Work

Performance is sensitive to batch size; while larger batches provide more relational data, they can also introduce noise. Additionally, the student's performance is inherently capped by the quality of the single-modal teachers. Future research will explore the theoretical bounds of modal imbalance and extend this framework to multi-modal tasks involving video and audio.

Summary updated

BibTeX
@article{yang2024rebalanced,
  author = {Yang Yang and Wenjuan Xi and Luping Zhou and Jinhui Tang},
  title = {{Rebalanced Vision-Language Retrieval Considering Structure-Aware Distillation}},
  journal = {IEEE Trans. Image Process.},
  year = {2024},
  doi = {10.1109/tip.2024.3518759},
}
Figure