Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment

Mustafa Shukor, Guillaume Couairon, Matthieu Cord

Abstract

Vision and Language Pretraining (VLP) has become the prevalent approach for tackling multimodal downstream tasks. However, the current trend toward ever-larger models and massive pretraining datasets is computationally unsustainable and often excludes academic laboratories with limited resources. In this work, we propose ViCHA (Visual Concepts and Hierarchical Alignment), a framework that efficiently exploits input data to boost learning through three key innovations: a hierarchical cross-modal alignment loss, a self-supervised masked image modeling scheme, and the use of image-level "Visual Concepts" to guide the image encoder. Despite being pretrained on four times less data than competing models, ViCHA outperforms existing approaches on benchmarks including Image-Text Retrieval, VQA, Visual Reasoning, and Visual Grounding.

Introduction

The state-of-the-art in Vision-Language Pretraining is currently defined by a "computational headlong rush." Models like CLIP and ALBEF rely on brute-force scaling, utilizing multi-billion-image datasets to bridge the gap between visual and textual modalities. While effective, this approach is inefficient and creates a barrier for researchers without access to industrial-scale compute.

ViCHA (Visual Concepts and Hierarchical Alignment) challenges this paradigm. By focusing on the quality of the learning strategy rather than the quantity of data, ViCHA demonstrates that smarter architectural choices—specifically early-stage semantic guidance and multi-layer alignment—can yield superior performance with a fraction of the training data.

The ViCHA Framework

ViCHA introduces three core methodological improvements to the standard transformer-based VLP architecture, which typically consists of a Vision Transformer (ViT), a text encoder (BERT), and a multimodal fusion transformer.

1. Visual Concepts (VCs) as Semantic Prompts

One of the primary challenges in VLP is the semantic gap between raw pixels and abstract text. ViCHA addresses this by extracting high-level labels (e.g., "chef knife," "lettuce") from captions using a scene graph parser. These labels are then filtered using a pretrained CLIP model to identify the most relevant concepts for a specific image.

These "Visual Concepts" are prepended as tokens to the Vision Transformer (ViT). They act as visual prompts, guiding the encoder to focus on semantically significant regions of the image. By introducing these concepts early, the model finds it easier to align a caption with a known word like "knife" than with an unorganized patch of pixels.

2. Hierarchical Image-Text Contrastive (H-ITC) Loss

Standard VLP models usually perform alignment only at the final output layer. ViCHA introduces a Hierarchical Image-Text Contrastive loss that aligns the class tokens of the vision and text encoders across multiple intermediate layers. This ensures that the model synchronizes both low-level structural features and high-level semantics throughout the encoding process. The authors found that aligning the final six layers of both encoders provided the optimal balance for downstream performance.

3. Unimodal Masked Image Modeling (U-MIM)

To further exploit the internal structure of visual data, ViCHA incorporates a self-supervised reconstruction task. Borrowing from Masked Autoencoder (MAE) techniques, the model learns to reconstruct masked-out image patches. Unlike some previous VLP frameworks where masked modeling was thought to hinder multimodal alignment, ViCHA’s unimodal implementation (U-MIM) significantly strengthens the vision encoder’s representations, providing better "fuel" for the multimodal fusion stage.

Experimental Results

Data Efficiency

The researchers trained ViCHA in a low-data regime to prove its efficiency. The primary pretraining dataset consisted of approximately 1.1 million images (from COCO, Visual Genome, and SBU). For comparison, many baseline models utilize 4 million images or more.

Performance Benchmarks

Despite the data disadvantage, ViCHA achieved state-of-the-art results across several tasks:

  • Image-Text Retrieval: On the Flickr30K benchmark, ViCHA outperformed the ALBEF baseline by +3.6% in R@1. In zero-shot settings, it showed a massive gain of +7.88% RSUM, indicating highly robust and generalizable representations.
  • Visual Question Answering (VQA): ViCHA outperformed models like UNITER and VILLA, which were trained on nearly four times as much data (4M+ images).
  • Visual Reasoning and Grounding: The model showed superior performance on NLVR² and RefCOCO+, demonstrating that its hierarchical alignment helps in complex spatial and logical reasoning.

Conclusion

ViCHA demonstrates that the path to better Vision-Language models does not solely depend on increasing dataset size. By utilizing Visual Concepts to bridge the semantic gap and Hierarchical Alignment to synchronize features across layers, ViCHA achieves high efficiency and competitive performance. This framework provides a more sustainable and accessible path for multimodal research, proving that algorithmic innovation can effectively compensate for limited data and compute resources.

Summary updated

Abstract

Vision and Language Pretraining has become the prevalent approach for tackling multimodal downstream tasks. The current trend is to move towards ever larger models and pretraining datasets. This computational headlong rush does not seem reasonable in the long term to move toward sustainable solutions, and de facto excludes academic laboratories with limited resources. In this work, we propose a new framework, dubbed ViCHA, that efficiently exploits the input data to boost the learning by: (a) a new hierarchical cross-modal alignment loss, (b) new self-supervised scheme based on masked image modeling, (c) leveraging image-level annotations, called Visual Concepts, obtained with existing foundation models such as CLIP to boost the performance of the image encoder. Although pretrained on four times less data, our ViCHA strategy outperforms other approaches on several downstream tasks such as Image-Text Retrieval, VQA, Visual Reasoning, Visual Entailment and Visual Grounding. The code will be made publicly available here: https://github.com/mshukor/ViCHA
BibTeX
@inproceedings{shukor2022efficient,
  author = {Mustafa Shukor and Guillaume Couairon and Matthieu Cord},
  title = {{Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment}},
  booktitle = {BMVC},
  year = {2022},
  url = {https://openreview.net/forum?id=TRINMohH2E},
}
Figure