Single-Stream Multi-Level Alignment for Vision-Language Pretraining

Vijay Kumar b g

TL;DR

This paper introduces SIMLA, a novel single-stream vision-language pretraining framework that achieves multi-level alignment between images and text. By employing symmetric cross-modality reconstruction and pseudo-labeled keyword prediction, SIMLA enables finer-grained understanding and outperforms existing methods on various downstream tasks with improved data efficiency.

What Problem Does This Paper Address?

Current state-of-the-art vision-language pretraining methods often rely on dual-stream architectures. While effective for global alignment, these approaches struggle to capture fine-grained semantic correspondences between image regions and text. Supervised methods that achieve finer alignment require extensive, costly annotations. This paper addresses the need for a self-supervised approach that can learn rich, multi-level alignments between vision and language without relying on object detectors or dense annotations, leading to better performance and data efficiency.

Methods

SIMLA proposes a single-stream transformer-only architecture that allows for earlier and more integrated interactions between image and language modalities. This contrasts with dual-stream approaches that process modalities separately before late fusion. The core of SIMLA's innovation lies in its multi-task learning objective, which includes:

  • Symmetric Cross-Modality Reconstruction (XMM): This task masks tokens from one modality (e.g., image patches or text tokens) and uses information from the other modality to reconstruct the masked tokens. This encourages fine-grained alignment between image regions and corresponding text segments.
  • Pseudo-Labeled Keyword Prediction (PSL): This task leverages the attention mechanisms of the multimodal encoder to identify salient keywords in a caption. It then uses a momentum encoder to generate pseudo-labels for concepts present in the image but potentially missing from the caption. The visual encoder is trained to predict these pseudo-labels, forcing it to learn semantic concepts crucial for grounding text to image regions.

These tasks, combined with standard image-text matching (ITM) and contrastive learning (ITC) losses, enable alignment at multiple levels: global, patch-level, and conceptual.

Datasets

SIMLA is trained on a combined dataset of 4 million image-text pairs from:

  • Conceptual Captions
  • SBU Captions
  • COCO
  • Visual Genome

For evaluation, the paper uses standard benchmarks:

  • Image-Text Retrieval: MSCOCO, Flickr30k, FashionGen
  • Visual Question Answering (VQA): Not explicitly detailed, but mentioned as a downstream task.
  • Visual Entailment: Not explicitly detailed, but mentioned as a downstream task.
  • Visual Grounding: RefCOCO+

Key Insights and Ideas

  • Single-stream for early interaction: By using a single transformer stack, SIMLA allows for earlier and more integrated fusion of visual and textual information, potentially leading to richer representations than late-fusion dual-stream models.
  • Multi-level alignment is crucial: The paper demonstrates that aligning at global, patch, and conceptual levels is more effective than relying solely on global alignment.
  • Reconstruction as a powerful alignment tool: The symmetric cross-modality reconstruction task effectively bridges the gap between image patches and text tokens, enabling fine-grained understanding.
  • Pseudo-labels for concept grounding: The keyword prediction task, using pseudo-labels from a momentum model, allows the visual encoder to learn to represent abstract concepts, even when captions are underspecified.
  • Data efficiency through richer supervision: The additional tasks provide stronger supervisory signals, allowing SIMLA to achieve competitive performance with less training data compared to prior methods.

Key Results

SIMLA achieves state-of-the-art results on several downstream tasks, including:

  • Zero-shot Image-Text Retrieval: Outperforms CLIP and ALIGN, despite being trained on significantly less data. For example, on Flickr30k, SIMLA achieves 91.9% TR@1 and 78.1% IR@1.
  • Fine-tuned Image-Text Retrieval: Shows substantial improvements over ALBEF (e.g., 3% increase on MSCOCO text retrieval Rank-1) and OSCAR.
  • Weakly-Supervised Visual Grounding: Outperforms ALBEF on RefCOCO+ TestA (67.4% vs. 65.9%) and TestB (47.5% vs. 46.3%).
  • Data Efficiency: SIMLA's zero-shot retrieval accuracy rises more smoothly and quickly during training compared to ALBEF, indicating better learning from fewer training steps.
  • Inference Speed: SIMLA is substantially faster at inference time (7.2 pairs/second) compared to models like UNITER/VILLA/OSCAR (approx. 1.1 pairs/second), due to its efficient single-stream architecture.

Limitations and Future Work

  • Computational Cost: While SIMLA is more data-efficient, the pre-training still requires significant computational resources (e.g., 30 epochs on 8 NVIDIA A100 GPUs with a batch size of 512).
  • Pseudolabel Quality: The paper notes that pseudolabels are derived from language tokens present in captions, and while they aim to capture cross-modal concepts, they might not always perfectly represent the image content, especially for abstract concepts. Future work could explore more sophisticated methods for pseudolabel generation or refinement.
  • Further Exploration of Multi-level Alignment: The paper demonstrates the effectiveness of its multi-level alignment strategy. Future research could investigate other combinations or types of alignment tasks to further enhance performance.
  • Broader Applications: While SIMLA shows strong performance on retrieval and grounding, exploring its application to other vision-language tasks like VQA and image captioning could reveal further benefits.

Summary updated

Abstract

Recent progress in large-scale vision-language pre-training has shown the importance of aligning the visual and text modalities for downstream vision-language tasks. Many methods use a dual-stream architecture that fuses visual tokens and language tokens after representation learning, which aligns only at a global level and cannot extract finer-scale semantics. In contrast, we propose a single stream model that aligns the modalities at multiple levels: i) instance level, ii) fine-grained patch level, iii) conceptual semantic level. We achieve this using two novel tasks: symmetric cross-modality reconstruction and a pseudo-labeled key word prediction. In the former part, we mask the input tokens from one of the modalities and use the cross-modal information to reconstruct the masked token, thus improving fine-grained alignment between the two modalities. In the latter part, we parse the caption to select a few key words and feed it together with the momentum encoder pseudo signal to self-supervise the visual encoder, enforcing it to learn rich semantic concepts that are essential for grounding a textual token to an image region. We demonstrate top performance on a set of Vision-Language downstream tasks such as zero-shot/fine-tuned image/text retrieval, referring expression, and VQA. We also demonstrate how the proposed models can align the modalities at multi
BibTeX
@inproceedings{g2022single,
  author = {Vijay Kumar b g},
  title = {{Single-Stream Multi-Level Alignment for Vision-Language Pretraining}},
  booktitle = {European Conference on Computer Vision},
  year = {2022},
  doi = {10.1007/978-3-031-20059-5\_42},
}
Figure