FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens

Christian Schlarmann, Francesco Croce, Nicolas Flammarion, Matthias HeinarXiv 2025 · DOI · arXiv

Introduction

While dual-encoder models like CLIP have revolutionized vision-language alignment, they inherently struggle with multimodal queries—inputs that combine both image and text (e.g., an image of a dog plus the instruction "make it look like it's on the left"). Traditional approaches rely on "late fusion," where unimodal encoders process inputs separately before merging high-level features. This often results in the loss of fine-grained structural and spatial information. FuseLIP addresses this by proposing an "early fusion" architecture where modalities interact from the very first layer.

Methodology

FuseLIP shifts the paradigm by treating images as a sequence of discrete tokens, effectively translating visual data into a synthetic language that a Transformer can process natively.

Discrete Image Tokenization

Instead of using raw image patches, FuseLIP utilizes a frozen TiTok tokenizer to convert an image into 128 discrete tokens. This allows the model to handle images and text within a unified framework, treating visual tokens similarly to words.

Single-Encoder Architecture

By representing both modalities as token sequences, they can be concatenated and fed into a single Transformer encoder. This "early fusion" enables the attention mechanism to model cross-modal interactions at every depth of the network, allowing text tokens to attend to image tokens and vice versa from the outset.

Comparison of multimodal fusion architectures: (a) Score-level fusion, (b) MagicLens (embedding-level), and (c) FuseLIP (token-level early fusion).

Training Objectives

The model is optimized using a dual objective:

  1. Sigmoid Contrastive Loss (SigLIP): To align the multimodal embeddings in a shared space.
  2. Masked Multimodal Modeling (MMM): A generative task where the model predicts masked tokens in both text and image sequences. This forces the model to learn richer structural representations and cross-modal dependencies.

Experimental Setup and Datasets

The researchers trained FuseLIP on standard datasets (CC3M and CC12M), augmented with specialized tasks designed to test multimodal reasoning:

  • Text-Guided Image Transformation (TGIT): Matching a query (image + transformation instruction) to a correctly modified target image.
  • Visual Grounding (VG-Crop & OI-Crop): Identifying specific image regions based on text descriptions.
  • Spatial Reasoning (OI-Pos): Distinguishing positional relationships, such as identifying an object on the left versus the right.
  • CC3M-VQA: A Visual Question Answering dataset generated using Llama-3 to convert captions into QA pairs.

Key Results

FuseLIP demonstrates significant improvements over late-fusion baselines, particularly in tasks requiring structural understanding:

  • Text-Guided Transformations: On the CC12M-TGIT benchmark, FuseLIP-B achieved 94.2% accuracy, significantly outperforming the best late-fusion baseline, SigLIP-MLF (81.0%).
  • Spatial Reasoning: In the OI-Pos task, FuseLIP-B reached 68.9%, compared to approximately 48% for late-fusion models.
  • Parameter Efficiency: Despite having fewer trainable parameters due to the frozen image tokenizer, FuseLIP remains competitive on standard unimodal benchmarks like ImageNet, proving it does not sacrifice general alignment for specialized multimodal skills.

Conclusion and Limitations

FuseLIP demonstrates that early fusion via discrete tokenization preserves critical visual structure that late-fusion models typically discard. While the model faces challenges such as higher inference latency due to longer sequence lengths and a dependency on the quality of the discrete tokenizer, it establishes a robust framework for complex multimodal retrieval and reasoning. Future work will explore scaling these architectures to larger datasets like LAION.

Summary updated

Abstract

Contrastive language-image pre-training aligns features of text-image pairs in a common latent space via distinct encoders for each modality. While this approach achieves impressive performance in several zero-shot tasks, it cannot natively handle multimodal inputs, i.e., encoding image and text into a single feature vector. As a remedy, it is common practice to use additional modules to merge the features extracted by unimodal encoders. In this work, we present FuseLIP, a new architecture for multimodal embedding. Leveraging recent progress in discrete image tokenizers, we propose to use a single transformer model operating on a unified vocabulary of text and image tokens. This early fusion approach allows the different modalities to interact at each depth of encoding and obtain richer representations compared to common late fusion. We collect new datasets for multimodal pre-training and evaluation, designing challenging tasks for multimodal encoders. We show that FuseLIP outperforms late fusion approaches in several multimodal and unimodal embedding tasks.
BibTeX
@misc{schlarmann2025fuselip,
  author = {Christian Schlarmann and Francesco Croce and Nicolas Flammarion and Matthias Hein},
  title = {{FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens}},
  year = {2025},
  eprint = {2506.03096},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.03096},
}
Figure