SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs

Yuanyang Yin, Yaqi Zhao, Yajie Zhang, Yuanxing Zhang, Ke Lin, Jiahao Wang, Xin Tao, Pengfei Wan, Wentao Zhang, Feng Zhao2025 · DOI

Abstract: Yuanyang Yin, Yaqi Zhao, Yajie Zhang, Yuanxing Zhang, Ke Lin, Jiahao Wang, Xin Tao, Pengfei Wan, Wentao Zhang, Feng Zhao. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

2025.

Introduction

In the development of Multimodal Large Language Models (MLLMs) such as LLaVA, the standard architecture involves a vision encoder (e.g., CLIP) and an adapter that translates visual features into tokens compatible with a Large Language Model (LLM). Traditionally, this alignment is achieved through image-level supervision, where the model is trained to generate captions for entire images.

However, this coarse approach imposes significant "alignment taxes." First, it leads to semantic information distortion, where individual visual patches are mapped to semantically unrelated words (e.g., misidentifying blueberries as "bluejeans"). Second, it creates a substantial modality representation gap, forcing the LLM to expend its internal reasoning capacity on "translating" messy visual inputs rather than solving the target task. This paper introduces Supervised Embedding Alignment (SEA) to address these issues by aligning visual features at the individual token level.

Methodology: Token-Level Supervision

SEA moves beyond global image descriptions to bridge the modality gap during pre-training. The framework consists of several key components:

  1. Semantic Label Extraction: Utilizing a well-aligned vision-language model and a massive 4-million-word vocabulary, the system identifies relevant semantic labels for every individual visual patch.
  2. Similarity-Weighted Sampling: To account for the continuous nature of visual data (where a patch might represent "fur," "kitten," and "ear" simultaneously), SEA employs a weighted sampling strategy.
  3. Localized Sampling: To maintain batch diversity and prevent confusion from redundant data, the model samples only one patch from every $k \times k$ window for contrastive loss calculation.
  4. The SEA Loss ($L_a$): A parallel contrastive alignment loss pulls adapter-processed visual tokens closer to the LLM’s native word embeddings.
Figure 1: Comparison between traditional image-level pretraining and SEA's token-level alignment, alongside a qualitative example of improved perception.

Crucially, SEA adds zero inference overhead and requires no additional training data, simply extracting more granular information from existing datasets.

Bridging the Modality Representation Gap

A core contribution of SEA is the reduction of the distance in vector space between visual tokens and the LLM's native text tokens. By forcing visual patches to map directly into the LLM’s native word-embedding space, the model reduces the computational burden on the LLM's transformer layers.

Figure 2: t-SNE visualization of the visual representation space, showing how SEA-LLaVA brings visual tokens significantly closer to the LLM's input space compared to standard LLaVA.

As shown in the t-SNE analysis, SEA-LLaVA tokens (green) cluster much more closely with the LLM's native input space (gold) than standard LLaVA tokens (orange). This tighter alignment allows the LLM to process visual information as if it were native text, preserving its original linguistic capabilities.

Experimental Results and Insights

The researchers developed the SEA-PRIME family of models, testing them across eight major benchmarks including VQAv2, GQA, and ScienceQA.

  • Force Multiplier for Small Models: While larger models (13B+) can sometimes compensate for poor alignment through sheer parameter count, smaller models benefit immensely. Applying SEA to a Gemma-2B backbone resulted in a 7.61% average performance gain.
  • State-of-the-Art Performance: SEA-PRIME (Llama3-8B) achieved an 83.1 on VQAv2 and 87.4 on POPE, significantly outperforming larger baselines and reducing object hallucinations.
  • The TACS Metric: The authors introduced the Token Alignment Consistency Score (TACS). Their findings show a direct correlation: as TACS increases, object hallucination rates (measured by POPE) decrease.
  • Cross-Encoder Transfer: SEA successfully transferred semantic understanding from CLIP-derived labels to a DINOv2 encoder, suggesting the method can enhance vision encoders not originally trained on text.

Conclusion

SEA demonstrates that token-level alignment is a highly efficient way to improve MLLM performance without increasing model size or data requirements. By reducing the "alignment tax," SEA enables even small-scale models to achieve high-level reasoning and perception. Future work will explore dynamic label selection and extending this framework to audio and video modalities.

Summary updated

BibTeX
@misc{yin2025sea,
  author = {Yuanyang Yin and Yaqi Zhao and Yajie Zhang and Yuanxing Zhang and Ke Lin and Jiahao Wang and Xin Tao and Pengfei Wan and Wentao Zhang and Feng Zhao},
  title = {{SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.55},
}
Figure