Heterogeneous Vision Encoders for Multimodal Large Language Models

This direction is split into other directions: heterogeneous vision encoder ensembling. Its last version stays readable and citable here.

Version 1 of 1102 papers125 conceptsfrozen 2026-10-04

Most Multimodal Large Language Models (MLLMs) rely on a single vision encoder—typically a CLIP-based model—to translate images into tokens the language model can understand. While CLIP is excellent at capturing high-level semantics (e.g., "a dog in a park"), it often suffers from "information loss" regarding fine-grained details, spatial coordinates, and document structures.

Recent research has shifted toward a multi-expert visual architecture, where a heterogeneous ensemble of specialized encoders provides a more comprehensive "visual vocabulary" to the language model.

The Complementarity Hypothesis

The core intuition behind this shift is that no single vision model can optimize for all visual tasks simultaneously. As Gregor Geigle and colleagues demonstrated in One does not fit all!, different architectures (region-based, grid-based, and patch-based) capture distinct and complementary features. A model trained on image-text contrastive learning (like CLIP) prioritizes global alignment but may be "blind" to the exact position of an object or the text within a document.

By combining these diverse perspectives, models can resolve complex inter-object relations that a single encoder might overlook. Gongjie Zhang and Wenbin An found that this multi-encoder strategy is critical for overcoming "relation blindness" in MLLMs, allowing them to better understand spatial and comparative interplay between objects.

Building a Visual Expert Team

Instead of scaling up a single general-purpose encoder, researchers like Longhui Wei propose the IVE (Incorporating Visual Experts) framework. This approach supplements the standard semantic encoder with specialized experts for:

  • Low-level textures: Capturing fine-grained details often discarded by high-level models.
  • Document structures: Specialized encoders for OCR and layout understanding.
  • Structural knowledge: Feeding object coordinates and detected text directly into the LLM.

Similarly, the Eagle family of models, developed by Min Shi and Fuxiao Liu, systematically explores this design space. They found that integrating a diverse set of vision experts—such as those focused on detection, segmentation, and OCR—provides synergistic benefits that reduce object-level hallucinations and improve performance on resolution-sensitive tasks.

Divergent Pre-training Objectives

The power of these ensembles comes not just from different architectures, but from different pre-training objectives. In BRAVE, Oğuzhan Fatih Kar and others show that combining contrastive learning (CLIP) with masked image modeling (DINOv2) significantly reduces visual blind spots.

While CLIP provides the semantic "what," self-supervised models like DINOv2 provide the geometric "where." This combination allows the model to capture both abstract concepts and fine-grained local geometry simultaneously. This is further echoed in the work of Jihoon Chung, whose MERV framework uses a "team" of four specialized encoders to capture spatial detail, temporal motion, and semantic alignment for video understanding.

Integration: From Concatenation to Routing

How do we actually feed these multiple streams of information into the LLM?

  1. Channel Concatenation: Shi et al. discovered that simple channel concatenation of diverse vision experts is surprisingly effective, often outperforming complex fusion architectures.
  2. Adaptive Projectors: Models like LLaVA-Octopus by Boyuan Sun use an instruction-driven "router" to dynamically select the best visual features for a specific task, such as choosing a high-resolution projector for document analysis but a temporal projector for action recognition.
  3. Training-Free Fusion: Jinwu Hu introduced VisionFuse, which demonstrates that even without additional training, concatenating visual tokens from different encoders within the same model family can boost perception performance.

By moving from a "universal" vision encoder to a heterogeneous mixture of experts, MLLMs can finally bridge the gap between high-level reasoning and low-level perception, enabling more robust performance in specialized domains like medical imaging, document understanding, and complex 3D scene analysis.

Papers in this version

Showing 60 of 102 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_heterogeneous_vision_encoders_for_multim_19f8c4b4,
  title = {Heterogeneous Vision Encoders for Multimodal Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/heterogeneous-vision-encoders-for-multimodal-large-language-models/rdr_19f8c4b4fbb647a9aa872ca9ae9ed3cc},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_heterogeneous_vision_encoders_for_multim_19f8c4b4_v1,
  title = {Heterogeneous Vision Encoders for Multimodal Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/heterogeneous-vision-encoders-for-multimodal-large-language-models/rdr_19f8c4b4fbb647a9aa872ca9ae9ed3cc/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}