Many recent multimodal language models see images through several different vision encoders at once. A CLIP-style encoder supplies meaning aligned with text. A self-supervised encoder such as DINOv2 supplies geometry. A segmentation model such as SAM supplies object boundaries. Systems that combine them are much better at fine detail, text in images, and avoiding hallucinated objects. The field disagrees, however, on a basic training decision: should these encoders be updated while the model learns to follow visual instructions? Eagle and Cambrian-1 report that unfreezing the encoders is essential. LEO reports that they must stay frozen, or their specialized skills erode. Both camps only tested the two extremes, all weights or none.
We propose to test the space between them with low-rank adaptation (LoRA). LoRA leaves each pretrained weight matrix untouched and learns a small additive update built from two thin matrices, so the size of the change is capped by a "rank" knob. Our claim is that per-encoder low-rank updates capture most of the accuracy that full unfreezing gives on fine-grained benchmarks while keeping each encoder's original abilities close to intact. Because the update is a separate add-on, each encoder can also be switched back to its untouched pretrained form whenever it is needed for its original task. We further predict that the benefit is uneven. Encoders that were never trained against language, such as SAM, should need and tolerate higher rank. Text-aligned CLIP-style encoders should need little or none. If so, encoder-specific LoRA could also replace Eagle's separate per-encoder "pre-alignment" training stage.
Core Claim
The freeze-versus-unfreeze contradiction is an artifact of measuring only two points on a curve that runs from "how much the encoder changes" to "how much the downstream model gains." We will trace that curve for each encoder in a heterogeneous ensemble and test three predictions. First, low-rank updates reach a better tradeoff than either extreme. Second, they reach a better tradeoff than the obvious way of making small full-weight updates, which is a reduced learning rate. Third, the best rank tracks how far an encoder starts from language alignment.
The third prediction and the comparison against a reduced learning rate carry the novelty. A result showing only that "LoRA lands between frozen and unfrozen" would be expected and uninteresting.
Relation to Prior Work
Three ensemble papers motivate the question by disagreeing. Eagle combines five encoders by stacking their features along the channel dimension. It first trains each non-language encoder alone with a frozen language model (its "pre-alignment" stage) and then fully unfreezes all encoders. Cambrian-1 compares over fifteen backbones, fuses them with a cross-attention aggregator, and also unfreezes. LEO (with its companion study Rethinking MoVE) pairs InternViT with SAM. It gives each encoder its own projector, alternates their tokens tile by tile, and keeps both encoders frozen. None of the three measures what happens to an encoder's original skill after tuning, so "drift" is asserted rather than quantified. We supply that measurement.
Two LoRA papers supply the method and its supporting evidence. CLIP-LoRA applies rank-2 updates to the attention projections of CLIP's image and text encoders. On 16-shot classification it beats prompt learning while training in 50 minutes instead of 15.5 hours. But that is one encoder doing classification, not an ensemble feeding a generative model. Kong et al. and Xu report that LoRA regularizes models against spurious features and preserves calibration. That is the property we hope extends to keeping encoder skills intact.
The closest work combines LoRA and multiple encoders for different goals:
- MoVE-KD distills several frozen teacher encoders into one student. It places a routed set of LoRA modules inside that student, so its LoRA serves compression, and the teachers are never adapted.
- Vision as LoRA removes external encoders entirely. It distills a single ViT into LoRA layers inside the language model.
- MixLoRA routes between low-rank factors in the language model to reduce interference between tasks. It was tested only with a single CLIP encoder.
All three leave the heterogeneous encoders themselves either absent or untouched. Our proposal adapts them directly and keeps the ensemble at inference time.
Research Program
We will build on LEO's design because it is the frozen extreme of the debate and is cheap enough to sweep. The base system uses InternViT-300M and SAM-L with independent projectors and tile-level interleaving. To test the alignment-distance hypothesis on a wider spread of encoders, we will run a second ensemble of CLIP, DINOv2 and Pix2Struct. These three span text-aligned, self-supervised, and document-parsing objectives, following Eagle's encoder choices.
For a frozen weight $W_0$ in an encoder's attention projections, LoRA adds an update $B_e A_e$, where $e$ indexes the encoder and $r_e$ is that encoder's rank. We sweep $r_e \in \{0, 2, 8, 32\}$ independently for each encoder, where 0 means frozen. Training uses LLaVA-595K for alignment and Eagle-1.8M for instruction tuning, the datasets named in the Eagle and LEO papers.
For every configuration we record two quantities. The first is downstream gain on the benchmarks below. The second is prior retention: how well the adapted encoder still performs its native function, scored by probes trained on the original frozen features and held fixed.
- For SAM and DINOv2, the probe is a linear segmentation readout on ADE20K and COCO images, kept separate from CV-Bench's test items.
- For CLIP, it is zero-shot accuracy on the 11-dataset few-shot classification suite used by CLIP-LoRA.
- For Pix2Struct, it is a text-recognition readout.
Plotting gain against retention gives a frontier for each encoder. The comparison of frontiers is the main result. As a secondary test, we drop Eagle's pre-alignment stage entirely and ask whether rank-adapted non-language encoders, trained jointly from the start, recover its effect.
Baselines
The do-nothing reference is the frozen ensemble with exactly LEO's recipe. This is the strongest frozen-camp competitor, because LEO reports DocVQA 80.1, ScienceQA 78.5 and MMBench 72.9 LEO with frozen encoders.
The strongest unfrozen competitor is full encoder fine-tuning during instruction tuning, as in Eagle. Eagle-X5 reports OCRBench 598 and MME perception 1605. Those figures come from a five-encoder system with its own data scale, so we will not compare against them directly. We will rerun full unfreezing inside our own ensembles.
Every arm shares the same data, the same number of optimizer steps, the same token budget and resolution scheme, and the same learning-rate search: five values per arm, chosen on a held-out split.
The baseline that isolates our key ingredient is low-learning-rate full fine-tuning. We will scale encoder learning rates so that the final distance $\lVert W - W_0 \rVert$ matches the LoRA arms. This tests whether the low-rank structure matters or only the size of the change.
Two further arms complete the comparison. One is LoRA on the language model alone, with encoders frozen, as a standard recipe. The other is that same arm with MixLoRA's conditional factor routing. MixLoRA reports MME 1509.61 against 1393.67 for standard LoRA, in a single-encoder setting.
Beating the field means two things. On OCRBench, MMVP and CV-Bench, a LoRA arm must sit at or above full unfreezing, within seed noise across three seeds. On every native probe, it must keep at least most of the gap closed back toward the frozen reference. And it must strictly dominate the norm-matched low-learning-rate arm on the gain-versus-retention frontier.
Why not just…?
Why not just keep the encoders frozen and invest in the connector, as LEO does? This is a serious threat. LEO's strong results come with frozen encoders, and post-adaptation projectors may already absorb most of the needed adaptation. Our rank-0 arm is exactly this recipe, and we will report the result honestly. If LoRA arms gain nothing on OCRBench, MMVP or CV-Bench over rank 0, the conclusion is that LEO was right and the gains Eagle and Cambrian-1 saw from unfreezing came from their connectors or data. That would settle the contradiction in LEO's favor and would itself be a useful finding.
Why not just unfreeze with a small learning rate? This is the most likely way the proposal fails, because small full-weight updates may preserve priors just as well. The norm-matched arm exists to decide it. If it matches LoRA's frontier, we will reframe the contribution around LoRA's practical advantages. The update can be detached to recover the exact pretrained encoder for its original task, the method uses far less optimizer memory, and it offers per-encoder rank control. We would stop claiming that the low-rank structure itself regularizes.
Why not just tune the language model with LoRA and leave the vision side alone? MixLoRA shows that language-side LoRA already gains a lot from routing. However, language-side updates cannot recover visual information an encoder discards. MMVP's single-encoder ceiling of 27.3% against BRAVE's 42.0% with more encoders BRAVE suggests the bottleneck is visual. Both language-side arms test this directly.
Why not just distill into one encoder? MoVE-KD answers a different question, inference cost rather than adaptation, and its reported gains are relative to LLaVA-1.5 (+3.1 MMBench). We treat it as complementary: a LoRA-adapted ensemble could serve as a better teacher. We do not run it as a competitor.
Decisive Experiments
The study is decisive on three counts.
The first is the shape of the per-encoder frontier against the norm-matched baseline. A clear separation means low-rank structure matters. Overlap means it is only update size.
The second is whether the best rank differs systematically by encoder. We expect CLIP near rank 0 to 2 and SAM or Pix2Struct higher. If ranks are flat across encoders, the alignment-distance hypothesis is wrong.
The third is whether removing pre-alignment costs Eagle-style setups accuracy once SAM and Pix2Struct carry LoRA. If it does not, a training stage can be dropped.
Benchmarks are OCRBench, POPE (random, popular and adversarial splits, F1), MMVP, CV-Bench, MME, DocVQA and ChartQA. CV-Bench and MMVP are the most diagnostic, because they isolate visual ability from language priors. Each result is reported over three seeds, together with the native-probe retention scores. Without those scores the drift question cannot be answered.
References
- Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
- Rethinking the Mixture of Vision Encoders Paradigm for Enhanced Visual Understanding in Multimodal LLMs
- Low-Rank Few-Shot Adaptation of Vision-Language Models
- Enhancing General Face Forgery Detection via Vision Transformer with Low-Rank Adaptation
- On the Calibration of Large Language Models and Alignment
- MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders
- Vision as LoRA
- Multimodal Instruction Tuning with Conditional Mixture of LoRA
- BRAVE: Broadening the Visual Encoding of Vision-Language Models