UNITE: Consolidating Embodied VLM Specialists via Shared-Anchor One-Bit Factorization

Tao Tao, Anke Tang, Enneng Yang, Zhi Hou, Xiaogang WangarXiv 2026 · arXiv

TL;DR

UNITE is a novel framework that consolidates multiple task-specific Vision-Language Model (VLM) specialists—like those for navigation, driving, and spatial reasoning—into a single, compact Mixture-of-Experts (MoE) model. By using a "shared-anchor one-bit factorization" technique, it achieves near-single-model storage (just 1.16× the footprint) while actually outperforming the original specialists on complex benchmarks.

What Problem Does This Paper Address?

Building an "embodied" agent—a robot or AI that can perceive and act in the physical world—requires a diverse set of skills: visual grounding, 3D spatial reasoning, navigation, and driving understanding. Currently, researchers typically fine-tune a separate VLM "specialist" for each task.

This leads to a difficult deployment trade-off:

  1. Separate Deployment: Storing four specialists (e.g., based on Qwen3-VL-8B) requires a massive 70.1 GB of storage.
  2. Static Merging: Averaging the weights of these specialists into one model reduces storage to 17.5 GB, but the task-specific updates often conflict, leading to "interference" and a significant drop in performance.

The authors tackle the challenge of combining these specialized capabilities into a single model that fits near the storage budget of a single specialist without losing their hard-won expertise.

Methods

UNITE (Unified Embodied Specialists) uses a three-stage pipeline to consolidate models:

  1. Shared-Backbone MoE Construction: The framework identifies that most task-specific knowledge resides in the Feed-Forward Network (FFN) blocks. It keeps the FFNs of each specialist as separate, routable "experts" while merging the remaining modules (like self-attention) into a shared backbone.
  2. Shared-Anchor One-Bit Factorization: To keep storage low, UNITE doesn't store full-precision weights for every expert. Instead, it represents each expert weight matrix ($W_e$) as a shared floating-point anchor ($C$) plus expert-specific, row-scaled binary deviations ($S_e$ and $\alpha_e$): $$\hat{W}_e = C + \text{diag}(\alpha_e)S_e$$ Crucially, the anchor is optimized jointly with the binary signs and scales using closed-form alternating updates. This allows the anchor to adapt to the limitations of one-bit deviations, reducing reconstruction error by over 20% compared to using a fixed mean anchor.

  3. Router Adaptation: A tiny linear router (initialized via SVD) is fine-tuned to select the best expert for each token. This step updates fewer than 0.6M parameters—less than 0.01% of the total model.

Overview of the UNITE framework, showing how specialists are merged into a shared-backbone MoE with compressed experts.

Datasets

The authors evaluated UNITE across a massive suite of 22 embodied understanding benchmarks and the R2R (Room-to-Room) navigation task:

  • Grounding: RoboSpatial, RefSpatial, PointArena, RoboAfford, and ShareRobot.
  • Embodied Spatial Reasoning: VSI-Bench, MMSI, MindCube, ScanQA, SQA3D, Scan2Cap, etc.
  • Driving Understanding: NuScenes-QA, NuPlanQA, MAPLM, and LingoQA.
  • Language Extension: They also tested the factorization on Llama-3.1-8B specialists from the MergeBench suite (covering math, coding, and safety).

Key Insights and Ideas

  • Joint Optimization Matters: Most previous compression methods fix a "base" model and then calculate deviations. UNITE shows that treating the shared anchor as a variable to be optimized alongside the experts significantly improves the fidelity of the compressed model.
  • Consolidation is Additive: Surprisingly, UNITE often surpasses the performance of the original source specialists. This is because token-level routing allows the model to "mix and match" expertise within a single forward pass—for example, using a grounding expert for one part of a scene and a spatial expert for another.
  • One-Bit is Sufficient for Experts: For 21 out of 22 benchmarks, the one-bit factorization was nearly lossless compared to a full-precision MoE. This suggests that the "direction" of task-specific updates is more important than their exact magnitude.

Key Results

  • Storage Efficiency: A four-expert UNITE model occupies 20.3 GB, which is 59.6% smaller than an uncompressed MoE and only 1.16× the size of a single dense specialist.
  • Navigation Superiority: On the R2R Val-Unseen navigation task, UNITE achieved a 59.6% success rate, outperforming the original navigation specialist (57.5%) and all six static merging baselines (which peaked at 55.2%).
  • 3D Scene Understanding: UNITE showed massive gains over source specialists, such as a +19.1 point improvement on the Scan2Cap benchmark.
Fig. 1. Storage versus capability retention for four Qwen3-VL-8B specialists. Keeping them separate (gray) costs 70.1 GB; static merging (violet, Table I) cuts this to 17.5 GB, but every merging method then falls below the navigation specialist’s 57.5% R2R Val-Unseen success rate, reaching at best 55.2% (Table III). UNITE (blue) needs only 20.3 GB, 1.16× the single-specialist footprint, and surpasses the specialist at 59.6% success rate.

Limitations and Future Work

The primary outlier in the results was VSI-Bench, a video-based spatial reasoning benchmark. UNITE's performance there dropped significantly compared to the uncompressed model. Further experiments revealed that the one-bit factorization seems to impair the model's ability to integrate evidence across many video frames (cross-frame evidence).

The authors suggest that future work could investigate using a larger bit budget for specific "bottleneck" layers or experts to recover this lost video-reasoning capability. Additionally, extending UNITE to Vision-Language-Action (VLA) models that directly output robot control signals is a promising next step.

Summary updated

Abstract

Embodied capabilities such as visual grounding, spatial reasoning, navigation, and driving understanding are often developed in separate vision-language specialists fine-tuned from a common pretrained model.Deploying the specialists separately multiplies storage, whereas merging them into one dense model forces conflicting task updates to share parameters and loses part of each specialist's expertise.We introduce UNITE, a framework that consolidates such specialists into a compact shared-backbone mixture of experts. UNITE keeps the feed-forward block of each specialist as a routable expert and represents the expert bank through a shared-anchor one-bit factorization: a shared floating-point anchor plus expert-specific, row-scaled binary deviations.The anchor is optimized jointly with the scales and signs through closed-form alternating updates, using only source weights and no calibration data.Router adaptation then adjusts token-level expert selection with the backbone and experts frozen, updating fewer than $0.6$M parameters ($<0.01\%$ of the $8.8$B active parameters). Built from four Qwen3-VL-8B specialists, the complete UNITE model occupies $20.3$ GB, $1.16\times$ the storage of one dense specialist and $59.6\%$ below the uncompressed MoE. Across $22$ benchmarks and R2R Val-Unseen, UNITE matches six static merging methods in mean retention of best-specialist performance, leads all of them on Scan2Cap, ScanQA, and all four navigation metrics, and surpasses every source specialist on all five 3D scene understanding benchmarks (up to $+19.1$ on Scan2Cap) and on R2R navigation. Relative to the uncompressed MoE, factorization is nearly lossless on $21$ of $22$ benchmarks; the exception is VSI-Bench, where frame-count experiments suggest impaired use of cross-frame evidence.UNITE thus consolidates complementary embodied capabilities near single-model storage.
BibTeX
@misc{tao2026unite,
  author = {Tao Tao and Anke Tang and Enneng Yang and Zhi Hou and Xiaogang Wang and Dacheng Tao},
  title = {{UNITE: Consolidating Embodied VLM Specialists via Shared-Anchor One-Bit Factorization}},
  year = {2026},
  eprint = {2611.20767},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20767},
}
Figure