TL;DR
UNITE is a novel framework that consolidates multiple task-specific Vision-Language Model (VLM) specialists—like those for navigation, driving, and spatial reasoning—into a single, compact Mixture-of-Experts (MoE) model. By using a "shared-anchor one-bit factorization" technique, it achieves near-single-model storage (just 1.16× the footprint) while actually outperforming the original specialists on complex benchmarks.
What Problem Does This Paper Address?
Building an "embodied" agent—a robot or AI that can perceive and act in the physical world—requires a diverse set of skills: visual grounding, 3D spatial reasoning, navigation, and driving understanding. Currently, researchers typically fine-tune a separate VLM "specialist" for each task.
This leads to a difficult deployment trade-off:
- Separate Deployment: Storing four specialists (e.g., based on Qwen3-VL-8B) requires a massive 70.1 GB of storage.
- Static Merging: Averaging the weights of these specialists into one model reduces storage to 17.5 GB, but the task-specific updates often conflict, leading to "interference" and a significant drop in performance.
The authors tackle the challenge of combining these specialized capabilities into a single model that fits near the storage budget of a single specialist without losing their hard-won expertise.
Methods
UNITE (Unified Embodied Specialists) uses a three-stage pipeline to consolidate models:
- Shared-Backbone MoE Construction: The framework identifies that most task-specific knowledge resides in the Feed-Forward Network (FFN) blocks. It keeps the FFNs of each specialist as separate, routable "experts" while merging the remaining modules (like self-attention) into a shared backbone.
-
Shared-Anchor One-Bit Factorization: To keep storage low, UNITE doesn't store full-precision weights for every expert. Instead, it represents each expert weight matrix ($W_e$) as a shared floating-point anchor ($C$) plus expert-specific, row-scaled binary deviations ($S_e$ and $\alpha_e$): $$\hat{W}_e = C + \text{diag}(\alpha_e)S_e$$ Crucially, the anchor is optimized jointly with the binary signs and scales using closed-form alternating updates. This allows the anchor to adapt to the limitations of one-bit deviations, reducing reconstruction error by over 20% compared to using a fixed mean anchor.
-
Router Adaptation: A tiny linear router (initialized via SVD) is fine-tuned to select the best expert for each token. This step updates fewer than 0.6M parameters—less than 0.01% of the total model.
Datasets
The authors evaluated UNITE across a massive suite of 22 embodied understanding benchmarks and the R2R (Room-to-Room) navigation task:
- Grounding: RoboSpatial, RefSpatial, PointArena, RoboAfford, and ShareRobot.
- Embodied Spatial Reasoning: VSI-Bench, MMSI, MindCube, ScanQA, SQA3D, Scan2Cap, etc.
- Driving Understanding: NuScenes-QA, NuPlanQA, MAPLM, and LingoQA.
- Language Extension: They also tested the factorization on Llama-3.1-8B specialists from the MergeBench suite (covering math, coding, and safety).
Key Insights and Ideas
- Joint Optimization Matters: Most previous compression methods fix a "base" model and then calculate deviations. UNITE shows that treating the shared anchor as a variable to be optimized alongside the experts significantly improves the fidelity of the compressed model.
- Consolidation is Additive: Surprisingly, UNITE often surpasses the performance of the original source specialists. This is because token-level routing allows the model to "mix and match" expertise within a single forward pass—for example, using a grounding expert for one part of a scene and a spatial expert for another.
- One-Bit is Sufficient for Experts: For 21 out of 22 benchmarks, the one-bit factorization was nearly lossless compared to a full-precision MoE. This suggests that the "direction" of task-specific updates is more important than their exact magnitude.
Key Results
- Storage Efficiency: A four-expert UNITE model occupies 20.3 GB, which is 59.6% smaller than an uncompressed MoE and only 1.16× the size of a single dense specialist.
- Navigation Superiority: On the R2R Val-Unseen navigation task, UNITE achieved a 59.6% success rate, outperforming the original navigation specialist (57.5%) and all six static merging baselines (which peaked at 55.2%).
- 3D Scene Understanding: UNITE showed massive gains over source specialists, such as a +19.1 point improvement on the Scan2Cap benchmark.
Limitations and Future Work
The primary outlier in the results was VSI-Bench, a video-based spatial reasoning benchmark. UNITE's performance there dropped significantly compared to the uncompressed model. Further experiments revealed that the one-bit factorization seems to impair the model's ability to integrate evidence across many video frames (cross-frame evidence).
The authors suggest that future work could investigate using a larger bit budget for specific "bottleneck" layers or experts to recover this lost video-reasoning capability. Additionally, extending UNITE to Vision-Language-Action (VLA) models that directly output robot control signals is a promising next step.
Summary updated