DeltaForge: A Post-hoc Mixture-of-Experts VLA for Multi-Domain Autonomous Driving

Bingzhang Wang, Mo Poorsartep, Yasaman EsfandiariarXiv 2026 · arXiv

TL;DR

DeltaForge is a new framework that merges multiple specialized autonomous driving models into a single, efficient "super-model" without needing their original training data. By using a clever mathematical trick (joint SVD) to find shared patterns across different driving domains, it achieves the performance of specialized experts at the computational cost of a single model.

What Problem Does This Paper Address?

In the world of autonomous driving, "one size fits all" rarely works. A model trained for urban Beijing struggles with off-road trails in the US or heavy trucking on European highways. Currently, operators take a base Vision-Language-Action (VLA) model and fine-tune it on their private data, creating a fragmented fleet of "domain experts."

Consolidating these experts into one versatile model is a nightmare for three reasons:

  1. Data Privacy: Operators often can't share their raw training data due to regulations.
  2. Heterogeneity: Merging a "truck" model with a "car" model often results in a "jack of all trades, master of none" that performs worse than either.
  3. Efficiency: Traditional Mixture-of-Experts (MoE) approaches, which route inputs to different sub-networks, are often too bulky and expensive to run in real-time on a vehicle.

Methods

The authors introduce DeltaForge, which uses a technique called Post-hoc Expert Factorization (PEF).

Instead of trying to merge the full weights of $N$ different models, DeltaForge looks at the updates (deltas) made during fine-tuning. It uses a joint Singular Value Decomposition (SVD) to decompose these updates into two parts:

  • A Shared Skill Basis: A frozen set of "building blocks" common to all driving tasks.
  • Tiny Domain-Specific Cores: Small matrices that tell the model how to combine those building blocks for a specific domain (e.g., "Urban China" or "Off-road ATV").
The DeltaForge workflow: from local fine-tuning to joint factorization and composition.

To make it work at runtime, they use two "blind" routers. Unlike standard LLMs that route every single word (token), DeltaForge routes at the clip level. It looks at the entire video clip and vehicle kinematics once to decide which experts to use for the whole trajectory. This is much more efficient and stable for driving.

Datasets

The researchers evaluated DeltaForge across a diverse range of environments and vehicles:

  • Urban: ONCE (China, right-hand traffic) and CoVLA (Japan, left-hand traffic).
  • Off-road: TartanDrive (ATVs in forests) and ORAD-3D (Cars in China).
  • Highway: TruckScenes (EU trucks) and PhysicalAI-AV (US/EU base domain).
  • Generalization/Onboarding: Torc TruckDrive (US trucks), ParkRecon3D (fisheye parking), and BeVIS (grayscale parking).

Key Insights and Ideas

  • Shared Structure in Diversity: Even though driving in Tokyo and off-roading in a forest seem different, the underlying weight updates to the neural network share significant structure. PEF exploits this to keep the model compact.
  • Clip-Level Routing is Key: In driving, the vehicle and environment don't change mid-sentence. Routing once per trajectory (clip) avoids the "noise" of token-level switching and keeps the math linear, meaning the experts can be mixed in parameter space with zero extra inference cost.
  • Decoupling Reasoning from Dynamics: By having one router for the Vision-Language Model (reasoning) and another for the Action Expert (dynamics), the model can mix-and-match. It might use "highway reasoning" but pair it with "truck dynamics" to handle a new trucking scenario.
  • "Anchor Repair": When a new domain is added, its projection onto the shared basis might be slightly lossy. A quick "repair" step using just 1% of the domain's data fixes this without needing to retrain the whole system.

Key Results

  • Accuracy: DeltaForge achieved a mean error (minADE@5s) of 1.0157m, which is remarkably close to the 0.9798m achieved by having six separate, massive models. It significantly outperformed standard weight-merging and federated learning baselines.
  • Efficiency: Adding a new domain adds only 42 million parameters—just 0.3% of the base model's size.
  • Zero-Shot Generalization: The model showed an "aha moment" when tested on unseen domains. For example, when presented with US highway trucking (which it wasn't trained on), it automatically routed the task to its EU trucking expert, dropping the error from 8.99m (base model) to 1.06m.
Architecture of the two-level routing system.

Limitations and Future Work

While DeltaForge is a major step forward, the authors note a few areas for growth:

  • Scaling: The current study uses a 10-billion parameter base. Scaling this to even larger foundation models remains to be seen.
  • Proxy Data Dependency: Although it only needs 1% of the data, it still requires some data for the "repair" and router training phases.
  • Basis Evolution: Currently, the shared skill basis is frozen after the initial experts are factorized. Future work could explore how to evolve this basis as more diverse domains are added over time.

In summary, DeltaForge provides a practical, privacy-preserving, and highly efficient way to build a "universal driver" from a fragmented ecosystem of specialized models.

Summary updated

Abstract

Fleets of driving vision-language-action (VLA) models are developed one domain at a time: each operator finetunes a shared base model on private data, producing specialized experts that are costly to consolidate into a single model. Weight merging loses accuracy as heterogeneous domains accumulate, and dense mixture-of-experts routing is prohibitively expensive to combine them. We present DeltaForge, to our knowledge the first post-hoc mixture-of-experts VLA for driving, which consolidates N independently finetuned per-domain models into a single multi-domain model from their weights and a small proxy set alone. When different operators finetune the same base model, their weight updates end up sharing a lot of structure. We exploit this by decomposing all the updates together using singular value decomposition (SVD), splitting each one into a shared part (the same across all domains) and a tiny domain-specific part. The router then only needs to work with those tiny domain-specific pieces, not the full expert weights, which is what keeps the approach efficient. Then two cooperating clip-level blind routers, one over the VLM reasoning experts and one over the flow-matching dynamics experts, are trained from soft labels on the proxy dataset. Across six diverse domains spanning urban, off-road, and highway driving on cars, trucks, and all-terrain vehicles (ATVs), DeltaForge approaches the accuracy of the per-domain experts and surpasses every merging, routing, centralized, and federated baseline at close to single-model parameter and compute cost. Adding a new domain requires only projecting its weight update onto the shared basis with no end-to-end retraining needed. The resulting model also generalizes to domain combinations it was never explicitly trained on.
BibTeX
@misc{wang2026deltaforge,
  author = {Bingzhang Wang and Mo Poorsartep and Yasaman Esfandiari},
  title = {{DeltaForge: A Post-hoc Mixture-of-Experts VLA for Multi-Domain Autonomous Driving}},
  year = {2026},
  eprint = {2611.11856},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11856},
}
Figure