Libra-VLA: Achieving Learning Equilibrium via Asynchronous Coarse-to-Fine Dual-System

Yifei Wei, Linqing Zhong, Yi Liu, Yuxiang Lu, Xindong He, Maoqing Yao, Guanghui RenarXiv 2026 · arXiv

TL;DR

Libra-VLA introduces a coarse-to-fine dual-system architecture that bridges the gap between high-level semantic reasoning and low-level physical control in robots. By decoupling action generation into discrete macro-intents and continuous micro-pose alignments, it achieves a "learning equilibrium" that yields state-of-the-art manipulation success while drastically reducing inference latency through asynchronous execution.

What Problem Does This Paper Address?

Current Vision-Language-Action (VLA) models typically rely on a monolithic architecture. They take in visual and textual instructions and map them directly to high-frequency motor commands in a flat, non-hierarchical fashion. This forces a single network to simultaneously handle abstract cognitive reasoning (e.g., "where should I move the arm?") and high-precision geometric tracking (e.g., "how do I align the gripper down to the millimeter?").

This monolithic approach creates a massive semantic-actuation gap, overloading the network's representational capacity. While some researchers have tried temporal decomposition (breaking tasks into sequential sub-goals), the robot still faces the same cross-modal complexity at every single execution step.

Methods

Libra-VLA solves this by introducing a Hybrid Action Space operated by a decoupled dual-system architecture:

Figure 1: Comparison of action generation paradigms. (a) Discrete autoregressive approaches discretize actions into massive bins. (b) Continuous diffusion approaches directly predict continuous signals. (c) Our proposed Libra-VLA operates in a hybrid action space, where discrete coarse bins representing macro-intents serve as anchors for continuous fine actions, naturally aligning with the inherent hierarchical characteristics.
  1. System 2: The Semantic Planner (Low Frequency): Powered by a Large Vision-Language Model (InternVL2.5-2B) augmented with a Parallel Coarse-Action Head. It discretizes continuous actions into a very small number of uniform bins ($N=10$). Instead of trying to be precise, it predicts a chunk of upcoming "macro-directional intents" (e.g., broad directional trajectories).
  2. System 1: The Action Refiner (High Frequency): A Diffusion Transformer equipped with its own independent, high-resolution visual encoder (SigLIP). It takes the coarse intent from System 2 as a geometric anchor and generates fine-grained continuous action residuals to handle precise, interaction-rich adjustments.

Asynchronous Execution Strategy

To keep the robot responsive, the two systems communicate via a First-In-First-Out (FIFO) Intent Buffer. The expensive Semantic Planner runs periodically in bursts, predicting an extended horizon of coarse actions. Meanwhile, the lightweight Action Refiner continuously pops intents from the buffer and executes at the robot's native control frequency, bypassing the time-consuming VLM inference loop at every step.

Figure 2: Architectural Overview of Libra-VLA. The framework adopts a Coarse-to-Fine generation paradigm, explicitly modeling actions within a Hybrid Action Space across two specialized phases. (Left) System 2: Semantic Planner runs at a low frequency, employing a VLM backbone augmented with a Parallel Coarse-Action Head to predict discrete macro-directional intents (coarse action tokens) within the discrete semantic subspace. (Right) System 1: Action Refiner runs at a high frequency, utilizing a diffusion transformer equipped with an independent visual encoder to synthesize continuous micro-pose alignments (fine action tokens) within the continuous geometric subspace. Crucially, the two systems are bridged via an asynchronous execution strategy managed by an intent buffer. Controlled by the Horizon Expansion Factor (M), the Semantic Planner anticipates an extended coarse action chunk in a single inference pass. Subsequently, the Action Refiner iteratively retrieves the specific coarse action slice corresponding to the current timestep from the buffer as the condition for its high-frequency generation.

Datasets

The authors evaluated Libra-VLA across simulation benchmarks and real-world hardware:

  • LIBERO Benchmark: Includes four suites evaluating different distribution shifts: LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-Long.
  • LIBERO-Plus: A robustness benchmark evaluating models against seven distinct real-world nuisances (camera jitter, lighting changes, background textures, etc.).
  • Real-World Evaluation: Conducted on an AgiBot G1 humanoid robot platform across three long-horizon tasks: "Wipe Stain", "Pour Water", and "Make Sandwich".

Key Insights and Ideas

  • The "Learning Equilibrium" (Inverted-U Curve): The most striking finding is that model performance follows an inverted-U curve relative to action discretization granularity ($N$).
  • If $N$ is too small ($N=2$), the coarse intent lacks information density, forcing the refiner to do all the heavy lifting.
  • If $N$ is too large ($N \ge 50$), the planner gets bogged down trying to predict exact metrics, degrading overall accuracy.
  • The sweet spot—the "Libra point"—occurs at $N=10$, where the learning difficulty is perfectly balanced between the two subsystems.
Figure 3: Performance trend with respect to the coarse bin size N.
  • Structural Decoupling Prevents "Visual Shortcuts": Simply adding a second visual encoder to a monolithic baseline actually hurts performance because the model learns lazy visual shortcuts. True performance gains only unlock when you structurally constrain the network through a coarse-to-fine hierarchy.
  • Dynamic Curriculum Training: To prevent the Action Refiner from becoming over-reliant on perfect planner predictions, the authors used a curriculum that transitions from ground-truth teacher forcing to sampling from the planner's actual predicted distribution. This teaches the refiner to inherently correct minor directional errors.

Key Results

  • LIBERO Benchmark: Libra-VLA set a new state-of-the-art with an average success rate of 97.2%, outperforming monolithic models like OpenVLA (76.5%) and continuous diffusion baselines like $\pi_0$ (94.1%).
  • Robustness (LIBERO-Plus): In zero-shot transfer tasks (testing on unseen perturbations without fine-tuning), Libra-VLA achieved 79.5% success, outperforming the next best baseline by nearly 10%.
  • Latency Reduction: By decoupling the VLM inference, Libra-VLA slashed average inference latency from 220ms down to 104ms (a 52.7% reduction) depending on the horizon expansion factor, enabling real-time control.

Limitations and Future Work

While highly robust, the authors noted a slight downward trend in success rate as the planning horizon expands (i.e., when the VLM runs less frequently). This is due to error accumulation during prolonged open-loop execution.

Currently, the communication protocol is strictly one-way: the planner pushes to the buffer, and the refiner executes sequentially. The authors note that future iterations will benefit from a real-time confidence estimation mechanism to allow the refiner to dynamically flag suboptimal anchors and trigger immediate replanning when things go awry.

Summary updated

Abstract

Vision-Language-Action (VLA) models are a promising paradigm for generalist robotic manipulation by grounding high-level semantic instructions into executable physical actions. However, prevailing approaches typically adopt a monolithic generation paradigm, directly mapping visual-linguistic features to high-frequency motor commands in a flat, non-hierarchical fashion. This strategy overlooks the inherent hierarchy of robotic manipulation, where complex actions can be naturally modeled in a Hybrid Action Space, decomposing into discrete macro-directional reaching and continuous micro-pose alignment, severely widening the semantic-actuation gap and imposing a heavy representational burden on grounding high-level semantics to continuous actions. To address this, we introduce Libra-VLA, a novel Coarse-to-Fine Dual-System VLA architecture. We explicitly decouple the learning complexity into a coarse-to-fine hierarchy to strike a training equilibrium, while simultaneously leveraging this structural modularity to implement an asynchronous execution strategy. The Semantic Planner predicts discrete action tokens capturing macro-directional intent, while the Action Refiner conditions on coarse intent to generate high-frequency continuous actions for precise alignment. Crucially, our empirical analysis reveals that performance follows an inverted-U curve relative to action decomposition granularity, peaking exactly when the learning difficulty is balanced between the two sub-systems. With the asynchronous design, our approach offers a scalable, robust, and responsive solution for open-world manipulation.
BibTeX
@misc{wei2026libra,
  author = {Yifei Wei and Linqing Zhong and Yi Liu and Yuxiang Lu and Xindong He and Maoqing Yao and Guanghui Ren},
  title = {{Libra-VLA: Achieving Learning Equilibrium via Asynchronous Coarse-to-Fine Dual-System}},
  year = {2026},
  eprint = {2604.24921},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.24921},
}

Loading related content...

Figure