FALCON: Actively Decoupled Visuomotor Policies for Loco-Manipulation with Foundation-Model-Based Coordination

Chengyang He, Ge Sun, Yue Bai, Junkai Lu, Jiadong Zhao, Guillaume SartorettiarXiv 2025 · DOI · arXiv

Introduction

Controlling a legged robot to move and manipulate objects simultaneously—known as loco-manipulation—is a significant challenge in robotics. Most contemporary approaches attempt to train a single, "monolithic" policy to manage all degrees of freedom simultaneously. However, these models often suffer from observation mismatch: the quadruped base requires proprioceptive data for balance and navigation, while the robotic arm relies on fine-grained visual cues (RGB) for precision tasks. Fusing these heterogeneous data streams into a single neural network frequently leads to "interference," resulting in instability or poor precision. Furthermore, monolithic models often fail to generalize to out-of-distribution (OOD) scenarios where the robot's starting configuration is unfamiliar.

Comparison of FALCON with existing loco-manipulation frameworks, highlighting its unique combination of foundation models, learning-based control, and decentralized architecture.

Methodology

The authors propose FALCON (FoundAtion-model-guided decoupled LoCO-maNipulation), a framework built on the principle of Active Decoupling. Instead of a single controller, FALCON employs two specialized diffusion policies: one for the quadruped base and another for the arm.

To ensure these independent policies remain coordinated, FALCON utilizes a frozen Vision-Language Model (VLM), specifically a CLIP backbone, as a semantic coordinator. This model encodes multi-view RGB images, robot states, and language instructions into a shared latent embedding that conditions both policies. A key component is the Phase-Progress Head, which uses human-interpretable text prompts to estimate task progress in a zero-shot manner, providing a temporal scaffold for long-horizon tasks. Additionally, a Coordination-Aware Contrastive Loss is applied during training to ensure that the latent space aligns compatible arm and base actions.

The FALCON architecture: A Vision-Language Model coordinates decoupled diffusion policies for the arm and quadruped base by processing multi-modal observations into shared latent embeddings.

Experimental Setup

FALCON was evaluated using a Unitree Go2 quadruped equipped with an Airbot arm. The training set consisted of 50 teleoperated demonstrations. Testing involved two primary tasks:

  1. Precise Manipulation: Navigating to a cabinet, opening a drawer, and placing a toy inside.
  2. Mobile Manipulation: Navigating while holding an object, placing it, and closing the drawer.

To test robustness, the robot was evaluated from various starting regions (Left, Center, Right) to assess its ability to handle distribution shifts.

Key Results and Insights

FALCON demonstrated significant improvements over centralized baselines:

  • Superior Success Rates: In mobile manipulation tasks, FALCON achieved a 100% success rate across all starting positions, whereas the centralized ACT baseline failed completely (0% success) due to collisions.
  • Robustness to Distribution Shift: While monolithic models failed when moved away from training starting points, FALCON maintained high performance, proving the benefits of decoupling for generalization.
  • Human-in-the-Loop Compatibility: The decoupled nature allows for seamless transitions between autonomous and manual control. In tests where a human controlled the base while the arm remained autonomous, the system maintained a 100% success rate.

Conclusion and Future Work

FALCON demonstrates that decoupling locomotion and manipulation, linked by a semantic foundation model, provides a more robust and generalizable solution than whole-body controllers. By leveraging "internet-scale" knowledge from frozen VLMs, the system avoids the computational burden of training large-scale models from scratch. Future research will focus on automating phase prompting using Large Language Models (LLMs) and extending the framework to even longer-horizon household tasks.

Summary updated

BibTeX
@misc{he2025falcon,
  author = {Chengyang He and Ge Sun and Yue Bai and Junkai Lu and Jiadong Zhao and Guillaume Sartoretti},
  title = {{FALCON: Actively Decoupled Visuomotor Policies for Loco-Manipulation with Foundation-Model-Based Coordination}},
  year = {2025},
  eprint = {2512.04381},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.04381},
}
Figure