Continuous Vision-Language-Action Co-Learning with Semantic-Physical Alignment for Behavioral Cloning

Xiuxiu Qi, Yu Yang, Jiannong Cao, Luyao Bai, Chongshan Fan, Chengtai Cao, Hongpeng WangarXiv 2025 · DOI · arXiv

Introduction

Language-Conditioned Manipulation (LCM) aims to enable robots to execute complex tasks based on natural language instructions, such as "place the cup on the shelf." Behavioral Cloning (BC) has emerged as a popular approach, where robots learn by mimicking expert demonstrations. However, traditional BC methods often struggle with two critical issues: compounding errors and physical/semantic discontinuity. Compounding errors occur when small initial deviations lead to catastrophic failures over time, a phenomenon known as covariate shift. Physical discontinuity manifests as jerky, non-smooth robotic movements, while semantic-physical misalignment occurs when a model fails to dynamically shift its attention as a task progresses—for instance, continuing to focus on a grasped object rather than its intended destination.

The CCoL Framework

To address these challenges, we introduce CCoL (Continuous Vision-Language-Action Co-Learning), a framework designed to treat robot motion as a continuous flow and synchronize linguistic instructions with physical actions through dynamic attention.

Figure 1: Overview of the CCoL framework, illustrating the integration of context-aware representation learning, multimodal continuous co-learning (MCC), and cross-modal semantic-physical alignment (CSA).

The architecture consists of three primary innovations:

  1. Multimodal Continuous Co-Learning (MCC): Instead of modeling robot states as discrete steps, CCoL utilizes Neural Ordinary Differential Equations (NeuralODEs). By learning the derivative of state changes, the model treats the robot's internal state (proprioception) as a continuous trajectory, naturally producing smoother and more physically feasible motions.
  2. Cross-Modal Semantic-Physical Alignment (CSA): This module employs bidirectional cross-attention to "anchor" specific words to corresponding visual pixels and physical trajectories. Unlike static fusion methods, CSA dynamically updates the robot's focus at every timestep, allowing it to re-align its linguistic understanding with physical progress (e.g., aligning the word "press" with a downward trajectory).
  3. Hybrid Loss Function: Training incorporates a standard imitation loss alongside a discontinuity penalty. This penalty enforces dynamical consistency, ensuring that predicted motions adhere to physical constraints like smooth acceleration.

Experimental Evaluation

CCoL was evaluated across diverse simulation environments and real-world hardware:

  • Aloha MuJoCo: Testing bimanual coordination, such as transferring objects between robotic arms.
  • RLBench: A multi-scenario benchmark involving tasks like "LampOn" and "OpenBottle."
  • Franka Kitchen: A long-horizon environment requiring multi-stage task execution.
  • Real-World Testing: Validated on a 7-DoF Franka Emika Panda robot for tasks like pen lifting and cube sliding under noisy, unseen conditions.

Key Results and Insights

The framework achieved an average 8.0% improvement over state-of-the-art methods like ACT (Action Chunking Transformer) and Diffusion Policy (DP). Notably, CCoL demonstrated a 19.2% relative improvement in complex bimanual insertion tasks. In the long-horizon Franka Kitchen environment, success rates improved by 17.2% over previous benchmarks.

Figure 5: Comparison of velocity and acceleration profiles across different models. CCoL significantly reduces fluctuations, achieving smoother trajectories compared to discrete baselines.

A core finding is that "smoothness is safety." By leveraging NeuralODEs, CCoL suppressed high-frequency jitter, reducing velocity fluctuations by 30.8% and acceleration fluctuations by 32.7% compared to discrete baselines. This physical consistency is crucial for reducing compounding errors and ensuring reliable real-world performance. In physical tests, the robot maintained an 86.7% success rate in cube placement, even when objects were placed in previously unseen positions.

Conclusion and Future Work

CCoL represents a significant step toward robust, language-conditioned robotic manipulation by bridging the gap between discrete linguistic commands and continuous physical execution. While the NeuralODE solvers are computationally more intensive, they remain capable of real-time inference (~67Hz). Future work will explore integrating Large Language Models (LLMs) to enhance open-world reasoning and handling more abstract human instructions.

Summary updated

BibTeX
@misc{qi2025continuous,
  author = {Xiuxiu Qi and Yu Yang and Jiannong Cao and Luyao Bai and Chongshan Fan and Chengtai Cao and Hongpeng Wang},
  title = {{Continuous Vision-Language-Action Co-Learning with Semantic-Physical Alignment for Behavioral Cloning}},
  year = {2025},
  eprint = {2511.14396},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.14396},
}
Figure