Fast Visuomotor Policy for Robotic Manipulation

Jingkai Jia, Tong Yang, Xueyao Chen, Chenhuan Liu, Wenqiang Zhang

Abstract

Robotic imitation learning faces a significant challenge in handling multimodal expert demonstrations while maintaining real-time inference speeds. Current solutions like Diffusion Models offer high precision but suffer from high computational costs due to iterative denoising. We present Energy Policy, a framework that achieves the performance of diffusion-based methods in a single forward pass. By utilizing an Energy MLP with Adaptive Layer Normalization (adaLN) and an energy score objective, our method runs up to 70x faster than standard diffusion baselines, enabling high-frequency control in dynamic environments.

Introduction

In robotic imitation learning, a policy must map visual observations to precise robot actions. A primary hurdle is multimodality: when an expert demonstrates multiple valid paths to a goal, standard regression models (using L1/L2 loss) often average these paths, leading to failure.

Existing approaches typically fall into two categories:

  1. Autoregressive Models: These discretize actions into tokens, often sacrificing fine-grained precision.
  2. Diffusion Models: These handle multimodality well but require dozens of iterative steps, making them too slow for real-time reaction in dynamic settings.

Energy Policy bridges this gap by providing a natively multimodal, continuous, and highly efficient "third way" for robotic control.

Performance comparison of Energy Policy (Ours) against baselines, showing superior success rates at significantly lower inference times across various robotic tasks.

Methods

The Energy Policy architecture replaces iterative refinement with a direct sampling approach.

Energy Score Objective

Instead of standard regression, the model is trained using a "strictly proper scoring rule" called the energy score. During training, the model generates two independent action samples and compares them to the ground truth. This loss function incentivizes the model to produce samples that accurately represent the target distribution while maintaining the natural diversity (multimodality) of the expert data.

Architecture: Energy MLP with adaLN

The system employs a Transformer decoder to process observations, followed by a dedicated Energy MLP. To enable the model to sample from complex distributions in a single pass, the authors utilize Adaptive Layer Normalization (adaLN). This mechanism modulates internal features with a noise vector, allowing the model to explore different action modes effectively.

The Energy Policy architecture, featuring a Transformer Decoder and an Energy MLP that utilizes Adaptive Layer Normalization (adaLN) to condition action predictions on noise.

Experimental Results

The framework was evaluated across extensive simulated benchmarks (Robomimic, Franka Kitchen, MimicGen, PushT, Meta-World) and real-world tasks (Cup, Rabbit, Catch).

Efficiency and Speed

Energy Policy demonstrates a dominant speed-to-performance ratio. It is 22x to 70x faster than Diffusion Policy and 2.3x to 7x faster than CARP. On the PushT task, it achieved state-of-the-art results with only 7.7M parameters, a fraction of the size of typical diffusion models (>250M).

Success Rates and Real-World Performance

On the MimicGen benchmark, Energy Policy achieved a 10% higher success rate than diffusion-based multitask models. Its speed is particularly critical in dynamic tasks like "Catch," where the robot must intercept a moving object. Energy Policy succeeded 13/20 times, significantly outperforming Diffusion Policy (8/20), which struggled with the latency of iterative planning.

Discussion and Limitations

While highly efficient, the Energy Policy is sensitive to the energy loss coefficient ($\alpha$), with performance degrading if set too high. Additionally, the model can overfit if the Energy MLP is made too wide (e.g., >1024 channels) relative to the available demonstration data. Future work will explore scaling this architecture to larger, more diverse foundation datasets for robotics to determine if these efficiency gains persist at the billion-parameter scale.

Summary updated

Abstract

We present a fast and effective policy framework for robotic manipulation, named $\textbf{Energy Policy}$, designed for high-frequency robotic tasks and resource-constrained systems. Unlike existing robotic policies, Energy Policy natively predicts multimodal actions in a single forward pass, enabling high-precision manipulation at high speed. The framework is built upon two core components. First, we adopt the energy score as the learning objective to facilitate multimodal action modeling. Second, we introduce an energy MLP to implement the proposed objective while keeping the architecture simple and efficient. We conduct comprehensive experiments in both simulated environments and real-world robotic tasks to evaluate the effectiveness of Energy Policy. The results show that Energy Policy matches or surpasses the performance of state-of-the-art manipulation methods while significantly reducing computational overhead. Notably, on the MimicGen benchmark, Energy Policy achieves superior performance with at a faster inference compared to existing approaches.
BibTeX
@misc{jia2025fast,
  author = {Jingkai Jia and Tong Yang and Xueyao Chen and Chenhuan Liu and Wenqiang Zhang},
  title = {{Fast Visuomotor Policy for Robotic Manipulation}},
  year = {2025},
  eprint = {2510.12483},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.12483},
}
Figure