Here is an engaging, technically grounded summary of the paper.
Fast Visuomotor Policy for Robotic Manipulation
TL;DR
Robotic policies trained with diffusion models handle multimodal continuous actions well but are computationally expensive due to iterative denoising steps. Energy Policy introduces an energy-score-based learning objective and a lightweight MLP head that enables single-forward-pass, multimodal action sequence prediction. This yields state-of-the-art success rates across diverse manipulation benchmarks while accelerating inference by 2.3× to 70× over diffusion and autoregressive baselines.
What Problem Does This Paper Address?
Imitation learning for visuomotor manipulation requires predicting continuous actions that reflect the multimodal nature of human demonstrations (e.g., reaching around an obstacle from either the left or the right).
Current approaches face a stark tradeoff between speed and distribution expressiveness:
- Direct regression (L1/L2 loss) predicts continuous actions quickly in a single forward pass, but inherently assumes a unimodal distribution, causing models to average conflicting actions and fail catastrophically.
- Autoregressive modeling (AM) predicts discrete tokens, which can sacrifice continuous, fine-grained action precision.
- Diffusion Policies (DP) model complex multimodal distributions effectively, but require multiple iterative denoising passes (often 10–100 steps), introducing high inference latency that limits deployment on high-frequency, resource-constrained robotic systems.
Methods
Energy Policy sidesteps iterative sampling altogether by training an action generator directly with a strictly proper scoring rule.
Observations + Action Queries
│
▼
Transformer Decoder
│ (latent representations z_t)
▼
Energy MLP (+ Noise via adaLN) ──► Continuous Action Chunk (Single Forward Pass)
-
Energy Loss Objective:
Based on the statistical energy score (a strictly proper scoring rule used to evaluate multivariate distributions), the policy is trained to minimize distributional discrepancy. Given ground-truth action $a_t$ and two independent predicted actions $\hat{a}^1_t, \hat{a}^2_t$ drawn from the model distribution: $$\mathcal{L}(p, a_t) = \|\hat{a}^1_t - a_t\|^\alpha + \|\hat{a}^2_t - a_t\|^\alpha - \|\hat{a}^1_t - \hat{a}^2_t\|^\alpha$$ The first two terms attract predictions to the expert demonstration, while the third term penalizes predictions that collapse to the same mode, actively enforcing sample diversity without mode-averaging. The authors empirically set $\alpha = 1.0$. -
Energy MLP with adaLN Modulation:
A decoder-only Transformer processes observation history and learnable action queries to produce feature tokens $z_t$. An Energy MLP takes $z_t$ alongside random noise sampled from a uniform distribution. Noise conditioning is implemented via adaLN-Zero (adaptive layer normalization) blocks with scale, shift, and gate operations. -
Single-Pass Inference:
During deployment, the model generates an entire action chunk (e.g., 16 continuous action steps) using a single noise vector in a single forward pass, completely removing the need for iterative denoising or consistency distillation.
Datasets
The authors evaluated Energy Policy on several standard simulation benchmarks and real-world setups:
- Robomimic: Single-object manipulation (
Lift,Can,Square) across both state-based and image-based modalities (200 demonstrations each). - Franka Kitchen: Long-horizon, multi-task planning with 566 demonstrations across four sequential subtasks.
- MimicGen: Large-scale multitask benchmark spanning 8 robosuite tasks with 1,000 demonstrations each.
- PushT & Robomimic Hard Tasks: High-precision, contact-rich tasks (
PushT,Square-mh,ToolHang,Transport). - Meta-World: 5 challenging 3D point-cloud manipulation tasks evaluated against 3D Diffusion Policy (DP3).
- Real-World Platforms: Two hardware rigs:
- Emergen CR3 arm with UMI gripper for static pick-and-place tasks (
Cup,Rabbit). - UR5 arm for a dynamic, reactive tracking task (
Catch).
Key Insights and Ideas
- Energy Scoring Bypasses Denoising: Rather than iteratively modeling the score function (gradient of the log-density) via diffusion, optimizing the energy score directly aligns the generative distribution to the data in one step.
- adaLN is Vital for Non-Diffusion Noise Injection: Simply concatenating noise vectors to Transformer representations leads to a drastic drop in success rate (dropping from 85% to 31% on
Square-mh). Adaptive normalization provides the necessary conditioning capacity to steer outputs between distinct multimodal modes. - Real-Time Responsiveness Matters Most in Dynamic Tasks: While Energy Policy modestly outperformed Diffusion Policy on static real-world pick-and-place (20/20 vs. 19/20), the advantage was massive on dynamic tasks like
Catch(13/20 vs. 8/20 success rate) due to eliminating the 100 ms diffusion latency down to 20 ms.
Key Results
- PushT & High-Precision Tasks: Achieves 0.87 average success rate, outperforming specialized one-step models: OneDP (0.84), IMLE (0.81), Consistency Policy (0.82), and VQ-BeT (0.68).
- Simulation Speedups:
- Robomimic Image: Matches/exceeds DP-T success rates while dropping latency from 0.67s to 0.03s (~22× speedup).
- Franka Kitchen: 100% success on subtasks p1–p3 and 96% on p4, operating at 0.02s per chunk (vs. 0.84s–0.91s for DP).
- MimicGen Multitask: Achieves 0.86 average success at 0.08s, outperforming Multitask CARP (0.85 at 0.18s) and Sparse Diffusion Policy (0.78 at 1.53s).
- Parameter Efficiency: Delivers superior performance on PushT with only 7.73M parameters, compared to 251.5M for OneDP and 255.2M for Consistency Policy.
Limitations and Future Work
- Theoretical Bounds on Sample Count: The energy score is approximated using only two independent samples ($M=2$) during training; the paper does not systematically study whether scaling up the number of samples during training aids distributions with higher modality counts.
- Hyperparameter Sensitivity ($\alpha$): Ablations show that while performance is stable for $\alpha \in [0.5, 1.0]$, setting $\alpha \ge 1.5$ degrades performance noticeably (down to 71%), indicating that tuning $\alpha$ requires care.
- Extremely High-Dimensional Action Spaces: While evaluated up to dual-arm manipulation, the framework was not benchmarked on high-DOF dexterous hands or whole-body humanoid locomotion.
Summary updated