TL;DR
The Spherical Diffusion Policy (SDP) introduces an end-to-end SE(3)-equivariant diffusion policy that allows robots to generalize to novel 3D object arrangements using only a handful of human demonstrations. By embedding states, actions, and the denoising process into spherical Fourier space, SDP achieves exceptional sample efficiency and robust spatial generalization while maintaining the fast inference speeds required for real-time control.
What Problem Does This Paper Address?
Imitation learning via Diffusion Policies has become a powerful paradigm for teaching robots complex, multi-modal manipulation tasks directly from human demonstrations. However, these policies typically generalize poorly to novel 3D spatial arrangements. If an object is tilted, rotated, or moved to an unseen position, a standard diffusion policy will likely fail unless it has been trained on an exhaustive (and costly) dataset covering all possible 3D poses.
To fix this, researchers look to geometric equivariance—ensuring that if a scene undergoes a 3D transformation (rotation or translation), the robot's predicted trajectory transforms accordingly.
Prior attempts at equivariant diffusion policies suffer from severe limitations: they are often restricted to flat 2D rotations (SO(2)), require tedious task-specific object segmentation pipelines, or rely on computationally heavy mathematical representations that are too slow for real-time robotic deployment. SDP tackles these challenges head-on to deliver a lightweight, end-to-end SE(3)-equivariant policy capable of handling multi-object scenes.
Methods
SDP achieves full SE(3) equivariance (3D translation + 3D rotation) by combining a clever coordinate formulation with frequency-domain representations:
- Translational Invariance via Relative Actions: Instead of predicting actions in the global workspace frame, SDP canonicalizes the inputs and outputs by formulating them relative to the robot's gripper frame. This effectively handles the translation ($T(3)$) component of SE(3).
- Rotational Equivariance via Spherical Fourier Features: To handle 3D rotations ($SO(3)$), SDP represents spatial data using truncated spherical Fourier coefficients (up to degree $L=2$). This provides a continuous, highly compact approximation of the 3D scene geometry without the massive computational overhead of traditional SO(3) irreducible representations.
- Spherical Denoising Temporal U-net (SDTU): The core denoising network is a 1D U-net architecture modified to process spherical Fourier features. It uses channel-mixing temporal convolutions to maintain spatiotemporal equivariance throughout the diffusion process.
- Spherical FiLM (SFiLM): To condition the denoising process on the scene features equivariantly, the authors developed a spherical extension of Feature-wise Linear Modulation (FiLM), modulating each spherical harmonic degree independently.
Datasets
The authors evaluated SDP across both simulated and physical environments:
- Simulation: They utilized MimicGen, benchmarking on 12 standard tasks with SE(2) variations and 4 tasks explicitly modified for SE(3) initialization by randomly tilting the workspace table up to $\pm 30^\circ$.
- Physical Experiments: They designed 5 real-world tasks using a dual UR5e manipulator setup: Turn Lever, Push Eraser, Grasp Box (bimanual), Flip Book (bimanual), and Pack Package (bimanual, long-horizon). Training data consisted of just 30 to 66 human demonstrations per task collected via GELLO teleoperation arms.
Key Insights and Ideas
- The Power of the Frequency Domain: Moving from spatial coordinates to spherical Fourier space allows the network to handle continuous 3D rotations via simple Wigner D-matrix multiplications. This eliminates the discretization errors found in voxel- or grid-based rotation methods.
- Bypassing Segmentation: Unlike recent SO(3) policies that require explicit object segmentation to isolate the target item, SDP operates directly on the entire unsegmented point cloud. Its mathematical formulation naturally handles multi-object interactions.
- Efficiency Matters: By opting for compact spherical Fourier features over heavy SE(3)-Transformers, the architecture achieves a massive speedup, making closed-loop diffusion control practical on standard hardware.
Key Results
- Robustness to 3D Tilts: In simulation tasks with a $30^\circ$ table tilt, SDP maintained stable performance while standard Diffusion Policy (DiffPo) and SO(2) baselines (EquiDiff) saw their success rates collapse to near zero.
- 10x Data Efficiency: SDP achieved a 48.5% success rate with only 100 demonstrations, outperforming standard Diffusion Policies trained on 1,000 demonstrations—a tenfold increase in sample efficiency.
- Real-World Success: In physical robot deployments, SDP achieved an average success rate of 78% across all tasks, vastly outperforming EquiDiff (17%) and DiffPo (7%).
- Inference Speed: SDP clocks an inference time of 0.44 seconds, which is roughly 66× faster than heavy trajectory-level SE(3) diffusion baselines like ET-SEED (29.4s).
Limitations and Future Work
Despite its strong performance, the authors note two primary limitations:
- Pure Position Control: SDP currently operates entirely via position control and ignores contact forces. This caused issues in contact-rich tasks like Flip Book, where the robot occasionally triggered safety stops due to excessive force. Integrating force-awareness equivariantly is a key future direction.
- Point Cloud Resolution: The observation encoder uses relatively low-resolution point clouds (1024 points), which can struggle to capture fine geometric details, such as the exact boundaries of a small eraser. Upgrading to more efficient Graph Neural Networks could resolve this bottleneck.
Summary updated