TL;DR
Spectrally Routed Diffusion Policy (SR-DP) solves the performance gap in multi-task robotic learning by preventing "gradient interference" between tasks. By decomposing a robot's neural network weights into orthogonal mathematical components and tuning them with a task-specific "router," SR-DP allows a single model to outperform specialized per-task models while training only 4% of the parameters in its second stage.
What Problem Does This Paper Address?
Diffusion policies—which treat robot actions like a denoising process—are incredibly effective at learning single tasks. However, when you try to train one model to handle multiple tasks (like stacking blocks and opening bottles), performance usually craters.
The authors identify two main culprits:
- Perceptual Blur: Standard visual encoders often use "global pooling," which squashes an entire image into one vector. This loses the fine spatial details needed to distinguish between different tasks happening in the same workspace.
- Gradient Interference: When a single set of dense weights is updated for Task A and Task B simultaneously, the gradients often conflict. It’s like two people trying to steer a car in different directions; the model ends up at a mediocre compromise that doesn't solve either task well.
Methods
SR-DP introduces a two-stage approach to multi-task learning:
1. The Visual "Pyramid"
Instead of global pooling, the authors use a frozen VGGT encoder paired with a spatial-compression adapter. As shown in Figure 2, this adapter creates a multi-scale "pyramid" of visual tokens (at 1x1, 2x2, and 3x3 resolutions). This ensures the model retains both the "big picture" of the scene and the "fine details" of object locations.
2. Spectral Routing
This is the core innovation. After training a base policy on all tasks (Stage 1), the authors freeze the weights and decompose the linear layers using Singular Value Decomposition (SVD).
- Rank-One Experts: SVD breaks a weight matrix into a set of orthogonal components. SR-DP treats these as "experts."
- The Router: A small, trainable "router" (Fig. 3) adjusts the "volume" (singular values) of these experts based on the specific task, the current step in the diffusion process, and the robot's state.
- Householder Rotations: The model uses Householder reflections to slightly rotate the input directions, allowing for fine-tuned adaptation while strictly maintaining the mathematical orthogonality that prevents tasks from interfering with each other.
Datasets
The researchers evaluated SR-DP on two main benchmarks:
- RoboTwin (Simulation): Six diverse bimanual tasks including "Block Hammer Beat," "Bottle Adjust," and "Tool Adjust." Each task provided 100 teleoperated demonstrations.
- Real-World TeleAvatar: Four complex bimanual tasks (e.g., sorting bottles by label color, flipping inverted bottles) performed on a physical dual-arm robot platform.
Key Insights and Ideas
- Orthogonality as a Shield: By forcing task-specific updates to happen along orthogonal (mathematically independent) directions, the model ensures that learning a new nuance for Task A doesn't "overwrite" what it knows about Task B.
- Dynamic Adaptation: The routing isn't static. Because it's conditioned on the diffusion step, the model can use different "experts" for the initial coarse planning of a movement versus the final, high-precision contact phase.
- Efficiency: In the second stage of training, the model only needs to optimize roughly 4.1% of its parameters. This makes it a highly parameter-efficient way to add new capabilities to a base robot policy.
Key Results
- Surpassing Specialists: On the RoboTwin benchmark, a single SR-DP model achieved a 68.4% average success rate. This not only beat other multi-task methods by over 20 points but actually outperformed the per-task specialists (which averaged 64.6%).
- Real-World Gains: In physical robot tests, the Stage 2 spectral routing improved success rates from 24% to 43%, a nearly 80% relative improvement over the base joint-trained model.
- Reduced Conflict: The authors visualized the "sensitivity conflict" between tasks (Fig. 6). They found that SR-DP significantly lowered the mathematical opposition between task gradients compared to standard training.
Limitations and Future Work
- Two-Stage Requirement: The SVD basis only becomes useful after the model has already converged on a general policy. This means the training isn't strictly "end-to-end" in one go.
- Task Identification: The current system relies on discrete task IDs or a task-prediction head. The authors suggest that language-conditioned routing (e.g., telling the robot "pick up the red bottle") is the next logical step to make the system more flexible.
- Computational Overhead: While parameter-efficient, the SVD decomposition and routing add some complexity to the architecture that needs to be managed for real-time deployment.
Summary updated