Spectrally Routed Diffusion Policy for Multi-Task Robotic Manipulation

Mingcong Li, Yihan Zhou, Jiayu Qu, Xueyang Guo, Yilin MoarXiv 2026 · arXiv

TL;DR

Spectrally Routed Diffusion Policy (SR-DP) solves the performance gap in multi-task robotic learning by preventing "gradient interference" between tasks. By decomposing a robot's neural network weights into orthogonal mathematical components and tuning them with a task-specific "router," SR-DP allows a single model to outperform specialized per-task models while training only 4% of the parameters in its second stage.

What Problem Does This Paper Address?

Diffusion policies—which treat robot actions like a denoising process—are incredibly effective at learning single tasks. However, when you try to train one model to handle multiple tasks (like stacking blocks and opening bottles), performance usually craters.

The authors identify two main culprits:

  1. Perceptual Blur: Standard visual encoders often use "global pooling," which squashes an entire image into one vector. This loses the fine spatial details needed to distinguish between different tasks happening in the same workspace.
  2. Gradient Interference: When a single set of dense weights is updated for Task A and Task B simultaneously, the gradients often conflict. It’s like two people trying to steer a car in different directions; the model ends up at a mediocre compromise that doesn't solve either task well.

Methods

SR-DP introduces a two-stage approach to multi-task learning:

1. The Visual "Pyramid"

Instead of global pooling, the authors use a frozen VGGT encoder paired with a spatial-compression adapter. As shown in Figure 2, this adapter creates a multi-scale "pyramid" of visual tokens (at 1x1, 2x2, and 3x3 resolutions). This ensures the model retains both the "big picture" of the scene and the "fine details" of object locations.

Fig. 2. Spatial-compression adapter. Three residual blocks form a coarse- to-fine pyramid; each level is pooled at 1×1, 2×2 or 3×3, flattened, linearly projected, and concatenated into 14 tokens per view (Eq. 3).

2. Spectral Routing

This is the core innovation. After training a base policy on all tasks (Stage 1), the authors freeze the weights and decompose the linear layers using Singular Value Decomposition (SVD).

  • Rank-One Experts: SVD breaks a weight matrix into a set of orthogonal components. SR-DP treats these as "experts."
  • The Router: A small, trainable "router" (Fig. 3) adjusts the "volume" (singular values) of these experts based on the specific task, the current step in the diffusion process, and the robot's state.
  • Householder Rotations: The model uses Householder reflections to slightly rotate the input directions, allowing for fine-tuned adaptation while strictly maintaining the mathematical orthogonality that prevents tasks from interfering with each other.
Fig. 3. Router that produces token-wise ˆσb,q in Eq. 6. A task-level branch (P ekb) is shared across denoiser tokens, while a token-state bottleneck takes xb,q and adds the diffusion-step bias Wnemb(nb).

Datasets

The researchers evaluated SR-DP on two main benchmarks:

  • RoboTwin (Simulation): Six diverse bimanual tasks including "Block Hammer Beat," "Bottle Adjust," and "Tool Adjust." Each task provided 100 teleoperated demonstrations.
  • Real-World TeleAvatar: Four complex bimanual tasks (e.g., sorting bottles by label color, flipping inverted bottles) performed on a physical dual-arm robot platform.

Key Insights and Ideas

  • Orthogonality as a Shield: By forcing task-specific updates to happen along orthogonal (mathematically independent) directions, the model ensures that learning a new nuance for Task A doesn't "overwrite" what it knows about Task B.
  • Dynamic Adaptation: The routing isn't static. Because it's conditioned on the diffusion step, the model can use different "experts" for the initial coarse planning of a movement versus the final, high-precision contact phase.
  • Efficiency: In the second stage of training, the model only needs to optimize roughly 4.1% of its parameters. This makes it a highly parameter-efficient way to add new capabilities to a base robot policy.

Key Results

  • Surpassing Specialists: On the RoboTwin benchmark, a single SR-DP model achieved a 68.4% average success rate. This not only beat other multi-task methods by over 20 points but actually outperformed the per-task specialists (which averaged 64.6%).
  • Real-World Gains: In physical robot tests, the Stage 2 spectral routing improved success rates from 24% to 43%, a nearly 80% relative improvement over the base joint-trained model.
  • Reduced Conflict: The authors visualized the "sensitivity conflict" between tasks (Fig. 6). They found that SR-DP significantly lowered the mathematical opposition between task gradients compared to standard training.
Heatmaps showing how Stage 2 spectral routing reduces the conflict between different tasks' learning requirements.

Limitations and Future Work

  • Two-Stage Requirement: The SVD basis only becomes useful after the model has already converged on a general policy. This means the training isn't strictly "end-to-end" in one go.
  • Task Identification: The current system relies on discrete task IDs or a task-prediction head. The authors suggest that language-conditioned routing (e.g., telling the robot "pick up the red bottle") is the next logical step to make the system more flexible.
  • Computational Overhead: While parameter-efficient, the SVD decomposition and routing add some complexity to the architecture that needs to be managed for real-time deployment.

Summary updated

Abstract

Diffusion-based policies have achieved strong performance on single-task robotic manipulation. However, a single model trained jointly on multiple tasks consistently underperforms the per-task specialists trained on the same demonstrations. We focus on two challenges. First, the convolutional encoders commonly used in diffusion-based policies produce visual conditions that are not discriminative enough. Second, gradients from incompatible action modes update the shared denoiser's dense parameters along the same directions, causing destructive interference during joint training. To address both, we propose SR-DP (Spectrally Routed Diffusion Policy). A frozen VGGT encoder with a spatial-compression adapter produces compact multi-scale semantic-geometric tokens, a task-discriminative visual condition for tasks that share a workspace. We first train the denoiser as a standard diffusion policy, then decompose selected linear layers by SVD into a frozen basis of orthogonal rank-one experts, adapted through spectral routing and Householder input rotations. On six RoboTwin tasks, SR-DP averages a 68.4% success rate, surpassing multi-task VO-DP by 21.8 points and the strongest per-task baseline by 3.8 points. On four real-world bimanual tasks, Stage 2 reaches a 43.0% average success rate and improves over Stage 1 by 19.0 points.
BibTeX
@misc{li2026spectrally,
  author = {Mingcong Li and Yihan Zhou and Jiayu Qu and Xueyang Guo and Yilin Mo},
  title = {{Spectrally Routed Diffusion Policy for Multi-Task Robotic Manipulation}},
  year = {2026},
  eprint = {2611.04477},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.04477},
}
Figure