TL;DR
This paper introduces DRIFT, a framework that dynamically adjusts the number of trainable parameters in diffusion policies for robotics. By leveraging Singular Value Decomposition (SVD) for rank modulation and a rank scheduler, DRIFT significantly speeds up training and improves sample efficiency in interactive imitation learning, without sacrificing performance.
What Problem Does This Paper Address?
Diffusion policies have shown impressive results in robotic motion generation, largely due to their overparameterized nature which allows for powerful representations. However, this overparameterization comes at a significant cost: high computational demands for training. This makes online interactive imitation learning (IL) with diffusion policies impractical, as it requires extensive training time. Existing methods for improving efficiency, like parameter-efficient fine-tuning, are often designed for pre-trained models and are not suitable for training diffusion policies from scratch, which is common in robotics. The challenge is to achieve the representational power of large diffusion models while maintaining computational efficiency, especially in interactive learning scenarios where rapid adaptation is crucial.
Methods
The core contribution is the DRIFT (Dynamic Rank-adjustable DIFfusion Policy Training) framework. DRIFT introduces two key components:
- Rank Modulation: This technique uses Singular Value Decomposition (SVD) to decompose weight matrices of convolutional layers into trainable and frozen components. By adjusting the "rank" (the number of trainable components), the framework can dynamically control the number of parameters being updated. This is done by splitting the U, Σ, and V matrices from the SVD. Crucially, this approach avoids introducing new parameters and maintains the same computational cost for forward passes as standard convolutions.
- Rank Scheduler: This component dynamically adjusts the number of trainable ranks throughout the training process. It uses a decay function (linear, cosine, sigmoid, exponential) that determines the current number of trainable ranks based on the training epoch, maximum rank, and a desired terminal rank. This allows the policy to start with more trainable parameters for initial learning and gradually reduce them for efficiency as training progresses.
DRIFT is instantiated in DRIFT-DAgger, an expert-gated interactive IL algorithm. DRIFT-DAgger combines an offline bootstrapping phase (behavior cloning) with an online adaptation phase. During online adaptation, DRIFT-DAgger uses the low-rank component to speed up training. If the expert policy detects an error, it intervenes to provide corrective demonstrations, which are then used to update the policy.
Datasets
The paper uses existing datasets for training and evaluation:
- Simulation:
- MVS (Manipulation with Viewpoint Selection) tasks, specifically a pick-and-place scenario.
- Robosuite environments (Lift and Can).
- Real-world:
- Tasks involving a 17-DOF robotic system (similar to MVS simulation) including block stacking, drawer assembling, and drawer interaction.
The paper does not introduce new datasets but leverages existing benchmarks to demonstrate the effectiveness of their proposed method.
Key Insights and Ideas
- Dynamic Rank Adjustment is Key for Efficient Interactive IL: The paper convincingly argues that dynamically adjusting trainable ranks is crucial for making interactive imitation learning with large diffusion models practical. Static low-rank methods or fine-tuning approaches are insufficient for training from scratch.
- SVD for Rank Modulation is Stable and Efficient: Using SVD for rank modulation offers a stable way to adjust trainable parameters without adding new ones or increasing computational overhead during forward passes, unlike some adapter-based methods.
- Sigmoid Decay Strikes a Good Balance: Among the tested decay functions for the rank scheduler, the sigmoid function (especially with a steepness parameter of 0.5) provided the best balance between training time and policy performance.
- Terminal Rank Matters, But Not Too Much: While a lower terminal rank (rmin) reduces training time, it can degrade performance. A value around 256 for the terminal rank appears to be a good heuristic for manipulation tasks, offering a solid trade-off.
- Overparameterization is Still Valuable: The paper emphasizes that even with rank reduction, maintaining a sufficient number of trainable ranks is important for initial learning and minimizing approximation errors, especially when training from scratch.
Key Results
- Reduced Training Time: DRIFT-DAgger consistently reduced mean batch training time across all stages (offline, online, and combined) compared to full-rank baselines like HG-DAgger. For instance, in simulation, the rank modulation variant achieved an 11% reduction in mean batch training time.
- Improved Sample Efficiency: DRIFT-DAgger variants demonstrated improved sample efficiency, requiring fewer expert labels to achieve comparable or better success rates than baselines.
- Comparable Performance: Despite the efficiency gains, DRIFT-DAgger achieved success rates comparable to or exceeding full-rank HG-DAgger on various simulation and real-world tasks.
- Stability: The rank modulation approach proved more stable than LoRA-based methods when combined with a rank scheduler, avoiding performance degradation and instability issues.
- Real-World Efficacy: The framework was successfully deployed on real-world robotic tasks, demonstrating its practical applicability and effectiveness in reducing training time and improving sample efficiency.
Limitations and Future Work
- Online RL Not Explored: The current work focuses on interactive IL. The authors suggest exploring the application of DRIFT within online Reinforcement Learning (RL) paradigms as a future direction.
- Monotonic Rank Adjustment: The current rank adjustment strategy is monotonic. Future work could explore more adaptive and intelligent strategies for adjusting trainable ranks, potentially varying them across different network blocks based on their intrinsic rank properties.
- Task-Dependent Design Choices: The optimal decay functions and terminal ranks might be task-dependent, requiring further investigation.
- Generalization to Other Architectures: While demonstrated on U-Net architectures, exploring DRIFT's applicability to other network architectures used in diffusion models would be valuable.
Summary updated