Introduction
Automating surgical tasks presents a unique set of challenges for robotic learning. Unlike standard manipulation tasks, surgery involves "long-horizon" procedures where success is often binary and only achieved at the very end of a complex sequence. This creates two primary hurdles for traditional Reinforcement Learning (RL) and imitation learning: the lack of historical context in Markovian decision-making and the problem of sparse rewards. Standard "Decision Transformers" (DT), which adapt Large Language Model architectures for robotics, typically rely on "returns-to-go" (accumulated future rewards). However, in surgical environments where partial rewards are non-existent, this signal becomes uninformative for the majority of the trajectory.
Methodology
To address these limitations, this research introduces the Goal-conditioned Decision Transformer (GCDT). The framework reframes surgical automation by replacing reward-based metrics with temporal indicators and utilizing multi-objective pretraining.
Goal-Conditioned Representation
Instead of predicting actions based on future rewards, the GCDT represents each timestep as a tuple: $(\hat{T}_t, g_t, o_t, a_t)$. Here, $\hat{T}_t$ represents the time-to-goal—an estimate of the remaining steps to completion. This provides a dense, informative signal throughout the entire procedure, even when rewards are sparse.
Multi-objective Pretraining
The model is made task-agnostic through pretraining across diverse surgical datasets using four distinct objectives:
- Action Prediction: Learning the mapping from states to movements.
- Forward Dynamics Prediction: Predicting subsequent states to model environmental physics.
- Time-to-goal Prediction: Estimating the temporal distance to the objective.
- Sequence Reconstruction: Utilizing a BERT-like masking objective to capture global dependencies within the trajectory.
To further enhance data efficiency, the researchers employed hindsight relabeling, a technique where failed attempts are treated as successful completions of an unintended goal, effectively densifying the training signal.
Experimental Setup
The framework was evaluated using the SurRoL simulator, an open-source platform for surgical robot learning. The benchmark included 10 tasks of varying complexity:
- Single-arm tasks: NeedleReach, GauzeRetrieve, NeedlePick, PegTransfer, NeedleRegrasp, PickAndPlace, and MatchBoard.
- Bimanual tasks: BiPegTransfer, BiPegBoard, and MatchBoardPanel.
Validation was also performed by deploying learned trajectories on a physical da Vinci Research Kit (dVRK) to test real-world transferability.
Key Results
The GCDT significantly outperformed state-of-the-art offline and online methods across all benchmarks.
- Performance on Complex Tasks: In the highly challenging BiPegBoard task, the GCDT achieved a 1.00 success rate, whereas the standard Decision Transformer reached only 0.67, and other baselines failed entirely.
- Average Improvement: Across all 10 tasks, the proposed method showed an average success rate improvement of 0.31 over the baseline Decision Transformer.
- Data Efficiency: The model demonstrated remarkable efficiency, achieving high performance with only 5% of the data typically required by standard online RL algorithms.
- Sim-to-Real Transfer: Successful deployment on the dVRK robot confirmed that the trajectories learned in simulation are robust enough for physical robotic hardware.
Discussion and Future Work
The results suggest that "time-to-goal" is a more stable and effective guiding signal than reward-based metrics in sparse-reward environments. Furthermore, cross-task pretraining allows the model to learn a "language" of surgical manipulation that generalizes better than task-specific training.
Future research will focus on transitioning from state-based coordinates to raw visual input (pixels) for a truly end-to-end system and adapting the model to dynamic environments where surgical tissues may deform unpredictably.
Summary updated