Multi-objective Cross-task Learning via Goal-conditioned GPT-based Decision Transformers for Surgical Robot Task Automation

Jiawei Fu, Yonghao Long, Kai Chen, Wang Wei, Qi DouICRA 2024

Introduction

Automating surgical tasks presents a unique set of challenges for robotic learning. Unlike standard manipulation tasks, surgery involves "long-horizon" procedures where success is often binary and only achieved at the very end of a complex sequence. This creates two primary hurdles for traditional Reinforcement Learning (RL) and imitation learning: the lack of historical context in Markovian decision-making and the problem of sparse rewards. Standard "Decision Transformers" (DT), which adapt Large Language Model architectures for robotics, typically rely on "returns-to-go" (accumulated future rewards). However, in surgical environments where partial rewards are non-existent, this signal becomes uninformative for the majority of the trajectory.

Methodology

To address these limitations, this research introduces the Goal-conditioned Decision Transformer (GCDT). The framework reframes surgical automation by replacing reward-based metrics with temporal indicators and utilizing multi-objective pretraining.

Goal-Conditioned Representation

Instead of predicting actions based on future rewards, the GCDT represents each timestep as a tuple: $(\hat{T}_t, g_t, o_t, a_t)$. Here, $\hat{T}_t$ represents the time-to-goal—an estimate of the remaining steps to completion. This provides a dense, informative signal throughout the entire procedure, even when rewards are sparse.

Figure 1: Overview of the GCDT framework, illustrating the training phase with cross-task datasets and the evaluation phase involving history sequence splicing for inference.

Multi-objective Pretraining

The model is made task-agnostic through pretraining across diverse surgical datasets using four distinct objectives:

  1. Action Prediction: Learning the mapping from states to movements.
  2. Forward Dynamics Prediction: Predicting subsequent states to model environmental physics.
  3. Time-to-goal Prediction: Estimating the temporal distance to the objective.
  4. Sequence Reconstruction: Utilizing a BERT-like masking objective to capture global dependencies within the trajectory.

To further enhance data efficiency, the researchers employed hindsight relabeling, a technique where failed attempts are treated as successful completions of an unintended goal, effectively densifying the training signal.

Experimental Setup

The framework was evaluated using the SurRoL simulator, an open-source platform for surgical robot learning. The benchmark included 10 tasks of varying complexity:

  • Single-arm tasks: NeedleReach, GauzeRetrieve, NeedlePick, PegTransfer, NeedleRegrasp, PickAndPlace, and MatchBoard.
  • Bimanual tasks: BiPegTransfer, BiPegBoard, and MatchBoardPanel.

Validation was also performed by deploying learned trajectories on a physical da Vinci Research Kit (dVRK) to test real-world transferability.

Key Results

The GCDT significantly outperformed state-of-the-art offline and online methods across all benchmarks.

Figure 3: Success rates across 10 surgical tasks, demonstrating the performance of the model as a function of the percentage of training data utilized.
  • Performance on Complex Tasks: In the highly challenging BiPegBoard task, the GCDT achieved a 1.00 success rate, whereas the standard Decision Transformer reached only 0.67, and other baselines failed entirely.
  • Average Improvement: Across all 10 tasks, the proposed method showed an average success rate improvement of 0.31 over the baseline Decision Transformer.
  • Data Efficiency: The model demonstrated remarkable efficiency, achieving high performance with only 5% of the data typically required by standard online RL algorithms.
  • Sim-to-Real Transfer: Successful deployment on the dVRK robot confirmed that the trajectories learned in simulation are robust enough for physical robotic hardware.

Discussion and Future Work

The results suggest that "time-to-goal" is a more stable and effective guiding signal than reward-based metrics in sparse-reward environments. Furthermore, cross-task pretraining allows the model to learn a "language" of surgical manipulation that generalizes better than task-specific training.

Future research will focus on transitioning from state-based coordinates to raw visual input (pixels) for a truly end-to-end system and adapting the model to dynamic environments where surgical tissues may deform unpredictably.

Summary updated

BibTeX
@inproceedings{fu2024multi,
  author = {Jiawei Fu and Yonghao Long and Kai Chen and Wang Wei and Qi Dou},
  title = {{Multi-objective Cross-task Learning via Goal-conditioned GPT-based Decision Transformers for Surgical Robot Task Automation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
}
Figure