Abstract
Robotic-assisted surgery (RAS) currently relies on a human-in-the-loop paradigm, where surgeons directly teleoperate robotic systems. Advancing toward supervised or full autonomy requires high-quality, multi-modal training data that accurately captures the complexities of surgical tasks. We present SurgSync, a framework designed to bridge the data gap by providing time-synchronized video, kinematics, and contact data. SurgSync features dual-mode recording, modern imaging hardware, and novel contact sensing. Our validation includes a dataset of 214 instances across canonical surgical tasks using ex-vivo tissue, demonstrating the framework's utility for surgical skill assessment and AI training.
Introduction
The transition from teleoperation to autonomous surgical assistance is hindered by a lack of robust, synchronized datasets. Existing surgical datasets often suffer from three major bottlenecks: temporal misalignment between video and kinematics, low visual fidelity from legacy hardware, and a lack of objective ground truth for tool-tissue interactions. SurgSync addresses these challenges by providing a comprehensive pipeline for high-fidelity data collection and post-processing.
The SurgSync Framework
The core of SurgSync is a ROS-based architecture that ensures precise temporal alignment across multiple data streams, including stereo video, side-view perspectives, and robot kinematics.
Dual-Mode Synchronization
To support both real-time inference and high-throughput training, SurgSync implements two recording modes:
- Online-Matching: Enforces a strict 10ms tolerance between modalities, which is essential for real-time AI feedback loops.
- Offline-Matching: Prioritizes raw data throughput during collection and uses post-collection interpolation to achieve a mean latency of just 1.35ms, ensuring near-perfect alignment for training sequence-based models.
Post-Processing Toolbox
The framework includes a suite of tools for stereo rectification, depth estimation (using the FoundationStereo model), and optical flow (via RAFT). A key innovation is Kinematic Reprojection, which projects 3D tool-tip positions onto 2D video frames as Gaussian heatmaps, providing a spatial "attention" mechanism that helps AI models focus on the area of interaction.
Hardware and Sensing Integration
SurgSync upgrades the standard da Vinci Research Kit (dVRK) with modern clinical-grade components and custom sensors to improve data quality and provide new ground truth modalities.
High-Fidelity Imaging
The legacy dVRK endoscope was replaced with a modern "chip-on-tip" stereo endoscope. This upgrade resulted in a 30-fold increase in image sharpness (measured via Laplacian variance), which is crucial for downstream computer vision tasks like tool tracking and depth estimation.
Capacitive Contact Sensing
To provide objective ground truth for tool-tissue contact—a notoriously difficult metric to capture—we integrated a custom Arduino-based capacitive sensor into the surgical tools. This allows the system to record binary contact signals directly into the data stream, achieving 99.1% accuracy during tissue manipulation tasks.
Dataset and Validation
The SurgSync dataset comprises 214 validated instances of canonical tasks: peg transfer, suturing, knot tying, and tissue manipulation. Data was collected from 13 participants with varying skill levels using ex-vivo tissues (chicken, beef, and pork) to ensure realistic tissue dynamics.
To validate the dataset's utility, we trained a multi-path regression model for surgical skill assessment. The model achieved a Spearman’s rank correlation coefficient (SROCC) of 0.803 for suturing and 0.765 for knot tying, proving that the synchronized multi-modal data effectively captures the nuances of surgical expertise.
Conclusion
SurgSync provides an open-source foundation for the next generation of surgical AI. By solving the challenges of synchronization, visual quality, and contact ground truth, the framework enables more accurate modeling of surgical tasks. All software and the dataset are available at surgsync.github.io.
Summary updated