SurgSync: Time-Synchronized Multi-Modal Data Collection Framework and Dataset for Surgical Robotics

Haoying Zhou, Chang Liu, Yimeng Wu, Junlin Wu, Zijian Wu, Yu Chung Lee, Sara Martuscelli, Spetimiu E. Salcudean, Gregory S. Fischer, Peter Kazanzides

Abstract

Robotic-assisted surgery (RAS) currently relies on a human-in-the-loop paradigm, where surgeons directly teleoperate robotic systems. Advancing toward supervised or full autonomy requires high-quality, multi-modal training data that accurately captures the complexities of surgical tasks. We present SurgSync, a framework designed to bridge the data gap by providing time-synchronized video, kinematics, and contact data. SurgSync features dual-mode recording, modern imaging hardware, and novel contact sensing. Our validation includes a dataset of 214 instances across canonical surgical tasks using ex-vivo tissue, demonstrating the framework's utility for surgical skill assessment and AI training.

Introduction

The transition from teleoperation to autonomous surgical assistance is hindered by a lack of robust, synchronized datasets. Existing surgical datasets often suffer from three major bottlenecks: temporal misalignment between video and kinematics, low visual fidelity from legacy hardware, and a lack of objective ground truth for tool-tissue interactions. SurgSync addresses these challenges by providing a comprehensive pipeline for high-fidelity data collection and post-processing.

The SurgSync Framework

The core of SurgSync is a ROS-based architecture that ensures precise temporal alignment across multiple data streams, including stereo video, side-view perspectives, and robot kinematics.

Overview of the SurgSync framework, illustrating the data collection flowchart and post-processing capabilities such as optical flow, kinematic reprojection, and depth estimation.

Dual-Mode Synchronization

To support both real-time inference and high-throughput training, SurgSync implements two recording modes:

  1. Online-Matching: Enforces a strict 10ms tolerance between modalities, which is essential for real-time AI feedback loops.
  2. Offline-Matching: Prioritizes raw data throughput during collection and uses post-collection interpolation to achieve a mean latency of just 1.35ms, ensuring near-perfect alignment for training sequence-based models.

Post-Processing Toolbox

The framework includes a suite of tools for stereo rectification, depth estimation (using the FoundationStereo model), and optical flow (via RAFT). A key innovation is Kinematic Reprojection, which projects 3D tool-tip positions onto 2D video frames as Gaussian heatmaps, providing a spatial "attention" mechanism that helps AI models focus on the area of interaction.

Hardware and Sensing Integration

SurgSync upgrades the standard da Vinci Research Kit (dVRK) with modern clinical-grade components and custom sensors to improve data quality and provide new ground truth modalities.

The SurgSync hardware setup, featuring the dVRK components (ECM, PSM), custom adapters, and the integrated side-view camera system.

High-Fidelity Imaging

The legacy dVRK endoscope was replaced with a modern "chip-on-tip" stereo endoscope. This upgrade resulted in a 30-fold increase in image sharpness (measured via Laplacian variance), which is crucial for downstream computer vision tasks like tool tracking and depth estimation.

Capacitive Contact Sensing

To provide objective ground truth for tool-tissue contact—a notoriously difficult metric to capture—we integrated a custom Arduino-based capacitive sensor into the surgical tools. This allows the system to record binary contact signals directly into the data stream, achieving 99.1% accuracy during tissue manipulation tasks.

Dataset and Validation

The SurgSync dataset comprises 214 validated instances of canonical tasks: peg transfer, suturing, knot tying, and tissue manipulation. Data was collected from 13 participants with varying skill levels using ex-vivo tissues (chicken, beef, and pork) to ensure realistic tissue dynamics.

To validate the dataset's utility, we trained a multi-path regression model for surgical skill assessment. The model achieved a Spearman’s rank correlation coefficient (SROCC) of 0.803 for suturing and 0.765 for knot tying, proving that the synchronized multi-modal data effectively captures the nuances of surgical expertise.

Conclusion

SurgSync provides an open-source foundation for the next generation of surgical AI. By solving the challenges of synchronization, visual quality, and contact ground truth, the framework enables more accurate modeling of surgical tasks. All software and the dataset are available at surgsync.github.io.

Summary updated

Abstract

Most existing robotic surgery systems adopt a human-in-the-loop paradigm, often with the surgeon directly teleoperating the robotic system. Adding intelligence to these robots would enable higher-level control, such as supervised autonomy or even full autonomy. However, artificial intelligence (AI) requires large amounts of training data, which is currently lacking. This work proposes SurgSync, a multi-modal data collection framework with offline and online synchronization to support training and real-time inference, respectively. The framework is implemented on a da Vinci Research Kit (dVRK) and introduces (1) dual-mode (online/offline-matching) synchronized recorders, (2) a modern stereo endoscope to achieve image quality on par with clinical systems, and (3) additional sensors such as a side-view camera and a novel capacitive contact sensor to provide ground truth contact data. The framework also incorporates a post-processing toolbox for tasks such as depth estimation, optical flow, and a practical kinematic reprojection method using Gaussian heatmap. User studies with participants of varying skill levels are performed with ex-vivo tissue to provide clinically realistic data, and a network for surgical skill assessment is employed to demonstrate utilization of the collected data. Through the user study experiments, we obtained a dataset of 214 validated instances across multiple canonical training tasks. All software and data are available at surgsync.github.io.
BibTeX
@misc{zhou2026surgsync,
  author = {Haoying Zhou and Chang Liu and Yimeng Wu and Junlin Wu and Zijian Wu and Yu Chung Lee and Sara Martuscelli and Spetimiu E. Salcudean and Gregory S. Fischer and Peter Kazanzides},
  title = {{SurgSync: Time-Synchronized Multi-Modal Data Collection Framework and Dataset for Surgical Robotics}},
  year = {2026},
  eprint = {2603.06919},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.06919},
}
Figure