Bridging 3D Slicer and ROS2 for Image-Guided Robotic Interventions

Laura Connolly, Anton Deguet, Simon Léonard, Junichi Tokuda, Tamas Ungi, Axel Krieger, Peter Kazanzides, Parvin Mousavi, Gabor Fichtinger, Russell H. TaylorSensors 2022 · DOI

TL;DR

This paper introduces SlicerROS2, a novel module that seamlessly integrates the 3D Slicer medical imaging platform with the Robot Operating System 2 (ROS2). This integration allows for real-time visualization and control of robotic systems within 3D Slicer, significantly simplifying the development of image-guided robotic interventions and reducing the need for custom interfaces.

What Problem Does This Paper Address?

Developing sophisticated image-guided robotic (IGT) systems for medical applications is a complex endeavor. Researchers and engineers often rely on specialized software for different aspects of the system: 3D Slicer is a popular open-source platform for medical image processing and visualization, while ROS/ROS2 is the de facto standard for robotics development, offering a rich ecosystem of tools for motion planning, kinematics, and control.

Historically, bridging these two powerful platforms has been challenging. Existing solutions often involved custom middleware, multiple network hops, and limited access to the full capabilities of either 3D Slicer or ROS. This resulted in time-consuming setup, potential latency issues, and a fragmented development experience. The paper aims to address this by creating a direct, efficient, and generalized bridge that treats 3D Slicer as a ROS2 node, enabling a more unified and streamlined approach to IGT system development.

Methods

The core of this work is the SlicerROS2 module, a loadable C++ plugin for 3D Slicer. The key innovation is treating 3D Slicer as a ROS2 node, allowing for direct communication within a single network hop. This approach leverages ROS2's capabilities for robot description and state management, and integrates them directly into 3D Slicer's visualization environment.

The module focuses on two primary aspects:

  • Robot Description: It utilizes the Universal Robot Description Format (URDF) to describe robot kinematics and geometry. This URDF can be loaded either directly or from the ROS2 parameter server, ensuring flexibility and ease of use.
  • Robot State: The module supports two methods for updating robot state:
  • ROS2 Transform (Tf2): This is the preferred method, leveraging ROS2's Tf2 package to manage the transformation hierarchy of robot links. This approach is generalizable to both serial and parallel linkage robots.
  • Forward Kinematics (FK): For serial link robots, the module can also calculate FK directly within 3D Slicer using the Orocos Kinematics and Dynamics (KDL) library.

Synchronization between ROS2 and 3D Slicer is achieved using a Qt timer within 3D Slicer, which spins the ROS2 node at 100 Hz. This ensures that robot state updates are reflected in the 3D Slicer viewer with minimal delay. The module also handles the translation of transformation hierarchies between 3D Slicer's MRML scene and ROS2's Tf2.

Datasets

Not discussed in the paper. The experiments utilized existing robotic hardware and prerecorded surgical data for demonstration purposes.

Key Insights and Ideas

  • Treating 3D Slicer as a ROS2 Node: This fundamental shift in perspective simplifies the integration significantly, eliminating the need for complex middleware and multiple network hops. It allows for a more direct and efficient data flow.
  • Generalizability through URDF and Tf2: By relying on standard ROS2 formats like URDF and Tf2, the module can accommodate a wide variety of robots, including those with complex parallel linkages, without requiring custom interfaces for each.
  • Minimal Latency for Real-Time Visualization: The direct integration and efficient synchronization mechanism result in very low latency, crucial for real-time robotic control and visualization in surgical scenarios.
  • Prototyping Platform for Advanced IGT: The module serves as a robust starting point for developing more complex IGT applications, such as virtual fixtures for surgical guidance, by providing the foundational integration of imaging and robotics.

Key Results

  • Generalizability: The SlicerROS2 module was successfully tested on two distinct robots: the serial-link Omni Bundle robot and the parallel-link dVRK patient-side manipulator (PSM). The visualization in 3D Slicer was identical to that in RViz (ROS's visualization tool).
  • Latency: The module achieved a reported latency of 11.2 ± 4.5 ms for rendering robot movement. This is significantly lower than previous approaches like the LightweightRobotIGT platform (60 ms for end-effector visualization only). This latency is considered low enough not to significantly impact operator tasks in real-world applications.
  • Surgical Use Case Demonstration: The module was integrated with prerecorded lumpectomy data from the LumpNav system. This demonstrated the ability to spatially and temporally sync anatomy, robot motion, and navigation systems, enabling the visualization of negative and positive margins during surgery.

Limitations and Future Work

  • Operating System Support: While ROS2 supports Linux, Windows, and Mac, the SlicerROS2 module has only been extensively tested on Linux. Further testing is required for other operating systems.
  • Slicer Build Requirement: The module currently requires 3D Slicer to be built from source on the deployment computer, which can be an inconvenience for users.
  • Distribution: The module is not yet set up for simple distribution via 3D Slicer's extension manager.
  • Multi-Robot Visualization: The current implementation has not been tested for displaying two robots simultaneously, which might be necessary for some clinical applications. The performance impact of this complexity needs evaluation.

Future work suggested includes:

  • Testing the module with a wider range of ROS2-compatible devices.
  • Potentially integrating more ROS2 libraries directly into 3D Slicer, such as those for motion planning or simulation.

Summary updated

BibTeX
@article{connolly2022bridging,
  author = {Laura Connolly and Anton Deguet and Simon Léonard and Junichi Tokuda and Tamas Ungi and Axel Krieger and Peter Kazanzides and Parvin Mousavi and Gabor Fichtinger and Russell H. Taylor},
  title = {{Bridging 3D Slicer and ROS2 for Image-Guided Robotic Interventions}},
  journal = {Sensors},
  year = {2022},
  doi = {10.3390/s22145336},
}
Figure