MBAPose: Mask and Bounding-Box Aware Pose Estimation of Surgical Instruments with Photorealistic Domain Randomization

Masakazu Yoshimura, Murilo M. Marinho, Kanako Harada, Mamoru Mitsuishi

TL;DR

MBAPose is a novel deep-learning framework designed to estimate the 3D pose of surgical instruments using only standard endoscopic images. By combining photorealistic rendering, a "mask-aware" double-encoder architecture, and strategic domain randomization, the researchers achieved a 21–26% improvement in accuracy over previous methods, moving closer to reliable "online" robot calibration during surgery.

What Problem Does This Paper Address?

Surgical robots are high-precision machines, but they aren't "set it and forget it." During a procedure, factors like temperature changes, mechanical stress, or interactions with tissue can cause the robot's physical state to drift away from its internal mathematical model. This loss of calibration degrades control accuracy.

While researchers want to use computer vision to "re-calibrate" the robot on the fly, training these algorithms is difficult. Real-world surgical data with perfect "ground truth" labels (knowing exactly where the tool is in 3D space) is incredibly hard to get. Synthetic data is the alternative, but it often suffers from the domain gap—the AI learns to recognize "video game" tools but fails when faced with the messy, bloody, and reflective reality of an actual human body.

Methods

The authors propose a three-pronged approach to bridge the domain gap:

  1. MBAPose Model: This is a "single-stage" pose estimation model. Unlike older models that struggle with small objects, MBAPose uses a double-encoder architecture (ResNet34 and ResNet18). Crucially, it uses Mask Guidance: during training, the model is forced to predict the instrument's silhouette (segmentation mask). This "awareness" of the object's contour significantly improves its ability to calculate 3D orientation.
  2. Photorealistic Rendering: Instead of simple "cut-and-paste" graphics, they used Physically-Based Rendering (PBR). They placed the 3D tool models inside a virtual cube textured with real surgical background images, allowing the metallic instruments to realistically reflect the surrounding environment.
  3. Advanced Domain Randomization (DR): To make the model robust, they randomized lighting, textures, and colors. They introduced a new technique using normal mapping to simulate "metallic scratches" and surface imperfections, mimicking the wear and tear seen on real surgical tools.

Datasets

  • Synthetic Training Set: 90,000 photorealistic images generated in Blender, featuring randomized poses, lighting, and textures.
  • Real-World Test Set: 5,301 images captured using a "BionicBrain" (an anatomically realistic head model).
  • Backgrounds: To vary the environment, they used 2,840 real endoscopic surgical scenes and images from the COCO dataset.
  • Markerless Calibration: The authors developed a custom method to label the real-world dataset by matching 3D CAD models to manually annotated contours in the video frames.

Key Insights and Ideas

  • The "Glossiness" Trap: Interestingly, the researchers found that randomizing the "glossiness" of tools too much actually hurt performance. It turns out that the specific way light glints off metal is a vital "feature" the AI uses to understand the tool's shape.
  • Contours Matter: Adding a segmentation loss (mask awareness) was a "lightbulb moment." By forcing the network to understand the edges of the tool, it became much better at estimating the tool's "pitch" and "yaw" in 3D space.
  • Normal Mapping vs. Color: Simply changing the color of a tool (Domain Randomization) isn't enough. Adding "scratches" via normal mapping provided a more effective way to prepare the AI for the visual noise of a real operating room.

Key Results

  • Synthetic Accuracy: On synthetic test data, the model achieved a translation error of just 0.88 mm and an orientation error of 2.66°.
  • Real-World Performance: When tested on real images (after a small amount of "fine-tuning" with 50 real images), the model maintained a translation error of 2.18 mm and an orientation error of 5.11°.
  • Speed: The model runs at approximately 79.4 FPS on a Titan RTX GPU, making it more than fast enough for real-time surgical use.

Limitations and Future Work

The authors acknowledge that their "ground truth" for real-world data still has an inherent error of 2–3 mm due to the limitations of optical trackers and manual annotation. This makes it hard to prove exactly how accurate the model is in the real world.

Furthermore, the current system looks at images frame-by-frame. The authors suggest that future versions should use temporal information (video sequences) to smooth out predictions and handle moments when the tool is partially hidden by tissue or blood.

Summary updated

Abstract

Surgical robots are usually controlled using a priori models based on the robots’ geometric parameters, which are calibrated before the surgical procedure. One of the challenges in using robots in real surgical settings is that those parameters can change over time, consequently deteriorating control accuracy. In this context, our group has been investigating online calibration strategies without added sensors. In one step toward that goal, we have developed an algorithm to estimate the pose of the instruments’ shafts in endoscopic images. In this study, we build upon that earlier work and propose a new framework to more precisely estimate the pose of a rigid surgical instrument. Our strategy is based on a novel pose estimation model called MBAPose and the use of synthetic training data. Our experiments demonstrated an improvement of 21 % for translation error and 26 % for orientation error on synthetic test data with respect to our previous work. Results with real test data provide a baseline for further research.
BibTeX
@inproceedings{yoshimura2021mbapose,
  author = {Masakazu Yoshimura and Murilo M. Marinho and Kanako Harada and Mamoru Mitsuishi},
  title = {{MBAPose: Mask and Bounding-Box Aware Pose Estimation of Surgical Instruments with Photorealistic Domain Randomization}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=xNdflqPddG},
}

Loading related content...

Figure