Straight to the point: reinforcement learning for user guidance in ultrasound

Fausto Milletari, Vighnesh Birodkar, Michal Sofka

Abstract

Point of care ultrasound (POCUS) consists in the use of ultrasound imaging in critical or emergency situations to support clinical decisions by healthcare professionals and first responders. In this setting it is essential to be able to provide means to obtain diagnostic data to potentially inexperienced users who did not receive an extensive medical training. Interpretation and acquisition of ultrasound images is not trivial. First, the user needs to find a suitable sound window which can be used to get a clear image, and then he needs to correctly interpret it to perform a diagnosis. Although many recent approaches focus on developing smart ultrasound devices that add interpretation capabilities to existing systems, our goal in this paper is to present a reinforcement learning (RL) strategy which is capable to guide novice users to the correct sonic window and enable them to obtain clinically relevant pictures of the anatomy of interest. We apply our approach to cardiac images acquired from the parasternal long axis (PLAx) view of the left ventricle of the heart.

Introduction

Ultrasound is a cost-effective, portable, and safe imaging modality, making it ideal for emergency medicine and point-of-care diagnostics. Recently, ultra-portable ultrasound probes have entered the market, intended for use by first responders or clinicians with minimal formal sonography training. However, acquiring a clinically useful ultrasound image is notoriously difficult. Before an AI or a clinician can interpret an image, the user must first find the correct "sonic window" by precisely positioning, tilting, and rotating the probe on the patient's body. While much of current medical AI research focuses on interpreting images, this work addresses the prerequisite challenge: providing real-time, physical guidance to a novice user to acquire the image.

Methods

The authors frame probe guidance as a Markov Decision Process (MDP) and solve it using a Deep Q-Network (DQN).

Agent and Action Space

The agent takes the current ultrasound image as input and outputs one of nine discrete actions designed to guide the user. These actions include translations, tilts, and rotations, as well as a "NOP" (No Operation) command to signal that the correct view has been reached.

Table of the nine discrete actions and their physical effects on the probe's position and orientation.

Simulated Acquisition Environment

Training an RL agent from scratch on live human patients is impractical due to the time required and the erratic nature of early-stage exploration. To solve this, the authors developed an offline simulated acquisition environment. They used an optical tracking system (NDI Polaris Vicra) to record dense, 7x7 mm grid patterns of ultrasound frames across 22 patients' chests. This allowed the RL agent to "navigate" a patient's chest offline, receiving a new frame and a reward based on whether its chosen action moved it closer to or further from the optimal target view (the parasternal long axis, or PLAx).

Experimental setup for data collection, showing the optical tracker and the coordinate transformations used to create the navigable grid of ultrasound images.

Baseline Comparison

To evaluate the RL approach, it was compared to a fully supervised learning model. An identical CNN architecture was trained as a classifier, with labels corresponding to the action that would minimize the Manhattan distance to the goal for each grid bin.

Results

The RL approach was tested on its ability to guide a virtual probe from random starting points to the correct sonic window across 5 test patient environments.

  • Superior Guidance Performance: The RL agent achieved 86.1% correct guidance (successfully terminating at the target), significantly outperforming the supervised classifier's 77.8%.
  • Reliability of Termination: The most striking difference was in the "NOP" (stop) command. The RL agent issued an incorrect stop command only 1.6% of the time, whereas the supervised model was highly prone to false positives, incorrectly telling the user to stop 25.9% of the time.
  • Handling Visual Ambiguity: RL handles visual ambiguity better than supervised learning. The human chest can produce similar-looking ultrasound images in different locations. While a supervised classifier struggles with these local ambiguities, the RL agent learns the expected cumulative reward (Q-value), implicitly learning the spatial arrangement of the chest and navigating around ambiguous "dead zones."

Conclusion

This research demonstrates that Deep Reinforcement Learning is a viable strategy for guiding novice users in ultrasound acquisition. By creating a navigable offline environment through optical tracking, the authors successfully trained an agent that outperforms traditional supervised methods, particularly in its ability to handle visual ambiguity and correctly identify the target view. Future work will focus on scaling the data collection process and bridging the gap between discrete simulation and continuous real-world scanning.

Summary updated

Abstract

Point of care ultrasound (POCUS) consists in the use of ultrasound imaging in critical or emergency situations to support clinical decisions by healthcare professionals and first responders. In this setting it is essential to be able to provide means to obtain diagnostic data to potentially inexperienced users who did not receive an extensive medical training. Interpretation and acquisition of ultrasound images is not trivial. First, the user needs to find a suitable sound window which can be used to get a clear image, and then he needs to correctly interpret it to perform a diagnosis. Although many recent approaches focus on developing smart ultrasound devices that add interpretation capabilities to existing systems, our goal in this paper is to present a reinforcement learning (RL) strategy which is capable to guide novice users to the correct sonic window and enable them to obtain clinically relevant pictures of the anatomy of interest. We apply our approach to cardiac images acquired from the parasternal long axis (PLAx) view of the left ventricle of the heart.
BibTeX
@inproceedings{milletari2019straight,
  author = {Fausto Milletari and Vighnesh Birodkar and Michal Sofka},
  title = {{Straight to the point: reinforcement learning for user guidance in ultrasound}},
  booktitle = {SUSI/PIPPI @ MICCAI},
  year = {2019},
  url = {https://openreview.net/forum?id=c9ZB11s2cK},
}
Figure