Straight to the Point: Reinforcement Learning for User Guidance in Ultrasound

Fausto Milletari, Vighnesh Birodkar, Michal Sofka

TL;DR

This paper introduces a Deep Reinforcement Learning (RL) system designed to guide inexperienced users in operating point-of-care ultrasound (POCUS) devices. By training a Deep Q-Network on spatially tracked, pre-recorded ultrasound sweeps, the system provides real-time probe manipulation instructions (move, tilt, rotate) to help users find the correct cardiac view, significantly outperforming standard supervised learning approaches.

What Problem Does This Paper Address?

Ultrasound is a cheap, portable, and safe imaging modality, making it ideal for emergency medicine and point-of-care diagnostics. Recently, ultra-portable ultrasound probes have hit the market, envisioned to be used by first responders or clinicians with minimal formal sonography training.

However, acquiring a clinically useful ultrasound image is notoriously difficult. Before an AI (or a doctor) can interpret an image, the user must first find the correct "sonic window" by precisely positioning, tilting, and rotating the probe on the patient's body. While much of the current medical AI research focuses on interpreting images, this paper tackles the prerequisite challenge: acquiring the image by providing real-time, turn-by-turn physical guidance to a novice user.

Methods

The authors frame probe guidance as a Markov Decision Process and solve it using a Deep Q-Network (DQN).

  • The Agent's Inputs and Outputs: The agent takes the current ultrasound image as input and outputs one of 9 discrete actions: move (lateral, medial, superior, inferior), tilt (supero-laterally, infero-medially), rotate (clockwise, counter-clockwise), or "NOP" (stop, you have reached the correct view).
  • The Simulation Hack: You cannot train an RL agent from scratch on a live human patient—it would take thousands of hours of erratic probe movements. To solve this, the authors built an offline simulated acquisition environment. They used an optical tracking system to record dense, 7x7 mm grid patterns of ultrasound frames across patients' chests. This allowed the RL agent to "navigate" a patient's chest offline, receiving a new frame and a reward based on whether its chosen action moved it closer to or further from the optimal target view (the parasternal long axis, or PLAx).
  • The Baseline: To prove the value of RL, they compared it to a fully supervised learning approach. They trained an identical CNN architecture as a classifier, labeling each grid bin's images with the action that would minimize the Manhattan distance to the goal.

Datasets

  • Training Data: 22 distinct patient environments, comprising approximately 160,000 tracked ultrasound images.
  • Testing Data: 5 distinct patient environments, comprising approximately 40,000 scans.
  • Data Collection: Data was acquired using an NDI Polaris Vicra optical tracker synchronized with the ultrasound video stream via the open-source PLUS framework.

Key Insights and Ideas

  • RL handles visual ambiguity better than supervised learning: The human chest can produce highly ambiguous ultrasound images; two locations far away from the target might look very similar. A supervised classifier struggles here because it only looks at the immediate image to guess the next move. The RL agent, however, learns the expected cumulative reward (Q-value) of a state. It implicitly learns the spatial arrangement of the chest, assigning very low Q-values to areas far from the goal, allowing it to navigate around ambiguous dead zones.
  • Offline simulation bridges the gap for medical RL: The creation of a "navigable" dataset using optical tracking is a clever blueprint for applying RL to physical medical tasks where live training is unsafe or impractical.

Key Results

The RL approach was tested end-to-end on its ability to guide a virtual probe from a random starting point to the correct sonic window.

  • Superior Guidance: The RL agent achieved 86.1% correct guidance (successfully terminating at the target), compared to only 77.8% for the supervised classifier.
  • Fewer False Stops: The most striking difference was in the "NOP" (stop) command. The RL agent issued an incorrect stop command only 1.6% of the time. The supervised model was highly prone to false positives, telling the user to stop at an incorrect location 25.9% of the time.
  • Reward Efficiency: The RL agent accumulated more positive rewards (69.6% vs 63.1%) and fewer negative rewards (30.3% vs 36.9%) during its trajectories than the supervised counterpart.

Limitations and Future Work

  • Tedious Data Collection: The biggest bottleneck is building the simulated environments. Scanning a dense grid on a single patient takes up to 20 minutes and requires expensive, dedicated optical tracking equipment that suffers if the line-of-sight is blocked.
  • Simulation vs. Reality: While the grid allows for offline simulation, the discrete jumps between pre-recorded frames will slightly differ from the fluid, continuous images obtained during a live scan.
  • Compute Constraints: The authors note that the RL training time is very long, which severely limited their ability to experiment with different hyperparameter tuning.
  • Future Work: The authors view this as a promising first step and suggest future work will focus on making end-to-end guidance more scalable and reliable, likely by finding ways to reduce the friction of the data acquisition process.

Summary updated

Abstract

Point of care ultrasound (POCUS) consists in the use of ultrasound imaging in critical or emergency situations to support clinical decisions by healthcare professionals and first responders. In this setting it is essential to be able to provide means to obtain diagnostic data to potentially inexperienced users who did not receive an extensive medical training. Interpretation and acquisition of ultrasound images is not trivial. First, the user needs to find a suitable sound window which can be used to get a clear image, and then he needs to correctly interpret it to perform a diagnosis. Although many recent approaches focus on developing smart ultrasound devices that add interpretation capabilities to existing systems, our goal in this paper is to present a reinforcement learning (RL) strategy which is capable to guide novice users to the correct sonic window and enable them to obtain clinically relevant pictures of the anatomy of interest. We apply our approach to cardiac images acquired from the parasternal long axis (PLAx) view of the left ventricle of the heart.
BibTeX
@inproceedings{milletari2019straight,
  author = {Fausto Milletari and Vighnesh Birodkar and Michal Sofka},
  title = {{Straight to the Point: Reinforcement Learning for User Guidance in Ultrasound}},
  booktitle = {SUSI/PIPPI @ MICCAI},
  year = {2019},
  url = {https://openreview.net/forum?id=c9ZB11s2cK},
}

Loading related content...

Figure