Abstract
Point of care ultrasound (POCUS) consists in the use of ultrasound imaging in critical or emergency situations to support clinical decisions by healthcare professionals and first responders. In this setting it is essential to be able to provide means to obtain diagnostic data to potentially inexperienced users who did not receive an extensive medical training. Interpretation and acquisition of ultrasound images is not trivial. First, the user needs to find a suitable sound window which can be used to get a clear image, and then he needs to correctly interpret it to perform a diagnosis. Although many recent approaches focus on developing smart ultrasound devices that add interpretation capabilities to existing systems, our goal in this paper is to present a reinforcement learning (RL) strategy which is capable to guide novice users to the correct sonic window and enable them to obtain clinically relevant pictures of the anatomy of interest. We apply our approach to cardiac images acquired from the parasternal long axis (PLAx) view of the left ventricle of the heart.
Introduction
Ultrasound is a cost-effective, portable, and safe imaging modality, making it ideal for emergency medicine and point-of-care diagnostics. Recently, ultra-portable ultrasound probes have entered the market, intended for use by first responders or clinicians with minimal formal sonography training. However, acquiring a clinically useful ultrasound image is notoriously difficult. Before an AI or a clinician can interpret an image, the user must first find the correct "sonic window" by precisely positioning, tilting, and rotating the probe on the patient's body. While much of current medical AI research focuses on interpreting images, this work addresses the prerequisite challenge: providing real-time, physical guidance to a novice user to acquire the image.
Methods
The authors frame probe guidance as a Markov Decision Process (MDP) and solve it using a Deep Q-Network (DQN).
Agent and Action Space
The agent takes the current ultrasound image as input and outputs one of nine discrete actions designed to guide the user. These actions include translations, tilts, and rotations, as well as a "NOP" (No Operation) command to signal that the correct view has been reached.
Simulated Acquisition Environment
Training an RL agent from scratch on live human patients is impractical due to the time required and the erratic nature of early-stage exploration. To solve this, the authors developed an offline simulated acquisition environment. They used an optical tracking system (NDI Polaris Vicra) to record dense, 7x7 mm grid patterns of ultrasound frames across 22 patients' chests. This allowed the RL agent to "navigate" a patient's chest offline, receiving a new frame and a reward based on whether its chosen action moved it closer to or further from the optimal target view (the parasternal long axis, or PLAx).
Baseline Comparison
To evaluate the RL approach, it was compared to a fully supervised learning model. An identical CNN architecture was trained as a classifier, with labels corresponding to the action that would minimize the Manhattan distance to the goal for each grid bin.
Results
The RL approach was tested on its ability to guide a virtual probe from random starting points to the correct sonic window across 5 test patient environments.
- Superior Guidance Performance: The RL agent achieved 86.1% correct guidance (successfully terminating at the target), significantly outperforming the supervised classifier's 77.8%.
- Reliability of Termination: The most striking difference was in the "NOP" (stop) command. The RL agent issued an incorrect stop command only 1.6% of the time, whereas the supervised model was highly prone to false positives, incorrectly telling the user to stop 25.9% of the time.
- Handling Visual Ambiguity: RL handles visual ambiguity better than supervised learning. The human chest can produce similar-looking ultrasound images in different locations. While a supervised classifier struggles with these local ambiguities, the RL agent learns the expected cumulative reward (Q-value), implicitly learning the spatial arrangement of the chest and navigating around ambiguous "dead zones."
Conclusion
This research demonstrates that Deep Reinforcement Learning is a viable strategy for guiding novice users in ultrasound acquisition. By creating a navigable offline environment through optical tracking, the authors successfully trained an agent that outperforms traditional supervised methods, particularly in its ability to handle visual ambiguity and correctly identify the target view. Future work will focus on scaling the data collection process and bridging the gap between discrete simulation and continuous real-world scanning.
Summary updated