Abstract
Anatomical landmark detection is a fundamental task in medical image analysis, supporting diagnosis, surgical planning, and fetal growth monitoring. Traditional manual annotation is labor-intensive and prone to inter-observer variability. This paper presents a novel approach using multi-agent reinforcement learning (MARL) to detect multiple landmarks simultaneously. Recognizing that anatomical structures are spatially interdependent, the proposed Collaborative Deep Q-Network (Collab-DQN) enables $K$ agents to navigate 3D environments together. By sharing convolutional weights for feature extraction while maintaining distinct policy heads, the agents achieve a 50% reduction in detection error and significantly faster training times compared to independent single-agent models.
Introduction
Locating specific anatomical points—such as the Anterior Commissure (AC) in the brain—is essential for medical imaging workflows. While deep learning has automated many aspects of this task, most reinforcement learning (RL) approaches treat landmarks as isolated targets. However, the human body is highly structured; the location of one landmark provides significant contextual information about the others. This research exploits these spatial relationships through a MARL framework, aiming to improve both accuracy and computational efficiency.
Methodology: The Collab-DQN Architecture
The core of the proposed system is the Collab-DQN, a multi-agent framework where agents collaborate through implicit communication. The architecture is designed to handle the high dimensionality of 3D medical scans while maintaining manageable computational costs.
Shared Features and Specialized Policies
The model employs a Siamese-inspired structure:
- Shared Convolutional Layers: All agents share the same convolutional weights, which function as a collective "sensory system." This allows the network to learn robust, generalized anatomical features that benefit all agents.
- Individual Policy Heads: Each agent possesses its own set of fully connected (FC) layers. This enables them to develop specialized navigation strategies tailored to their specific target landmark.
Reinforcement Learning Setup
The task is modeled as a Partially Observable Markov Decision Process (POMDP). Each agent observes a small Region of Interest (ROI) within a 3D volume (MRI or Ultrasound). Agents can take six actions (moving along the x, y, or z axes). Rewards are granted based on the reduction in Euclidean distance to the target landmark, incentivizing efficient pathfinding.
Experimental Results
The Collab-DQN was evaluated on three diverse datasets: Brain MRI (ADNI), Cardiac MRI, and Fetal Brain Ultrasound. These datasets present varying levels of difficulty, from the structured brain anatomy to the noisy and variable environment of fetal ultrasounds.
Accuracy and Efficiency
The results demonstrate significant improvements over state-of-the-art methods. In Brain MRI, the detection error for the Anterior Commissure (AC) was reduced from 2.46mm (using a standard DQN) to just 0.93mm. Furthermore, the collaborative approach converged 25,000 to 50,000 time steps earlier than single-agent counterparts.
The shared architecture also provides a memory advantage, reducing the total parameter count by approximately 5-6% compared to training $K$ separate models. This efficiency makes the system more viable for deployment in resource-constrained clinical environments.
Discussion and Conclusion
The success of Collab-DQN stems from its ability to use joint learning as a form of implicit regularization. By requiring the shared features to satisfy multiple agents simultaneously, the model avoids overfitting to local noise. While the system showed immense gains for most landmarks, some distinct structures (like the Cavum Septum Pellucidum in fetal brains) saw less benefit from collaboration, suggesting that the degree of anatomical interdependence varies.
Future work will explore Bayesian exploration strategies to improve pathfinding in complex volumes and investigate competitive dynamics between agents to further refine detection boundaries in crowded anatomical regions.
Summary updated