Multiple Landmark Detection Using Multi-agent Reinforcement Learning

Athanasios Vlontzos, Amir Alansary, Konstantinos Kamnitsas, Daniel Rueckert, Bernhard Kainz

Abstract

Anatomical landmark detection is a fundamental task in medical image analysis, supporting diagnosis, surgical planning, and fetal growth monitoring. Traditional manual annotation is labor-intensive and prone to inter-observer variability. This paper presents a novel approach using multi-agent reinforcement learning (MARL) to detect multiple landmarks simultaneously. Recognizing that anatomical structures are spatially interdependent, the proposed Collaborative Deep Q-Network (Collab-DQN) enables $K$ agents to navigate 3D environments together. By sharing convolutional weights for feature extraction while maintaining distinct policy heads, the agents achieve a 50% reduction in detection error and significantly faster training times compared to independent single-agent models.

Introduction

Locating specific anatomical points—such as the Anterior Commissure (AC) in the brain—is essential for medical imaging workflows. While deep learning has automated many aspects of this task, most reinforcement learning (RL) approaches treat landmarks as isolated targets. However, the human body is highly structured; the location of one landmark provides significant contextual information about the others. This research exploits these spatial relationships through a MARL framework, aiming to improve both accuracy and computational efficiency.

Methodology: The Collab-DQN Architecture

The core of the proposed system is the Collab-DQN, a multi-agent framework where agents collaborate through implicit communication. The architecture is designed to handle the high dimensionality of 3D medical scans while maintaining manageable computational costs.

The Collab-DQN architecture featuring shared convolutional layers for feature extraction and distinct fully connected modules for individual agent policies.

Shared Features and Specialized Policies

The model employs a Siamese-inspired structure:

  1. Shared Convolutional Layers: All agents share the same convolutional weights, which function as a collective "sensory system." This allows the network to learn robust, generalized anatomical features that benefit all agents.
  2. Individual Policy Heads: Each agent possesses its own set of fully connected (FC) layers. This enables them to develop specialized navigation strategies tailored to their specific target landmark.

Reinforcement Learning Setup

The task is modeled as a Partially Observable Markov Decision Process (POMDP). Each agent observes a small Region of Interest (ROI) within a 3D volume (MRI or Ultrasound). Agents can take six actions (moving along the x, y, or z axes). Rewards are granted based on the reduction in Euclidean distance to the target landmark, incentivizing efficient pathfinding.

Experimental Results

The Collab-DQN was evaluated on three diverse datasets: Brain MRI (ADNI), Cardiac MRI, and Fetal Brain Ultrasound. These datasets present varying levels of difficulty, from the structured brain anatomy to the noisy and variable environment of fetal ultrasounds.

Comparison of detection errors (mean ± std) between supervised CNN, single-agent DQN, and the proposed Collab-DQN across various landmarks.

Accuracy and Efficiency

The results demonstrate significant improvements over state-of-the-art methods. In Brain MRI, the detection error for the Anterior Commissure (AC) was reduced from 2.46mm (using a standard DQN) to just 0.93mm. Furthermore, the collaborative approach converged 25,000 to 50,000 time steps earlier than single-agent counterparts.

The shared architecture also provides a memory advantage, reducing the total parameter count by approximately 5-6% compared to training $K$ separate models. This efficiency makes the system more viable for deployment in resource-constrained clinical environments.

Discussion and Conclusion

The success of Collab-DQN stems from its ability to use joint learning as a form of implicit regularization. By requiring the shared features to satisfy multiple agents simultaneously, the model avoids overfitting to local noise. While the system showed immense gains for most landmarks, some distinct structures (like the Cavum Septum Pellucidum in fetal brains) saw less benefit from collaboration, suggesting that the degree of anatomical interdependence varies.

Future work will explore Bayesian exploration strategies to improve pathfinding in complex volumes and investigate competitive dynamics between agents to further refine detection boundaries in crowded anatomical regions.

Summary updated

Abstract

The detection of anatomical landmarks is a vital step for medical image analysis and applications for diagnosis, interpretation and guidance. Manual annotation of landmarks is a tedious process that requires domain-specific expertise and introduces inter-observer variability. This paper proposes a new detection approach for multiple landmarks based on multi-agent reinforcement learning. Our hypothesis is that the position of all anatomical landmarks is interdependent and non-random within the human anatomy, thus finding one landmark can help to deduce the location of others. Using a Deep Q-Network (DQN) architecture we construct an environment and agent with implicit inter-communication such that we can accommodate K agents acting and learning simultaneously, while they attempt to detect K different landmarks. During training the agents collaborate by sharing their accumulated knowledge for a collective gain. We compare our approach with state-of-the-art architectures and achieve significantly better accuracy by reducing the detection error by $$50\%$$ , while requiring fewer computational resources and time to train compared to the naïve approach of training K agents separately. Code and visualizations available: https://github.com/thanosvlo/MARL-for-Anatomical-Landmark-Detection
BibTeX
@inproceedings{vlontzos2019multiple,
  author = {Athanasios Vlontzos and Amir Alansary and Konstantinos Kamnitsas and Daniel Rueckert and Bernhard Kainz},
  title = {{Multiple Landmark Detection Using Multi-agent Reinforcement Learning}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2019},
  url = {https://openreview.net/forum?id=J8-fVriT_FL},
}
Figure