Medical Phrase Grounding with Region-Phrase Context Contrastive Alignment

Zhihao Chen, Yang Zhou, Anh Tran, Junting Zhao, Liang Wan, Gideon Ooi, Lionel Cheng, Choon Hua Thng, Xinxing Xu, Yong Liu, Huazhu Fu

Abstract

Medical phrase grounding (MPG) aims to locate the most relevant region in a medical image, given a phrase query describing certain medical findings, which is an important task for medical image analysis and radiological diagnosis. However, existing visual grounding methods rely on general visual features for identifying objects in natural images and are not capable of capturing the subtle and specialized features of medical findings, leading to sub-optimal performance in MPG. In this paper, we propose MedRPG, an end-to-end approach for MPG. MedRPG is built on a lightweight vision-language transformer encoder and directly predicts the box coordinates of mentioned medical findings, which can be trained with limited medical data, making it a valuable tool in medical image analysis. To enable MedRPG to locate nuanced medical findings with better region-phrase correspondences, we further propose Tri-attention Context contrastive alignment (TaCo). TaCo seeks context alignment to pull both the features and attention outputs of relevant region-phrase pairs close together while pushing those of irrelevant regions far away. This ensures that the final box prediction depends more on its finding-specific regions and phrases. Experimental results on three MPG datasets demonstrate that our MedRPG outperforms state-of-the-art visual grounding approaches by a large margin. Additionally, the proposed TaCo strategy is effective in enhancing finding localization ability and reducing spurious region-phrase correlations.

Introduction

Medical Phrase Grounding (MPG) is the task of automatically drawing a bounding box around a clinical finding described in a radiology report. While "natural" visual grounding (e.g., "find the dog under the table") has seen massive progress, medical images present unique hurdles. Medical findings often have very low contrast or subtle textural differences compared to healthy tissue. Furthermore, annotating medical images requires expert radiologists, making large-scale labeled datasets rare and expensive. Finally, general-purpose models trained on natural images often fail to capture the specialized spatial and anatomical relationships crucial for diagnosis. Existing models are often too "heavy," leading to overfitting on small medical datasets, or rely on general visual features that miss fine-grained pathological details.

Methods

The proposed framework, MedRPG, consists of three main components designed to address the challenges of medical domain grounding.

Lightweight Architecture

Instead of a massive multi-modal model, MedRPG uses a ResNet-50 backbone for vision, a pre-trained BERT for text, and a slim 3-layer Vision-Language Transformer (VLT) to fuse the two modalities. This design is intentionally "thin" to prevent overfitting on small datasets.

Direct Bounding Box Prediction

The model appends a special [REG] (regression) token to the input. The final state of this token is fed into a simple MLP to directly predict the $(x, y, w, h)$ coordinates of the finding.

Tri-attention Context Contrastive Alignment (TaCo)

TaCo is the core innovation, utilizing a contrastive loss to ensure robust alignment. At the feature-level, it ensures visual features of the correct region and text features of the phrase are mathematically close. At the attention-level, it aligns the model’s internal attention weights for the [REG] token, the [CLS] (text) token, and the specific region. This prevents the model from relying on spurious correlations.

The MedRPG architecture featuring the TaCo training strategy for region-phrase alignment.

Results and Discussion

MedRPG was evaluated on three datasets: MS-CXR (1,153 triples), ChestX-ray8 (984 images with hand-labeled boxes), and an In-house Dataset (1,824 samples featuring 23 categories of complex abnormalities).

Performance Metrics

MedRPG consistently outperformed state-of-the-art (SOTA) general grounding models:

  • MS-CXR: Achieved 69.86% Accuracy (at IoU > 0.5), outperforming the previous best (TransVG) by 6.1%.
  • ChestX-ray8: Outperformed the second-best method by 4.3% in Accuracy.
  • Ablation Study: Adding the TaCo strategy provided a significant boost (approx. 3.8% in Accuracy) over the baseline MedRPG architecture.

Qualitative Analysis

TaCo ensures the model is actually "looking" at the relevant anatomy. In medical imaging, certain pathologies often appear in specific spots (e.g., pneumothorax at the top of the lung). Without TaCo, models tend to overfit to these common locations. MedRPG with TaCo demonstrates superior ability in finding pathologies in "uncommon" locations, such as the lower lung.

Qualitative comparison showing improved localization accuracy (IoU) with the TaCo strategy.

Conclusion

MedRPG offers a lightweight, effective solution for medical phrase grounding. By keeping the transformer layers few and the fusion simple, the model remains trainable on small medical datasets without losing the ability to capture complex cross-modal relationships. Future work includes extending the model to predict irregular shapes via segmentation masks, grounding multiple findings simultaneously from full reports, and expanding generalizability to CT or MRI scans.

Summary updated

Abstract

Medical phrase grounding (MPG) aims to locate the most relevant region in a medical image, given a phrase query describing certain medical findings, which is an important task for medical image analysis and radiological diagnosis. However, existing visual grounding methods rely on general visual features for identifying objects in natural images and are not capable of capturing the subtle and specialized features of medical findings, leading to sub-optimal performance in MPG. In this paper, we propose MedRPG, an end-to-end approach for MPG. MedRPG is built on a lightweight vision-language transformer encoder and directly predicts the box coordinates of mentioned medical findings, which can be trained with limited medical data, making it a valuable tool in medical image analysis. To enable MedRPG to locate nuanced medical findings with better region-phrase correspondences, we further propose Tri-attention Context contrastive alignment (TaCo). TaCo seeks context alignment to pull both the features and attention outputs of relevant region-phrase pairs close together while pushing those of irrelevant regions far away. This ensures that the final box prediction depends more on its finding-specific regions and phrases. Experimental results on three MPG datasets demonstrate that our MedRPG outperforms state-of-the-art visual grounding approaches by a large margin. Additionally, the proposed TaCo strategy is effective in enhancing finding localization ability and reducing spurious region-phrase correlations.
BibTeX
@inproceedings{chen2023medical,
  author = {Zhihao Chen and Yang Zhou and Anh Tran and Junting Zhao and Liang Wan and Gideon Ooi and Lionel Cheng and Choon Hua Thng and Xinxing Xu and Yong Liu and Huazhu Fu},
  title = {{Medical Phrase Grounding with Region-Phrase Context Contrastive Alignment}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2023},
  url = {https://openreview.net/forum?id=RGJLo2Wdzl9},
}
Figure