Abstract
Medical phrase grounding (MPG) aims to locate the most relevant region in a medical image, given a phrase query describing certain medical findings, which is an important task for medical image analysis and radiological diagnosis. However, existing visual grounding methods rely on general visual features for identifying objects in natural images and are not capable of capturing the subtle and specialized features of medical findings, leading to sub-optimal performance in MPG. In this paper, we propose MedRPG, an end-to-end approach for MPG. MedRPG is built on a lightweight vision-language transformer encoder and directly predicts the box coordinates of mentioned medical findings, which can be trained with limited medical data, making it a valuable tool in medical image analysis. To enable MedRPG to locate nuanced medical findings with better region-phrase correspondences, we further propose Tri-attention Context contrastive alignment (TaCo). TaCo seeks context alignment to pull both the features and attention outputs of relevant region-phrase pairs close together while pushing those of irrelevant regions far away. This ensures that the final box prediction depends more on its finding-specific regions and phrases. Experimental results on three MPG datasets demonstrate that our MedRPG outperforms state-of-the-art visual grounding approaches by a large margin. Additionally, the proposed TaCo strategy is effective in enhancing finding localization ability and reducing spurious region-phrase correlations.
Introduction
Medical Phrase Grounding (MPG) is the task of automatically drawing a bounding box around a clinical finding described in a radiology report. While "natural" visual grounding (e.g., "find the dog under the table") has seen massive progress, medical images present unique hurdles. Medical findings often have very low contrast or subtle textural differences compared to healthy tissue. Furthermore, annotating medical images requires expert radiologists, making large-scale labeled datasets rare and expensive. Finally, general-purpose models trained on natural images often fail to capture the specialized spatial and anatomical relationships crucial for diagnosis. Existing models are often too "heavy," leading to overfitting on small medical datasets, or rely on general visual features that miss fine-grained pathological details.
Methods
The proposed framework, MedRPG, consists of three main components designed to address the challenges of medical domain grounding.
Lightweight Architecture
Instead of a massive multi-modal model, MedRPG uses a ResNet-50 backbone for vision, a pre-trained BERT for text, and a slim 3-layer Vision-Language Transformer (VLT) to fuse the two modalities. This design is intentionally "thin" to prevent overfitting on small datasets.
Direct Bounding Box Prediction
The model appends a special [REG] (regression) token to the input. The final state of this token is fed into a simple MLP to directly predict the $(x, y, w, h)$ coordinates of the finding.
Tri-attention Context Contrastive Alignment (TaCo)
TaCo is the core innovation, utilizing a contrastive loss to ensure robust alignment. At the feature-level, it ensures visual features of the correct region and text features of the phrase are mathematically close. At the attention-level, it aligns the model’s internal attention weights for the [REG] token, the [CLS] (text) token, and the specific region. This prevents the model from relying on spurious correlations.
Results and Discussion
MedRPG was evaluated on three datasets: MS-CXR (1,153 triples), ChestX-ray8 (984 images with hand-labeled boxes), and an In-house Dataset (1,824 samples featuring 23 categories of complex abnormalities).
Performance Metrics
MedRPG consistently outperformed state-of-the-art (SOTA) general grounding models:
- MS-CXR: Achieved 69.86% Accuracy (at IoU > 0.5), outperforming the previous best (TransVG) by 6.1%.
- ChestX-ray8: Outperformed the second-best method by 4.3% in Accuracy.
- Ablation Study: Adding the TaCo strategy provided a significant boost (approx. 3.8% in Accuracy) over the baseline MedRPG architecture.
Qualitative Analysis
TaCo ensures the model is actually "looking" at the relevant anatomy. In medical imaging, certain pathologies often appear in specific spots (e.g., pneumothorax at the top of the lung). Without TaCo, models tend to overfit to these common locations. MedRPG with TaCo demonstrates superior ability in finding pathologies in "uncommon" locations, such as the lower lung.
Conclusion
MedRPG offers a lightweight, effective solution for medical phrase grounding. By keeping the transformer layers few and the fusion simple, the model remains trainable on small medical datasets without losing the ability to capture complex cross-modal relationships. Future work includes extending the model to predict irregular shapes via segmentation masks, grounding multiple findings simultaneously from full reports, and expanding generalizability to CT or MRI scans.
Summary updated