Interpretable Rheumatoid Arthritis Scoring via Anatomy-Aware Multiple Instance Learning

Zhiyan Bo, Laura C. Coates, Bartlomiej W. Papiez

Introduction

Rheumatoid Arthritis (RA) is a chronic inflammatory condition requiring precise monitoring of disease progression. The gold standard for assessment is the Sharp/van der Heijde (SvdH) scoring system, which evaluates joint erosion and joint space narrowing in hand and foot radiographs. Despite its clinical importance, manual SvdH scoring is labor-intensive—often requiring up to 25 minutes per patient—and prone to significant inter-observer variability.

Automating this process with deep learning presents a dichotomy: granular joint-level models require exhaustive, high-quality annotations that are difficult to scale, while holistic whole-image models often function as "black boxes," potentially relying on spurious non-clinical features (e.g., bone shafts or background artifacts) to make predictions. This study proposes a novel pipeline that combines anatomy-aware region extraction with Attention-Based Multiple Instance Learning (ABMIL) to achieve high-accuracy, interpretable RA scoring.

Methodology

The researchers developed a two-stage pipeline designed to bridge the gap between local joint analysis and global image scoring.

Stage 1: Region Extraction

Two distinct schemes for identifying relevant anatomical regions were evaluated:

  • Scheme 1 (RA-severity based patches): A grid-based approach where a weakly supervised classifier identifies "abnormal" tiles, sampling the top $K$ patches for further analysis.
  • Scheme 2 (Joint patches): An anatomy-aware approach utilizing a multiresolution Hybrid Transformer-CNN (HTC) to detect 37 specific joint landmarks per hand. These landmarks guide the extraction of 25 standardized patches covering the exact joints required for SvdH scoring.

Stage 2: Score Prediction via ABMIL

The extracted patches are processed through a feature extractor (e.g., ResNet-50). An ABMIL network then assigns attention weights to these instances, aggregating them into a single image-level representation to predict the continuous SvdH score.

Overview of the RA scoring pipeline and performance correlation.

Results

The anatomy-aware approach (Scheme 2) consistently outperformed the grid-based sampling of Scheme 1. The best-performing individual model (JLval-ResNet50) achieved a Pearson’s correlation coefficient (PCC) of 0.943. An ensemble of the top models further improved performance to a PCC of 0.945 and a root mean squared error (RMSE) of 15.57.

This performance is highly competitive with experienced radiologists. On the same dataset, human inter-rater reliability reached a PCC of 0.97 with an RMSE of 18.75. Notably, the model's prediction error is lower than the typical disagreement between two human experts, suggesting a high degree of clinical utility.

Interpretability and Visual Validation

A critical component of clinical adoption is trust. By utilizing ABMIL, the model provides inherent interpretability through attention weights. These weights indicate which patches contributed most to the final score.

Comparison of attention maps across different scoring schemes.

Qualitative analysis by a rheumatologist confirmed that Scheme 2 correctly prioritizes damaged joints, such as the metacarpophalangeal joints, whereas whole-image or grid-based methods often highlight irrelevant background areas. This visual evidence confirms that the model is learning clinically relevant features rather than shortcuts.

Conclusion and Future Work

The study demonstrates that forcing deep learning models to focus on specific anatomical landmarks significantly improves both accuracy and interpretability in RA scoring. While the system excels in moderate to late-stage cases, challenges remain in early-stage detection and handling severe finger deformities where landmark detection may fail. Future research will focus on enhancing the robustness of joint localization and conducting large-scale clinical trials to evaluate the pipeline's impact on radiologist workflows.

Summary updated

BibTeX
@inproceedings{bo2025interpretable,
  author = {Zhiyan Bo and Laura C. Coates and Bartlomiej W. Papiez},
  title = {{Interpretable Rheumatoid Arthritis Scoring via Anatomy-Aware Multiple Instance Learning}},
  booktitle = {AMAI @ MICCAI},
  year = {2025},
  doi = {10.1007/978-3-032-09569-5\_6},
}
Figure