CLIMAT: Clinically-Inspired Multi-Agent Transformers for Knee Osteoarthritis Trajectory Forecasting

Huy Hoang Nguyen, Simo Saarakkala, Matthew B. Blaschko, Aleksei Tiulpin

TL;DR

CLIMAT is a novel transformer-based framework that mimics the collaborative workflow between a radiologist and a general practitioner to predict how knee osteoarthritis (OA) will progress over eight years. By fusing X-ray image features with clinical data (like BMI and injury history), the model outperforms standard time-series baselines in forecasting the long-term trajectory of the disease.

What Problem Does This Paper Address?

In clinical settings, a diagnosis tells you what is happening now, but a prognosis tells you what will happen later. For chronic, degenerative conditions like knee osteoarthritis, knowing the future trajectory is vital for early intervention. However, most machine learning research focuses on simple classification (Does the patient have OA?) rather than forecasting (How severe will it be in 5 years?).

Predicting this trajectory is difficult because it requires "multimodal" reasoning—combining visual evidence from X-rays with a patient’s medical history. Current methods often struggle to integrate these different data types effectively or require heavy, manual feature engineering. The authors aim to automate this by creating a model that "thinks" like a clinical team.

Methods

The authors introduce CLIMAT (Clinically-Inspired Multi-Agent Transformers), which breaks the forecasting problem into three specialized transformer modules:

  1. Transformer D (The Radiologist): This module focuses on "visual reasoning." It takes an X-ray, divides it into "superpixels," and extracts feature vectors. Its job is to perform a diagnosis of the current state ($y_0$) based on the Kellgren-Lawrence (KL) grading system.
  2. Transformer F (The Fusion Agent): This module processes auxiliary clinical data—such as age, sex, BMI, and WOMAC scores (a measure of pain and function)—to create a "context embedding."
  3. Transformer P (The Prognosis/GP Agent): This is the "decision-maker." It takes the visual findings from the Radiologist and the context from the Fusion agent to forecast the disease's severity at multiple future time points ($y_{1:T}$).

The model is trained using multi-task learning, meaning it learns to diagnose the current state and predict the future simultaneously. This "one-to-many" forecasting approach allows the model to handle missing data points (common in long-term medical studies) using a specialized masking loss function.

Datasets

The study utilizes the Osteoarthritis Initiative (OAI) dataset, a massive longitudinal study of nearly 4,800 participants.

  • Imaging: 2D knee X-rays.
  • Clinical Variables: Age, sex, BMI, history of injury/surgery, and WOMAC scores.
  • Timeline: Data includes a baseline visit and follow-ups at 1, 2, 3, 4, 6, and 8 years.
  • Evaluation: The authors used a "one-center-out" cross-validation, training on data from four clinical sites and testing on a fifth to ensure the model generalizes across different medical facilities.

Key Insights and Ideas

  • Mimicking Human Workflow Works: The core "aha" moment is that structuring a neural network to mirror the interaction between a radiologist and a GP leads to better performance than a "flat" model that just shoves all data into one big transformer.
  • Diagnosis Improves Prognosis: The researchers found that forcing the model to learn the current diagnosis ($y_0$) actually helped it become more accurate at predicting the future. The visual features learned for diagnosis are highly relevant for forecasting.
  • Shallow is Sufficient: Surprisingly, increasing the depth of the transformers didn't help much. A depth of just 2 layers was sufficient, suggesting that for this specific medical task, the structure of the information flow is more important than the sheer number of parameters.
  • Interpretability via Attention: Because it uses transformers, the model provides "attention maps." These showed that the model naturally focused on the intercondylar notch and areas of joint space narrowing—the same regions human doctors look at to assess OA.

Key Results

CLIMAT consistently outperformed standard baselines like LSTMs, GRUs, and other multimodal transformers (MMTF).

  • Near-Future Accuracy: For predictions within 4 years, CLIMAT achieved a Balanced Accuracy (BA) of 55.3% at the 1-year mark, significantly higher than the second-best model (53.7%).
  • Long-Term Forecasting: At the 6 and 8-year marks, CLIMAT maintained lower Root Mean Square Error (RMSE) than sequential models, proving it is more robust for long-term "trajectory" forecasting.
  • Ablation Success: In "ablation studies" (where parts of the model are removed to see what matters), the full CLIMAT model (43.47 Average BA) beat versions that lacked the modular structure or the diagnosis task.

Limitations and Future Work

The primary limitation is that the model was trained and tested on a research-grade dataset (OAI). While the OAI is high quality, real-world clinical data is often messier, with more varied X-ray angles and inconsistent medical records.

The authors suggest that future work should focus on evaluating the model in a real-world clinical setting to see if it truly assists doctors in decision-making. Additionally, while the model identifies what variables are important (like WOMAC scores and sex), further work is needed to understand the complex causal relationships between these variables and disease progression.

Summary updated

Abstract

In medical applications, deep learning methods are designed to automate diagnostic tasks. However, a clinically relevant question that practitioners usually face, is how to predict the future trajectory of a disease (prognosis). Current methods for such a problem often require domain knowledge, and are complicated to apply. In this paper, we formulate the prognosis prediction problem as a one-to-many forecasting problem from multimodal data. Inspired by a clinical decision-making process with two agents – a radiologist and a general practitioner, we model a prognosis prediction problem with two transformer-based components that share information between each other. The first block in this model aims to analyze the imaging data, and the second block leverages the internal representations of the first one as inputs, also fusing them with auxiliary patient data. We show the effectiveness of our method in predicting the development of structural knee osteoarthritis changes over time. Our results show that the proposed method outperforms the state-of-the-art baselines in terms of various performance metrics. In addition, we empirically show that the existence of the multi-agent transformers with depths of 2 is sufficient to achieve good performances. Our code is publicly available at https://github.com/MIPT-Oulu/CLIMAT.
BibTeX
@inproceedings{nguyen2022climat,
  author = {Huy Hoang Nguyen and Simo Saarakkala and Matthew B. Blaschko and Aleksei Tiulpin},
  title = {{CLIMAT: Clinically-Inspired Multi-Agent Transformers for Knee Osteoarthritis Trajectory Forecasting}},
  booktitle = {ISBI},
  year = {2022},
  doi = {10.1109/isbi52829.2022.9761545},
}

Loading related content...

Figure