Abstract
In medical applications, deep learning methods are built to automate diagnostic tasks. However, a clinically relevant question that practitioners usually face, is how to predict the future trajectory of a disease (prognosis). Current methods for such a problem often require domain knowledge, and are complicated to apply. In this paper, we formulate the prognosis prediction problem as a one-to-many forecasting problem from multimodal data. Inspired by a clinical decision-making process with two agents -- a radiologist and a general practitioner, we model a prognosis prediction problem with two transformer-based components that share information between each other. The first block in this model aims to analyze the imaging data, and the second block leverages the internal representations of the first one as inputs, also fusing them with auxiliary patient data. We show the effectiveness of our method in predicting the development of structural knee osteoarthritis changes over time. Our results show that the proposed method outperforms the state-of-the-art baselines in terms of various performance metrics. In addition, we empirically show that the existence of the multi-agent transformers with depths of 2 is sufficient to achieve good performances.
Introduction
Predicting the future trajectory of a disease—prognosis—is a critical yet challenging task in clinical practice. While artificial intelligence has made significant strides in diagnosis (identifying a current condition), forecasting how a disease like knee osteoarthritis (OA) will evolve over several years remains difficult. Knee OA is a degenerative condition with no cure; early and accurate prognosis allows for interventions such as physical therapy or lifestyle changes that can significantly slow disease progression.
The researchers introduce CLIMAT (Clinically-Inspired Multi-Agent Transformers), a framework that treats prognosis as a one-to-many forecasting problem. By mimicking the collaborative workflow between a radiologist and a general practitioner, CLIMAT integrates multimodal data—including X-ray imaging and clinical history—to predict future disease states.
The CLIMAT Framework
The core innovation of CLIMAT is its modular, multi-agent architecture, which replicates the specialized roles in a clinical setting. The framework consists of three primary Transformer modules:
- Transformer D (The Radiologist): This module focuses on visual reasoning. It processes feature vectors from knee X-rays (extracted via a ResNet18 backbone) to predict the patient's current Kellgren-Lawrence (KL) grade.
- Transformer F (The Fusion Engine): This module handles auxiliary clinical data, such as age, BMI, and WOMAC scores (measuring pain and function), projecting them into a shared feature space.
- Transformer P (The Prognosis/GP): Acting as the final decision-maker, this module fuses the internal representations from the "Radiologist" and the "Fusion Engine" to forecast the disease trajectory over an 8-year horizon.
The model is trained using multi-task learning, optimizing for both the current diagnosis and all future predicted time points simultaneously. This structure ensures that the visual features learned are clinically relevant for long-term forecasting.
Performance Evaluation
The framework was evaluated using the Osteoarthritis Initiative (OAI) dataset, which includes longitudinal data from 4,796 individuals over 8 years. The researchers employed a "one-center-out" cross-validation strategy to ensure robust generalization to unseen geographic locations.
CLIMAT was benchmarked against several state-of-the-art sequential and multimodal models, including LSTMs, GRUs, and Multimodal Transformers (MMTF).
As shown in the results, CLIMAT consistently outperformed the baselines. It achieved a Balanced Accuracy (BA) of 55.3% for 1-year predictions, compared to 53.7% for the next-best model. Crucially, CLIMAT maintained superior stability and lower error rates (RMSE) for long-term targets (6–8 years), where traditional sequential models like LSTMs often suffer from error accumulation.
Key Insights and Interpretability
The study yielded several important findings:
- Optimal Depth: A Transformer depth of just 2 layers was found to be sufficient. Increasing depth beyond this point led to overfitting, particularly for near-future predictions.
- Clinical Relevance: The model's internal attention mechanisms naturally focused on clinically significant regions, such as joint space narrowing, without explicit supervision.
- Risk Indicators: The most impactful clinical variables identified by the model—WOMAC scores, sex, and injury history—align with established medical knowledge regarding OA progression.
Conclusion
CLIMAT demonstrates that a clinically-inspired, multi-agent Transformer approach can significantly improve the accuracy of long-term disease prognosis. By effectively fusing imaging and clinical data, the model provides a powerful tool for early intervention. Future work will focus on testing the framework in real-world clinical settings with heterogeneous data quality and adapting it to other degenerative diseases such as Alzheimer's.
Paper Citation: Nguyen, H. H., Saarakkala, S., Blaschko, M. B., & Tiulpin, A. (2021). CLIMAT: Clinically-Inspired Multi-Agent Transformers for Knee Osteoarthritis Trajectory Forecasting. arXiv:2104.03642v3. Link to Code
Summary updated