Abstract
In medical applications, deep learning methods are built to automate diagnostic tasks. However, a clinically relevant question that practitioners usually face, is how to predict the future trajectory of a disease (prognosis). Current methods for such a problem often require domain knowledge, and are complicated to apply. In this paper, we formulate the prognosis prediction problem as a one-to-many forecasting problem from multimodal data. Inspired by a clinical decision-making process with two agents -- a radiologist and a general practitioner, we model a prognosis prediction problem with two transformer-based components that share information between each other. The first block in this model aims to analyze the imaging data, and the second block leverages the internal representations of the first one as inputs, also fusing them with auxiliary patient data. We show the effectiveness of our method in predicting the development of structural knee osteoarthritis changes over time. Our results show that the proposed method outperforms the state-of-the-art baselines in terms of various performance metrics. In addition, we empirically show that the existence of the multi-agent transformers with depths of 2 is sufficient to achieve good performances.
Introduction
In clinical settings, a diagnosis identifies the current state of a patient, but a prognosis—predicting the future trajectory of a disease—is often more critical for long-term management. For chronic, degenerative conditions like knee osteoarthritis (OA), understanding how the disease will progress over several years is vital for early intervention. However, most machine learning research focuses on simple classification (e.g., presence of OA) rather than forecasting long-term severity.
Predicting this trajectory is challenging because it requires multimodal reasoning, combining visual evidence from X-rays with a patient’s medical history (BMI, age, injury history). Current methods often struggle to integrate these disparate data types effectively. The authors propose CLIMAT (Clinically-Inspired Multi-Agent Transformers) to automate this by mimicking the collaborative workflow of a clinical team.
Methodology: The Multi-Agent Approach
CLIMAT breaks the forecasting problem into three specialized transformer modules that simulate the interaction between medical specialists:
- Transformer D (The Radiologist): This module performs visual reasoning. It processes X-ray images, dividing them into "superpixels" to extract feature vectors. Its primary task is to diagnose the current state ($y_0$) based on the Kellgren-Lawrence (KL) grading system.
- Transformer F (The Fusion Agent): This module processes auxiliary clinical data—such as age, sex, BMI, and WOMAC scores—to create a "context embedding."
- Transformer P (The Prognosis/GP Agent): Acting as the final decision-maker, this module integrates the visual findings from the Radiologist and the context from the Fusion agent to forecast disease severity at multiple future time points ($y_{1:T}$).
The model is trained using multi-task learning, simultaneously learning to diagnose the current state and predict the future. This "one-to-many" approach allows the model to handle missing data points, a common issue in longitudinal studies, through a specialized masking loss function.
Experimental Results
The study utilized the Osteoarthritis Initiative (OAI) dataset, involving nearly 4,800 participants with data spanning up to 8 years. Evaluation was performed using "one-center-out" cross-validation to ensure generalizability across different clinical sites.
CLIMAT consistently outperformed standard baselines, including LSTMs, GRUs, and other multimodal transformers (MMTF).
- Accuracy: For near-future predictions (1 year), CLIMAT achieved a Balanced Accuracy (BA) of 55.3%, significantly higher than the second-best model (53.7%).
- Long-Term Robustness: At the 6 and 8-year marks, CLIMAT maintained a lower Root Mean Square Error (RMSE) compared to sequential models, demonstrating its effectiveness for long-term trajectory forecasting.
- Ablation Success: Removing the modular structure or the diagnosis task resulted in lower performance, confirming that the "clinically-inspired" design is key to its success.
Discussion and Conclusion
A key insight from this work is that mirroring human clinical workflows in neural network architecture leads to superior performance. The researchers found that forcing the model to learn a current diagnosis ($y_0$) improved its ability to forecast the future, as visual features relevant for diagnosis are highly predictive of progression. Furthermore, the model's attention maps revealed that it focuses on the same anatomical regions as human doctors, such as the intercondylar notch and joint space narrowing.
While the results are promising, the study is limited by its reliance on research-grade data. Future work should evaluate CLIMAT in real-world clinical settings where data is often messier and less consistent. Nonetheless, CLIMAT represents a significant step forward in automated disease prognosis, providing a robust framework for multimodal longitudinal forecasting.
Summary updated