Explaining the Effectiveness of Multi-Task Learning for Efficient Knowledge Extraction from Spine MRI Reports

Arijit Sehanobish, McCullen Sandora, Nabila Abraham, Jayashri Pawar, Danielle Torres, Anasuya Das, Murray Becker, Richard Herzog, Benjamin Odry, Ron Vianu

Abstract

Extracting structured clinical data from unstructured radiology reports is a critical task in medical AI. While specialized single-task models are common, they are computationally expensive and difficult to scale. This paper explores the use of Multi-Task Learning (MTL) to consolidate multiple pathology extraction tasks into a single efficient model. By analyzing internal representations via Central Kernel Alignment (CKA) and monitoring gradient alignment during training, we demonstrate that different spinal pathology tasks naturally converge on similar features. Our findings show that MTL models achieve performance parity with single-task baselines while providing a 3x inference speedup, offering a scalable solution for high-stakes medical NLP.

Introduction

In medical AI, the transition from unstructured radiologist notes to structured data is essential for downstream clinical decision support. Traditionally, practitioners deploy individual "single-task" models for every specific condition—such as disc herniation or spinal stenosis. However, this approach is resource-intensive, as large Transformer-based models like BERT require significant computational power for both training and deployment. Furthermore, medical data is often scarce and plagued by character-level noise from OCR errors. This study investigates whether Multi-Task Learning (MTL) can provide a more efficient alternative and seeks to explain the underlying mechanisms that allow diverse medical NLP tasks to share a single model backbone effectively.

Methodology

The researchers developed a modular pipeline designed to process complex spine MRI reports. The system first utilizes a "Report Segmenter" to isolate specific motion segments (e.g., L1-L2, L5-S1). These segments are then passed to classifier models to determine the severity of various pathologies.

Overview of the spine MRI report processing pipeline.

To evaluate the synergy between tasks, the study employed two diagnostic tools:

  1. Central Kernel Alignment (CKA): A scalar similarity index used to compare hidden layer representations. High CKA scores indicate that different models are extracting similar features from the same input.
  2. Gradient Alignment: The authors tracked the Average Proportion of Aligned Gradients (APAG) during training to determine if different tasks were pushing the model's weights in consistent directions.

The experiments utilized PubMedBERT as the primary backbone, given its specialized pre-training on medical literature.

Key Insights: Why MTL Works

The study reveals that even when trained independently, models for different spinal pathologies develop remarkably similar internal representations. CKA scores across hidden layers were consistently high, typically exceeding 0.86. This suggests that the linguistic features required to identify "stenosis" are fundamentally similar to those needed for "disc herniation."

CKA similarity scores across hidden layers for various pathology pairs.

Furthermore, the researchers observed a "convergent evolution" in gradients. While tasks may initially conflict during early training epochs, the gradients for different tasks become increasingly aligned as the model matures. This alignment explains the success of MTL: forcing tasks to share a backbone does not cause conflict but rather provides a helpful regularization effect, as the tasks naturally move toward the same mathematical trajectory.

Results and Performance

The MTL approach demonstrated significant advantages in both accuracy and efficiency:

  • Performance Parity: MTL models matched or slightly outperformed single-task baselines. In the Lumbar dataset, the MTL model achieved a Macro F1 of 0.84 for stenosis and 0.86 for nerve root impingement.
  • Inference Efficiency: By consolidating tasks into a single forward pass, the team achieved a 3x improvement in latency per study. This efficiency is vital for production environments where cost and processing speed are paramount.
  • Domain-Specific Vocabulary: PubMedBERT consistently outperformed general-purpose BERT models, confirming that a vocabulary tailored to medical literature is essential for handling the nuances of clinical text.

Conclusion

This research demonstrates that Multi-Task Learning is not only a viable alternative to specialized single-task models but is also scientifically supported by the natural alignment of medical NLP tasks. By sharing internal representations, MTL models achieve high performance with significantly lower computational overhead. Future work will explore whether these findings extend to other medical domains, such as oncology or cardiology, and investigate "gradient surgery" techniques to further optimize multi-task training.

Summary updated

Abstract

Pretrained Transformer based models finetuned on domain specific corpora have changed the landscape of NLP. However, training or fine-tuning these models for individual tasks can be time consuming and resource intensive. Thus, a lot of current research is focused on using transformers for multi-task learning (Raffel et al.,2020) and how to group the tasks to help a multi-task model to learn effective representations that can be shared across tasks (Standley et al., 2020; Fifty et al., 2021). In this work, we show that a single multi-tasking model can match the performance of task specific models when the task specific models show similar representations across all of their hidden layers and their gradients are aligned, i.e. their gradients follow the same direction. We hypothesize that the above observations explain the effectiveness of multi-task learning. We validate our observations on our internal radiologist-annotated datasets on the cervical and lumbar spine. Our method is simple and intuitive, and can be used in a wide range of NLP problems.
BibTeX
@inproceedings{sehanobish2022explaining,
  author = {Arijit Sehanobish and McCullen Sandora and Nabila Abraham and Jayashri Pawar and Danielle Torres and Anasuya Das and Murray Becker and Richard Herzog and Benjamin Odry and Ron Vianu},
  title = {{Explaining the Effectiveness of Multi-Task Learning for Efficient Knowledge Extraction from Spine MRI Reports}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-industry.16},
}
Figure