Abstract
The challenge of Multimodal Deformable Image Registration (MDIR) lies in the conversion and alignment of features between images of different modalities. Generative models (GMs) cannot retain the necessary information enough from the source modality to the target one, while non-GMs struggle to align features across these two modalities. In this paper, we propose a novel coarse-to-fine MDIR framework, LLM-Morph, which is applicable to various pre-trained Large Language Models (LLMs) to solve these concerns by aligning the deep features from different modal medical images. Specifically, we first utilize a CNN encoder to extract deep visual features from cross-modal image pairs, then we use the first adapter to adjust these tokens, and use LoRA in pre-trained LLMs to fine-tune their weights, both aimed at eliminating the domain gap between the pre-trained LLMs and the MDIR task. Third, for the alignment of tokens, we utilize other four adapters to transform the LLM-encoded tokens into multi-scale visual features, generating multi-scale deformation fields and facilitating the coarse-to-fine MDIR task. Extensive experiments in MR-CT Abdomen and SR-Reg Brain datasets demonstrate the effectiveness of our framework and the potential of pre-trained LLMs for MDIR task. Our code is available at: https://github.com/ninjannn/LLM-Morph.
Introduction
Multimodal Deformable Image Registration (MDIR) is the process of aligning two medical images of the same anatomy taken with different sensors—such as Magnetic Resonance (MR) and Computed Tomography (CT). This is vital for tumor localization and surgical planning, but it is notoriously difficult because the same tissue looks completely different across modalities.
Current approaches fall into two camps, both with flaws. Generative Models (GMs) try to "translate" one modality into another (e.g., making an MR look like a CT), but they often lose critical source-specific details during the translation process. Non-Generative Models try to align features directly but often struggle to find a "common language" between modalities and frequently rely on single-scale deformation fields, which fail to capture large anatomical shifts.
Methods
The authors introduce LLM-Morph, a coarse-to-fine registration framework that treats a pre-trained LLM as an "intermediate modality" to bridge the gap between different image types.
Key components include:
- Feature Extraction: A CNN-based encoder (DualPyramid) extracts multi-scale visual features from the image pairs.
- Tokenization: These visual features are flattened into tokens, similar to how words are processed in text models.
- LLM Encoding Blocks (LEBs): This is the core innovation. The authors insert specific layers from pre-trained LLMs (like LLaMA 3 or Phi3) to process the tokens. They use Adapter modules to map visual tokens into the LLM's hidden dimension and LoRA (Low-Rank Adaptation) to fine-tune the LLM's weights specifically for the registration task.
- Hierarchical Decoding: Four specialized adapters transform the LLM-aligned tokens back into multi-scale visual features, allowing the model to predict deformation fields from coarse to fine resolutions.
Experiments and Results
The model was evaluated on two primary public benchmarks: the Abdomen MR-CT Dataset (sourced from the Learn2Reg challenge) and the SR-Reg Brain Dataset (derived from the SynthRAD 2023 dataset).
LLM-Morph consistently outperformed established baselines like VoxelMorph, TransMorph, and the recent MambaMorph:
- Accuracy: On the Abdomen MR-CT dataset, LLM-Morph achieved a Dice score of 80.11%, a 2.07% improvement over the second-best model (MambaMorph) and a nearly 10% jump over TransMorph.
- Edge Matching: It achieved the lowest 95% Hausdorff Distance (HD95) of 9.89mm on the abdomen dataset, indicating much more accurate boundary alignment.
- Robustness: Qualitative visualizations showed that LLM-Morph was particularly effective at handling large displacements in organs like the liver and spleen, where other models often produced "warped" or unrealistic results.
Discussion
The study highlights the potential of LLMs as Universal Feature Aligners. Despite being trained on text, LLMs possess a structural understanding that can be adapted to align abstract medical image features, acting as a sophisticated bridge that unifies heterogeneous data.
Crucially, the authors found that layer depth matters: middle and later layers of LLMs (e.g., layers 15/16 or 30/31 in LLaMA 3) are significantly more effective for registration than early layers. Early layers focus on general features, while deeper layers capture the complex, high-dimensional relationships necessary for precise anatomical alignment. By using LoRA and lightweight adapters, the power of massive models can be harnessed without the prohibitive cost of full fine-tuning.
Conclusion
While LLM-Morph shows great promise, challenges remain regarding computational overhead and optimal layer selection for specific tasks. Future work will focus on further distilling these models for real-time clinical use and testing the framework on other modalities like PET or Ultrasound.
Summary updated