LLM Itself Can Read and Generate CXR Images

Suhyeon Lee, Won Jun Kim, Jong Chul Ye

Abstract: Following the impressive development of LLMs, vision-language alignment in LLMs is actively being researched to enable multimodal reasoning and visual IO. This direction of research is particularly relevant to medical imaging because medical image analysis and generation consist of reasoning based on a combination of visual features and prior knowledge. Many recent works have focused on training adapter networks that serve as an information bridge between image processing networks and LLMs; but presumably, in order to achieve maximum reasoning potential of LLMs on visual information as well, visual and language features should be allowed to interact more freely. This is especially important in the medical domain because understanding and generating medical images such as chest X-rays (CXR) require not only accurate visual and language-based reasoning but also a more intimate mapping between the two modalities. Thus, taking inspiration from previous work on the transformer and VQ-GAN combination for bidirectional image and text generation, we build upon this approach and develop a method for instruction-tuning an LLM pre-trained only on text to gain vision-language capabilities for medical images. Specifically, we leverage a pretrained LLM's existing question-answering and instruction-following abilities to teach it to understand visual inputs by instructing it to answer questions about image inputs and, symmetrically, output both text and image responses appropriate to a given query by tuning the LLM with diverse tasks that encompass image-based text-generation and text-based image-generation. We show that our model, LLM-CXR, trained in this approach shows better image-text alignment in both CXR understanding and generation tasks while being smaller in size compared to previously developed models that perform a narrower range of tasks.

Introduction

Most current multimodal Large Language Models (LLMs) rely on "adapter networks" to bridge the gap between a frozen image encoder (like CLIP) and a frozen LLM. While effective for general-purpose vision tasks, this approach creates an information bottleneck that is particularly detrimental in the medical domain. In radiology, where subtle textures and specific anatomical locations are critical for diagnosis, this bottleneck can lead to "hallucinations" or missed findings.

Furthermore, most existing models are unidirectional—they can either describe an image or generate one, but rarely both within the same architecture. To truly "understand" medical imaging, a model requires an intimate mapping where visual and textual features interact freely without the constraints of a separate adapter. This paper introduces LLM-CXR, a model that treats chest X-ray (CXR) images and medical text as equals in a shared token space.

Methodology

LLM-CXR is built upon the Dolly-v2-3b (GPT-NeoX) architecture. The methodology centers on three technical pillars designed to facilitate deep vision-language integration:

  1. Clinical Information-Preserving Tokenization: The researchers utilized a VQ-GAN (Vector Quantized Generative Adversarial Network) to convert 256x256 CXR images into 256 discrete tokens. To ensure medical details were preserved during compression, a specialized "clinical loss" was added using features from a pretrained medical encoder (TorchXRayVision).
  2. Shared Embedding Space: Instead of an adapter, the LLM’s vocabulary was expanded to include 1,024 new "image tokens." This allows the LLM to process image data exactly like text tokens, enabling it to "read" and "write" images autoregressively.
  3. Bidirectional Instruction Fine-tuning: The model was trained on four task types simultaneously: standard natural language instruction following (NL-IF), CXR-to-Report generation, Report-to-CXR synthesis, and CXR Visual Question Answering (VQA).
The architecture of LLM-CXR compared to traditional adapter-based models and the instruction fine-tuning pipeline.

Data and Training

The model was trained primarily on the MIMIC-CXR (v2.0.0) dataset, containing over 377,000 CXR-report pairs. To comply with licensing for visual presentations, the Indiana University Chest X-ray Dataset was also utilized.

A critical component of the training was the creation of a Synthetic VQA Dataset. To boost reasoning, the researchers used Llama 2 to generate approximately 126,000 question-answer pairs from existing radiology reports. This taught the model to link specific text descriptions to visual evidence, effectively using VQA as a "supercharger" for performance. By forcing the model to learn "local" features (like the specific location of a lesion) rather than just "global" patterns, the VQA tasks significantly improved standard report generation.

Example of the synthetic VQA generation process where Llama 2 extracts question-answer pairs from a radiology report to ground visual reasoning.

Key Results and Insights

Despite having only 3 billion parameters, LLM-CXR outperformed 7B-parameter models like XrayGPT. This suggests that architectural integration through a shared token space is more critical for domain-specific tasks than raw model size.

  • Superior Image Generation: In the report-to-CXR task, LLM-CXR achieved a Frechet Inception Distance (FID) of 22.75, significantly outperforming specialized models like RoentGen (42.38) and UniXGen (78.19).
  • State-of-the-Art Understanding: On CXR-to-report tasks, the model showed better alignment with ground-truth clinical labels (measured by AUROC and F1 scores) than larger competitors.
  • Zero-Shot Capability: As an instruction-tuned LLM, it can handle novel queries and conversational interactions that specialized, non-LLM medical models cannot.

Limitations and Future Work

While promising, LLM-CXR faces several challenges. Like all LLMs, it occasionally generates "false positives" or hallucinations. The current resolution is limited to 256x256, which may miss microscopic lesions. Additionally, generating a full image (256 tokens) currently takes 30-60 seconds on a consumer GPU, which is a hurdle for real-time clinical use. Future work will focus on longitudinal reasoning, incorporating a patient's history to better mimic a radiologist's workflow.

Summary updated

Abstract

Following the impressive development of LLMs, vision-language alignment in LLMs is actively being researched to enable multimodal reasoning and visual IO. This direction of research is particularly relevant to medical imaging because medical image analysis and generation consist of reasoning based on a combination of visual features and prior knowledge. Many recent works have focused on training adapter networks that serve as an information bridge between image processing networks and LLMs; but presumably, in order to achieve maximum reasoning potential of LLMs on visual information as well, visual and language features should be allowed to interact more freely. This is especially important in the medical domain because understanding and generating medical images such as chest X-rays (CXR) require not only accurate visual and language-based reasoning but also a more intimate mapping between the two modalities. Thus, taking inspiration from previous work on the transformer and VQ-GAN combination for bidirectional image and text generation, we build upon this approach and develop a method for instruction-tuning an LLM pre-trained only on text to gain vision-language capabilities for medical images. Specifically, we leverage a pretrained LLM's existing question-answering and instruction-following abilities to teach it to understand visual inputs by instructing it to answer questions about image inputs and, symmetrically, output both text and image responses appropriate to a given query by tuning the LLM with diverse tasks that encompass image-based text-generation and text-based image-generation. We show that our model, LLM-CXR, trained in this approach shows better image-text alignment in both CXR understanding and generation tasks while being smaller in size compared to previously developed models that perform a narrower range of tasks. The code is at https://github.com/hyn2028/llm-cxr.
BibTeX
@inproceedings{lee2024llm,
  author = {Suhyeon Lee and Won Jun Kim and Jong Chul Ye},
  title = {{LLM Itself Can Read and Generate CXR Images}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=BqHaLnans2},
}
Figure