TempA-VLP: Temporal-Aware Vision-Language Pretraining for Longitudinal Exploration in Chest X-Ray Image

Zhuoyi Yang, Liyue Shen

Introduction

In clinical practice, radiologists rarely analyze a single chest X-ray (CXR) in isolation. Instead, they perform longitudinal analysis, comparing current scans with prior examinations to determine if a condition is improving, stable, or worsening. While medical reports are rich with comparative temporal language (e.g., "the opacity has increased"), most existing Vision-Language Pre-training (VLP) models are static, pairing a single image with a single report. This leads to "temporal misalignment," where models struggle to map words describing change to the visual differences between two distinct time points.

To address this, the authors propose TempA-VLP, a framework designed to explicitly model temporal changes by comparing current and prior images against temporal descriptions in medical reports.

Methods: The TempA-VLP Architecture

TempA-VLP introduces a dual-stream learning approach to separate static observations from dynamic changes.

  1. Static Contrastive Learning: Matches the current image with "static" report sentences describing the current state.
  2. Temporal-Aware Contrastive Learning: This core innovation utilizes a Cross-Exam Encoder—a 3-layer Transformer that processes image patches from both current and prior exams. By incorporating temporal and positional encodings, the model learns to distinguish between "then" and "now."
  3. Dynamic Representation: The Cross-Exam Encoder generates a "dynamic representation" capturing the differences between scans. This is aligned with "temporal" report phrases (describing change) via a contrastive loss function.
Overall architecture of TempA-VLP and the cross-exam encoder structure.

Datasets and Evaluation

The model was pre-trained on the MIMIC-CXR dataset. To facilitate temporal learning, the authors leveraged the Chest-ImaGenome (Silver) dataset to automatically partition reports into static and temporal components. Evaluation was conducted on:

  • MS-CXR-T: A benchmark for temporal image-text alignment.
  • Chest-ImaGenome (Gold): Used for a new "dynamic phrase grounding" task, identifying where changes occurred based on text queries.

Key Insights

  • Isolating Change: By filtering reports into "what is" versus "what changed," the model avoids the confusion of trying to find temporal trends in a single static image.
  • Dynamic Phrase Grounding: TempA-VLP can localize specific changes (e.g., "Right lower lung atelectasis has completely resolved") across two images.
  • Temporal Context for Static Tasks: Access to dynamic features from a previous exam improves classification performance even on single-image tasks, highlighting "change" as a vital diagnostic signal.

Key Results

TempA-VLP demonstrates superior performance in tracking disease progression and localizing changes. On the MS-CXR-T benchmark, it outperformed the previous state-of-the-art, BioViL-T, particularly in low-data regimes.

Performance comparison between TempA-VLP and BioViL-T across varying amounts of labeled training data.

In a 10% few-shot setting, TempA-VLP achieved 61.6% accuracy compared to BioViL-T's 56.3%. Furthermore, the model showed a 12% to 21% increase in Contrast-to-Noise Ratio (CNR) for localizing changes, and boosted the Area Under the Curve (AUC) for disease classification by approximately 3% in scenarios involving improvement or worsening.

Limitations and Future Work

Current limitations include a dependency on structured scene graphs for report parsing and a focus on only two consecutive exams. Future research could extend this to longer sequences of exams and other 3D longitudinal modalities like CT or MRI scans.

Summary updated

Abstract

Longitudinal medical image processing is a significant task to understand the dynamic changes of disease by taking and comparing image series over time, providing insights into how conditions evolve and enabling more accurate di-agnosis and treatment planning. While recent advance-ments in biomedical Vision-Language Pre-training (VLP) have enabled label-efficient representation learning with paired medical images and reports, existing methods pri-marily pair a single image with the corresponding textual report, limiting their ability to capture temporal relation-ships. To address this limitation, it is essential to learn temporal-aware cross-modal representations from sequen-tial medical images and text reports that highlight the tem-poral changes occurring between examinations. Specifi-cally, we introduce TempA- Vlp, a temporal-aware vision language pre-training framework with a cross-exam en-coder to integrate the information from both prior and cur-rent examinations. This approach enables the model to capture dynamic representations that reflect disease pro-gression over time, which allows us to (i) achieve state-of-the-art performance in disease progression classification, (ii) localize dynamic progression regions across consecutive examinations, as demonstrated in our new task, dynamic phrase grounding on the Chest-Imagenome Gold dataset, and (iii) highlight progression localized regions, often rele-vant to lesion areas, which in turn improves disease classi-fication tasks on a single image.
BibTeX
@inproceedings{yang2025tempa,
  author = {Zhuoyi Yang and Liyue Shen},
  title = {{TempA-VLP: Temporal-Aware Vision-Language Pretraining for Longitudinal Exploration in Chest X-Ray Image}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00454},
}
Figure