OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Data

Patrick Langer, Thomas Kaar, Max Rosenblattl, Maxwell A. Xu, Winnie Chow, Martin Maritsch, Aradhana Verma, Brian Han, Daniel Seung Kim, Henry Chubb, Scott R. Ceresnak, Aydin Zahedivash, Alexander T. Sandhu, Fátima Rodriguez, Daniel McDuff, Elgar Fleisch, Oliver O. Aalami, Filipe Barata, Paul Schmiedmayer

Introduction

Medicine is inherently temporal. Clinicians track how vital signs, lab values, and ECG patterns evolve over time to make informed diagnoses. However, current Large Language Models (LLMs) possess a significant "blind spot" regarding time-series data. OpenTSLM is a novel framework designed to integrate multivariate time-series data as a native modality into LLMs, enabling them to perform natural language reasoning over complex medical sensor data.

Overview of the OpenTSLM framework and the Text-Time-Series-Prompt (TTSP) approach for generating Chain-of-Thought rationales.

Problem Statement

Prior attempts to bridge the gap between LLMs and time-series data often involved converting numerical values into plain text. This approach is token-heavy and loses structural context. Alternatively, "reprogrammed" models acting as simple classifiers lose the LLM's inherent ability to explain its reasoning. The challenge lies in allowing an LLM to process raw sensor data—such as 12-lead ECGs or longitudinal accelerometer data—and reason about it alongside clinical text without exceeding memory constraints.

Methodology

OpenTSLM introduces two primary architectural approaches:

  1. OpenTSLM-SoftPrompt: Utilizes a time-series encoder (e.g., PatchTST) to convert sensor data into "soft tokens" concatenated directly with text embeddings.
  2. OpenTSLM-Flamingo: Inspired by the Flamingo vision-language model, this variant treats time-series as a separate modality. A Perceiver Resampler extracts fixed-size latent features, which are fused into the LLM via gated cross-attention layers.

The training involves a two-stage curriculum: a "warmup" on synthetic tasks to learn basic temporal patterns, followed by fine-tuning on complex medical reasoning tasks using a Chain-of-Thought (CoT) approach.

Datasets and Benchmarks

The researchers introduced three novel CoT benchmarks to evaluate multimodal reasoning:

  • HAR-CoT: Human Activity Recognition using triaxial accelerometer data.
  • Sleep-CoT: Sleep stage classification based on EEG segments.
  • ECG-QA-CoT: Question answering over 12-lead ECGs, integrating raw waveforms with clinical context.

Results and Discussion

OpenTSLM models consistently outperformed baselines. In sleep staging (Sleep-CoT), OpenTSLM achieved a 69.88% F1 score, significantly higher than the 9.05% achieved by the best fine-tuned text-only baseline.

A critical finding was the "Scaling Wall" associated with soft prompting. While effective for short sequences, it scales poorly with long inputs. For instance, a 12-lead ECG can exceed 80,000 tokens, requiring over 100GB of VRAM. In contrast, the Flamingo-style cross-attention approach maintains near-constant memory usage regardless of input length.

Peak VRAM usage comparison between SoftPrompt and Flamingo architectures, highlighting the memory efficiency of the Flamingo-style approach for long-term clinical monitoring.

Expert cardiologists validated the clinical interpretability of the models, finding that the generated rationales correctly identified relevant ECG features in 92.9% of cases.

Conclusion

OpenTSLM demonstrates that treating time-series as a distinct modality is far more effective than text-based representations. The Flamingo-style architecture provides a scalable and memory-efficient solution for long-term clinical monitoring, enabling LLMs to "think" in time-series and provide transparent, reasoned clinical insights.

Summary updated

BibTeX
@inproceedings{langer2026opentslm,
  author = {Patrick Langer and Thomas Kaar and Max Rosenblattl and Maxwell A. Xu and Winnie Chow and Martin Maritsch and Aradhana Verma and Brian Han and Daniel Seung Kim and Henry Chubb and Scott R. Ceresnak and Aydin Zahedivash and Alexander T. Sandhu and Fátima Rodriguez and Daniel McDuff and Elgar Fleisch and Oliver O. Aalami and Filipe Barata and Paul Schmiedmayer},
  title = {{OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Data}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=FHDRzhKm0f},
}
Figure