OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Data

Patrick Langer, Thomas Kaar, Max Rosenblattl, Maxwell A Xu, Winnie Chow, Martin Maritsch, Aradhana Verma, Brian Han, Daniel Seung Kim, Henry Chubb, Scott R. Ceresnak, Aydin Zahedivash, Alexander T Sandhu, Fatima Rodriguez, Daniel McDuff, Elgar Fleisch, Oliver Oppers Aalami, Filipe Barata, Paul SchmiedmayerSubmission to ICLR 2026 · Rejected

TL;DR

OpenTSLM is an open-source framework that integrates continuous multivariate time-series data (such as ECGs, EEGs, and accelerometer streams) as a native modality into open-weight large language models. By comparing implicit soft-prompting with an explicit Flamingo-style cross-attention architecture, the authors demonstrate that cross-attention maintains constant GPU memory across sequence lengths while allowing small models (1B–3B parameters) to drastically outperform frontier models like GPT-4o on clinical reasoning tasks.

What Problem Does This Paper Address?

Clinical medicine is fundamentally temporal: diagnosis and patient monitoring depend on longitudinal biomarkers, telemetry streams, and wearable sensor trajectories. However, large language models (LLMs) have a blind spot for continuous time-series data.

Prior efforts to bridge this gap suffer from severe drawbacks:

  1. Tokenizing numbers as text: Representing numerical values as raw text strings creates tens of thousands of tokens (e.g., a 10-second 12-lead ECG consumes ~80k tokens), saturating context windows, triggering out-of-memory (OOM) errors, and yielding poor numerical reasoning.
  2. Reprogramming LLMs as classifiers: Replacing language generation heads with classification heads strips the model of its defining capability: natural language reasoning and chain-of-thought (CoT) generation.
  3. Soft-prompting context limits: Concatenating learned time-series embeddings directly into the prompt context scales quadratically with sequence length and sensor count, quickly exhausting GPU memory on multivariate inputs.

Methods

OpenTSLM frames time-series interpretation as an autoregressive text generation task, training the model to output a structured rationale before predicting an answer (<reasoning> Answer: <final answer>).

The architecture consists of:

  • Time-Series Patch Encoder: Continuous signals are split into non-overlapping 1D patches, mapped to embeddings via 1D convolutions, and augmented with learnable positional encodings. To retain signal amplitude lost during $[-1, 1]$ normalization, statistical metadata (mean, standard deviation, sampling frequency) is embedded directly into the text prompt.
  • OpenTSLM-SoftPrompt (Implicit Integration): Patch embeddings pass through a Transformer encoder and an MLP projector, and are interleaved directly with text embeddings in the context window.
  • OpenTSLM-Flamingo (Explicit Integration): Patch embeddings are compressed via a Perceiver Resampler into a fixed number of latent tokens. Gated cross-attention layers are inserted throughout the frozen LLM backbone, triggered by special modality tokens (<TS> and <endofchunk>). This decouples the length and channel count of raw sensor inputs from the LLM’s text context length.

Models are trained using a two-stage curriculum: pretraining on synthetic time-series question answering and captioning, followed by task-specific CoT instruction tuning with LoRA. Backbones include Llama-3.2 (1B, 3B) and Gemma-3 (270M, 1B).

Datasets

The authors curated three new multivariate Chain-of-Thought benchmarks (generated using GPT-4o prompted with plots and ground-truth labels, then manually filtered):

  1. HAR-CoT: Triaxial accelerometer data (50 Hz, 2.56-second windows) combined from six public benchmarks (DOMINO, PAMAP2, DaLiAc, etc.) covering 8 physical activities (~85k samples).
  2. Sleep-CoT: Single-channel EEG data (Fpz-Cz, 100 Hz, 30-second epochs) from Sleep-EDF covering 5 sleep stages (~9.3k samples).
  3. ECG-QA-CoT: 12-lead ECGs (100 Hz, 10-second recordings) paired with clinical context from PTB-XL across 42 question templates (~231k samples).

Pretraining tasks used the synthetic TSQA dataset and M4-Captions (derived from the M4 forecasting competition).

Key Insights and Ideas

  • Cross-attention eliminates the memory wall: In simulation benchmarks, OpenTSLM-Flamingo maintained near-constant VRAM consumption (~20 GB on Llama-1B, ~61 GB on Llama-3B) as sequence lengths increased up to 10,000 steps across 5 concurrent channels. By contrast, SoftPrompt exploded in memory (>180 GB or OOM), proving cross-attention is necessary for high-frequency or multi-sensor telemetry.
  • Raw text tokenization fails silently: Off-the-shelf and fine-tuned text-only baselines frequently degenerated into repetitive loops or began sequentially reciting numbers, scoring near 0.0% F1 on activity recognition.
  • Small, specialized models beat frontier giants: Even sub-billion parameter models (Gemma-270M and Llama-1B) outperformed GPT-4o operating over text or plotted images by wide margins.

Key Results

  • Sleep Staging (Sleep-CoT): OpenTSLM-SoftPrompt (Llama-1B) attained 69.88% macro-F1 (81.08% accuracy), outperforming fine-tuned text baselines (9.05% F1) and zero-shot GPT-4o (15.47% F1).
  • Human Activity Recognition (HAR-CoT): OpenTSLM achieved 65.44% F1 (71.48% accuracy), compared to 2.95% F1 for zero-shot GPT-4o and 10.83% for GPT-4o prompted with image plots.
  • Cardiovascular QA (ECG-QA-CoT): OpenTSLM-Flamingo (Llama-3B) achieved 40.25% F1 (46.25% accuracy). Fine-tuned tokenized LLM baselines failed completely due to OOM errors on 12-lead sequences.
  • Clinician Evaluation: A panel of five Stanford cardiologists reviewed 84 model-generated ECG rationales under a clinical competence rubric. The model produced fully or partially correct clinical interpretations in 92.9% of cases, achieving an 85.1% positive rate for clinical context integration.

Limitations and Future Work

  • Rationale Quality Dependence: Training rationales were distilled from GPT-4o prompted with visual plots—a setup shown to be imperfect. Expert-curated clinical rationales could improve reasoning quality.
  • Objective Mismatch: Training via standard autoregressive language modeling does not explicitly penalize the model when the final diagnostic label is incorrect.
  • Scaling Representation: Amplitude and duration scale must currently be provided via explicit text prompts rather than being learned directly by the sensor encoder.
  • Generalization: Models were evaluated on in-distribution benchmark splits; out-of-distribution transfer to entirely unseen clinical cohorts remains to be proven.

Summary updated

Abstract

Large Language Models have emerged as powerful tools for interpreting multimodal data (e.g., images, audio, text), often surpassing specialized models. In medicine, they hold particular promise for synthesizing large volumes of clinical information into actionable insights and patient-facing digital health applications. Yet, a major limitation remains their inability to handle time series data. To overcome this gap, we present OpenTSLM, a family of Time Series Language Models created by integrating time series as a native modality to pretrained LLMs, enabling natural-language prompting and reasoning over multiple time series of any length. We investigate two architectures that differ in how they model time series. The first, OpenTSLM-SoftPrompt, models time series implicitly by concatenating learnable time series tokens with text tokens via soft prompting. Although parameter-efficient, we hypothesize that explicit time series modeling scales better and outperforms implicit approaches. We thus introduce OpenTSLM-Flamingo, which integrates time series with text via cross-attention. We benchmark both variants with LLaMa and Gemma backbones against baselines that treat time series as text tokens or plots, across a suite of text–time-series reasoning tasks. We introduce three time-series Chain-of-Thought (CoT) datasets: HAR-CoT (human activity recognition), Sleep-CoT (sleep staging), and ECG-QA-CoT (ECG question answering). Across all, OpenTSLM models consistently outperform baselines, reaching 69.9% F1 in sleep staging and 65.4% in HAR, compared to 9.05% and 52.2% for finetuned text-only models. Notably, even 1B-parameter OpenTSLM models surpass GPT-4o (15.47% and 2.95%). OpenTSLM-Flamingo matches OpenTSLM-SoftPrompt in performance and outperforms on longer sequences, while maintaining stable memory requirements. By contrast, SoftPrompt exhibits exponential memory growth with sequence length, requiring ~110 GB compared to ~40 GB VRAM when training on ECG-QA with LLaMA-3B. Expert reviews by clinicians find strong reasoning capabilities and temporal understanding of raw sensor data exhibited by OpenTSLMs on ECG-QA. To facilitate further research, we provide all code, datasets, and models open-source.
BibTeX
@inproceedings{langer2026opentslm,
  author = {Patrick Langer and Thomas Kaar and Max Rosenblattl and Maxwell A Xu and Winnie Chow and Martin Maritsch and Aradhana Verma and Brian Han and Daniel Seung Kim and Henry Chubb and Scott R. Ceresnak and Aydin Zahedivash and Alexander T Sandhu and Fatima Rodriguez and Daniel McDuff and Elgar Fleisch and Oliver Oppers Aalami and Filipe Barata and Paul Schmiedmayer},
  title = {{OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Data}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=FHDRzhKm0f},
}
Figure