TL;DR
OpenTSLM is an open-source framework that integrates continuous multivariate time-series data (such as ECGs, EEGs, and accelerometer streams) as a native modality into open-weight large language models. By comparing implicit soft-prompting with an explicit Flamingo-style cross-attention architecture, the authors demonstrate that cross-attention maintains constant GPU memory across sequence lengths while allowing small models (1B–3B parameters) to drastically outperform frontier models like GPT-4o on clinical reasoning tasks.
What Problem Does This Paper Address?
Clinical medicine is fundamentally temporal: diagnosis and patient monitoring depend on longitudinal biomarkers, telemetry streams, and wearable sensor trajectories. However, large language models (LLMs) have a blind spot for continuous time-series data.
Prior efforts to bridge this gap suffer from severe drawbacks:
- Tokenizing numbers as text: Representing numerical values as raw text strings creates tens of thousands of tokens (e.g., a 10-second 12-lead ECG consumes ~80k tokens), saturating context windows, triggering out-of-memory (OOM) errors, and yielding poor numerical reasoning.
- Reprogramming LLMs as classifiers: Replacing language generation heads with classification heads strips the model of its defining capability: natural language reasoning and chain-of-thought (CoT) generation.
- Soft-prompting context limits: Concatenating learned time-series embeddings directly into the prompt context scales quadratically with sequence length and sensor count, quickly exhausting GPU memory on multivariate inputs.
Methods
OpenTSLM frames time-series interpretation as an autoregressive text generation task, training the model to output a structured rationale before predicting an answer (<reasoning> Answer: <final answer>).
The architecture consists of:
- Time-Series Patch Encoder: Continuous signals are split into non-overlapping 1D patches, mapped to embeddings via 1D convolutions, and augmented with learnable positional encodings. To retain signal amplitude lost during $[-1, 1]$ normalization, statistical metadata (mean, standard deviation, sampling frequency) is embedded directly into the text prompt.
- OpenTSLM-SoftPrompt (Implicit Integration): Patch embeddings pass through a Transformer encoder and an MLP projector, and are interleaved directly with text embeddings in the context window.
- OpenTSLM-Flamingo (Explicit Integration): Patch embeddings are compressed via a Perceiver Resampler into a fixed number of latent tokens. Gated cross-attention layers are inserted throughout the frozen LLM backbone, triggered by special modality tokens (
<TS>and<endofchunk>). This decouples the length and channel count of raw sensor inputs from the LLM’s text context length.
Models are trained using a two-stage curriculum: pretraining on synthetic time-series question answering and captioning, followed by task-specific CoT instruction tuning with LoRA. Backbones include Llama-3.2 (1B, 3B) and Gemma-3 (270M, 1B).
Datasets
The authors curated three new multivariate Chain-of-Thought benchmarks (generated using GPT-4o prompted with plots and ground-truth labels, then manually filtered):
- HAR-CoT: Triaxial accelerometer data (50 Hz, 2.56-second windows) combined from six public benchmarks (DOMINO, PAMAP2, DaLiAc, etc.) covering 8 physical activities (~85k samples).
- Sleep-CoT: Single-channel EEG data (Fpz-Cz, 100 Hz, 30-second epochs) from Sleep-EDF covering 5 sleep stages (~9.3k samples).
- ECG-QA-CoT: 12-lead ECGs (100 Hz, 10-second recordings) paired with clinical context from PTB-XL across 42 question templates (~231k samples).
Pretraining tasks used the synthetic TSQA dataset and M4-Captions (derived from the M4 forecasting competition).
Key Insights and Ideas
- Cross-attention eliminates the memory wall: In simulation benchmarks, OpenTSLM-Flamingo maintained near-constant VRAM consumption (~20 GB on Llama-1B, ~61 GB on Llama-3B) as sequence lengths increased up to 10,000 steps across 5 concurrent channels. By contrast, SoftPrompt exploded in memory (>180 GB or OOM), proving cross-attention is necessary for high-frequency or multi-sensor telemetry.
- Raw text tokenization fails silently: Off-the-shelf and fine-tuned text-only baselines frequently degenerated into repetitive loops or began sequentially reciting numbers, scoring near 0.0% F1 on activity recognition.
- Small, specialized models beat frontier giants: Even sub-billion parameter models (Gemma-270M and Llama-1B) outperformed GPT-4o operating over text or plotted images by wide margins.
Key Results
- Sleep Staging (Sleep-CoT): OpenTSLM-SoftPrompt (Llama-1B) attained 69.88% macro-F1 (81.08% accuracy), outperforming fine-tuned text baselines (9.05% F1) and zero-shot GPT-4o (15.47% F1).
- Human Activity Recognition (HAR-CoT): OpenTSLM achieved 65.44% F1 (71.48% accuracy), compared to 2.95% F1 for zero-shot GPT-4o and 10.83% for GPT-4o prompted with image plots.
- Cardiovascular QA (ECG-QA-CoT): OpenTSLM-Flamingo (Llama-3B) achieved 40.25% F1 (46.25% accuracy). Fine-tuned tokenized LLM baselines failed completely due to OOM errors on 12-lead sequences.
- Clinician Evaluation: A panel of five Stanford cardiologists reviewed 84 model-generated ECG rationales under a clinical competence rubric. The model produced fully or partially correct clinical interpretations in 92.9% of cases, achieving an 85.1% positive rate for clinical context integration.
Limitations and Future Work
- Rationale Quality Dependence: Training rationales were distilled from GPT-4o prompted with visual plots—a setup shown to be imperfect. Expert-curated clinical rationales could improve reasoning quality.
- Objective Mismatch: Training via standard autoregressive language modeling does not explicitly penalize the model when the final diagnostic label is incorrect.
- Scaling Representation: Amplitude and duration scale must currently be provided via explicit text prompts rather than being learned directly by the sensor encoder.
- Generalization: Models were evaluated on in-distribution benchmark splits; out-of-distribution transfer to entirely unseen clinical cohorts remains to be proven.
Summary updated