Introduction
Medicine is inherently temporal. Clinicians track how vital signs, lab values, and ECG patterns evolve over time to make informed diagnoses. However, current Large Language Models (LLMs) possess a significant "blind spot" regarding time-series data. OpenTSLM is a novel framework designed to integrate multivariate time-series data as a native modality into LLMs, enabling them to perform natural language reasoning over complex medical sensor data.
Problem Statement
Prior attempts to bridge the gap between LLMs and time-series data often involved converting numerical values into plain text. This approach is token-heavy and loses structural context. Alternatively, "reprogrammed" models acting as simple classifiers lose the LLM's inherent ability to explain its reasoning. The challenge lies in allowing an LLM to process raw sensor data—such as 12-lead ECGs or longitudinal accelerometer data—and reason about it alongside clinical text without exceeding memory constraints.
Methodology
OpenTSLM introduces two primary architectural approaches:
- OpenTSLM-SoftPrompt: Utilizes a time-series encoder (e.g., PatchTST) to convert sensor data into "soft tokens" concatenated directly with text embeddings.
- OpenTSLM-Flamingo: Inspired by the Flamingo vision-language model, this variant treats time-series as a separate modality. A Perceiver Resampler extracts fixed-size latent features, which are fused into the LLM via gated cross-attention layers.
The training involves a two-stage curriculum: a "warmup" on synthetic tasks to learn basic temporal patterns, followed by fine-tuning on complex medical reasoning tasks using a Chain-of-Thought (CoT) approach.
Datasets and Benchmarks
The researchers introduced three novel CoT benchmarks to evaluate multimodal reasoning:
- HAR-CoT: Human Activity Recognition using triaxial accelerometer data.
- Sleep-CoT: Sleep stage classification based on EEG segments.
- ECG-QA-CoT: Question answering over 12-lead ECGs, integrating raw waveforms with clinical context.
Results and Discussion
OpenTSLM models consistently outperformed baselines. In sleep staging (Sleep-CoT), OpenTSLM achieved a 69.88% F1 score, significantly higher than the 9.05% achieved by the best fine-tuned text-only baseline.
A critical finding was the "Scaling Wall" associated with soft prompting. While effective for short sequences, it scales poorly with long inputs. For instance, a 12-lead ECG can exceed 80,000 tokens, requiring over 100GB of VRAM. In contrast, the Flamingo-style cross-attention approach maintains near-constant memory usage regardless of input length.
Expert cardiologists validated the clinical interpretability of the models, finding that the generated rationales correctly identified relevant ECG features in 92.9% of cases.
Conclusion
OpenTSLM demonstrates that treating time-series as a distinct modality is far more effective than text-based representations. The Flamingo-style architecture provides a scalable and memory-efficient solution for long-term clinical monitoring, enabling LLMs to "think" in time-series and provide transparent, reasoned clinical insights.
Summary updated