The first step is the hardest: pitfalls of representing and tokenizing temporal data for large language models

Dimitris Spathis, Fahim Kawsar

TL;DR

Large Language Models (LLMs) are increasingly being applied to sensor and health data, but they face a fundamental "modality gap": their tokenizers are designed for words, not numbers. This paper identifies how standard tokenization mangles temporal data—breaking integers and floats into arbitrary, meaningless chunks—and argues that bridging this gap requires specialized adapters and numerical-aware encoding rather than just clever text prompting.

What Problem Does This Paper Address?

As LLMs evolve into "universal computing engines," researchers are feeding them data from wearables, medical records, and mobile sensors. However, these models were primarily trained on natural language. The authors point out a critical, often overlooked bottleneck: Tokenization.

In natural language, tokenizers (like Byte-Pair Encoding) group frequent character clusters into meaningful units. When applied to time-series or sensor data, this process fails spectacularly. A number like "480" might be a single token, while "481" is split into two. This inconsistency destroys the mathematical and temporal relationships inherent in the data, forcing the model to "memorize" arbitrary patterns rather than "understand" the underlying signal.

Methods

The paper functions as a critical analysis and position paper. It evaluates the three most common subword tokenization methods—Byte-Pair Encoding (BPE), WordPiece, and SentencePiece—through the lens of numerical and temporal data.

The authors propose a hierarchy of solutions to better integrate this data:

  1. Prompt Engineering (Hard Prompts): Using manual text templates to provide context.
  2. Prompt Tuning (Soft Prompts): Adding a small, trainable embedding layer to the input that "learns" how to represent the sensor data for the frozen LLM.
  3. Model Grafting & Adapters: Using specialized encoders (e.g., for ECG or motion data) and mapping their outputs into the LLM’s embedding space.
  4. Specialized Tokenizers: Moving toward digit-level splitting (as seen in LLaMA) to ensure consistent numerical representation.

Datasets

To demonstrate these pitfalls, the authors conduct a case study using the WISDM (Wireless Indoor Surveillance and Device Monitoring) activity recognition dataset. This dataset contains:

  • Participant IDs
  • Activity labels (e.g., walking, running)
  • UNIX timestamps
  • 3-axis accelerometer data (x, y, z)

They processed the first 200 rows of this dataset through the OpenAI tokenizer (used for ChatGPT) to visualize how raw sensor data is fragmented.

Key Insights and Ideas

  • The "Modality Gap": There is a fundamental mismatch between how LLMs "see" text and how they "see" numbers. Without addressing this at the input level, the model's output for sensor tasks is essentially "hallucinated" based on fragmented patterns.
  • The Negative Sign Trap: In their case study, the authors found that negative signs in sensor readings were often grouped with the preceding comma separator (e.g., ,-). This effectively strips the directional information from the sensor value, making it invisible to the model.
  • Context as an Anchor: Previous research cited in the paper shows that adding a simple text string like "Classify the following heart rate sequence" significantly improves performance. Textual context helps "anchor" the numerical data, allowing the LLM to leverage its broader domain knowledge.
  • Inconsistent Chunking: The most "aha" moment is the demonstration of digit chunking. Because tokenizers prioritize frequent strings, common years (like "1998") might be one token, while a specific sensor value (like "3.14159") is shattered into four unrelated IDs (3, ., 14, 159).

Key Results

The WISDM case study revealed that just 200 rows of sensor data exploded into 6,077 tokens and 10,769 characters. The resulting token IDs showed no logical mathematical progression.

The authors highlight that while Zero-shot inference (asking the model to predict without training) performs poorly on sensor data, Prompt Tuning (training a tiny fraction of parameters) dramatically improves performance. This suggests that the "knowledge" to solve these tasks exists within the LLM, but we need better "translators" (adapters) to help the model access that knowledge using non-textual inputs.

Limitations and Future Work

The authors acknowledge that their scope is limited to temporal/numerical data and that most current "successes" in this field rely on heavy preprocessing (downsampling, aggregating, and rounding) to make the data look more like text.

Future work should focus on:

  • Longer Context Windows: Sensor data is high-frequency; LLMs need to handle much longer sequences to capture meaningful behavior.
  • Architectural Constraints: Moving beyond just "fixing" tokenization to creating models that natively understand the "physics" of time-series data.
  • Generalization: Testing if these numerical-aware approaches can translate to other non-language tasks, such as financial forecasting or climate modeling.

Summary updated

Abstract

Large Language Models (LLMs) have demonstrated remarkable generalization across diverse tasks, leading individuals to increasingly use them as personal assistants and universal computing engines. Nevertheless, a notable obstacle emerges when feeding numerical/temporal data into these models, such as data sourced from wearables or electronic health records. LLMs employ tokenizers in their input that break down text into smaller units. However, tokenizers are not designed to represent numerical values and might struggle to understand repetitive patterns and context, treating consecutive values as separate tokens and disregarding their temporal relationships. Here, we discuss recent works that employ LLMs for human-centric tasks such as in mobile health sensing and present a case study showing that popular LLMs tokenize temporal data incorrectly. To address that, we highlight potential solutions such as prompt tuning with lightweight embedding layers as well as multimodal adapters, that can help bridge this "modality gap". While the capability of language models to generalize to other modalities with minimal or no finetuning is exciting, this paper underscores the fact that their outputs cannot be meaningful if they stumble over input nuances.
BibTeX
@article{spathis2024first,
  author = {Dimitris Spathis and Fahim Kawsar},
  title = {{The first step is the hardest: pitfalls of representing and tokenizing temporal data for large language models}},
  journal = {J. Am. Medical Informatics Assoc.},
  year = {2024},
  doi = {10.1093/jamia/ocae090},
}

Loading related content...

Figure