The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Models

Dimitris Spathis, Fahim Kawsar

Abstract

Large Language Models (LLMs) have demonstrated remarkable generalization across diverse tasks, leading individuals to increasingly use them as personal assistants and universal computing engines. Nevertheless, a notable obstacle emerges when feeding numerical/temporal data into these models, such as data sourced from wearables or electronic health records. LLMs employ tokenizers in their input that break down text into smaller units. However, tokenizers are not designed to represent numerical values and might struggle to understand repetitive patterns and context, treating consecutive values as separate tokens and disregarding their temporal relationships. Here, we discuss recent works that employ LLMs for human-centric tasks such as in mobile health sensing and present a case study showing that popular LLMs tokenize temporal data incorrectly. To address that, we highlight potential solutions such as prompt tuning with lightweight embedding layers as well as multimodal adapters, that can help bridge this "modality gap". While the capability of language models to generalize to other modalities with minimal or no finetuning is exciting, this paper underscores the fact that their outputs cannot be meaningful if they stumble over input nuances.

Introduction

As Large Language Models (LLMs) evolve into "universal computing engines," researchers are increasingly feeding them data from wearables, medical records, and mobile sensors. However, these models were primarily trained on natural language, creating a fundamental "modality gap." Their tokenizers are designed for words, not numbers. This paper identifies how standard tokenization mangles temporal data—breaking integers and floats into arbitrary, meaningless chunks—and argues that bridging this gap requires specialized adapters and numerical-aware encoding rather than just clever text prompting.

The Tokenization Bottleneck

In natural language, tokenizers (like Byte-Pair Encoding) group frequent character clusters into meaningful units. When applied to time-series or sensor data, this process fails. A number like "480" might be a single token, while "481" is split into two. This inconsistency destroys the mathematical and temporal relationships inherent in the data, forcing the model to "memorize" arbitrary patterns rather than "understand" the underlying signal.

One particularly detrimental effect is the "negative sign trap." In sensor readings, negative signs are often grouped with preceding separators (e.g., ,-). This effectively strips directional information from the sensor value, making it invisible or confusing to the model. Furthermore, because tokenizers prioritize frequent strings, common years (like "1998") might be one token, while a specific sensor value (like "3.14159") is shattered into unrelated IDs (e.g., 3, ., 14, 159).

Methodological Hierarchy for Data Integration

The paper evaluates common subword tokenization methods—Byte-Pair Encoding (BPE), WordPiece, and SentencePiece—through the lens of numerical data. To better integrate this data, the authors propose a hierarchy of solutions:

  1. Prompt Engineering (Hard Prompts): Using manual text templates to provide context.
  2. Prompt Tuning (Soft Prompts): Adding a small, trainable embedding layer to the input that "learns" how to represent sensor data for a frozen LLM.
  3. Model Grafting & Adapters: Using specialized encoders (e.g., for ECG or motion data) and mapping their outputs into the LLM’s embedding space.
  4. Specialized Tokenizers: Moving toward digit-level splitting (as seen in LLaMA) to ensure consistent numerical representation.

Case Study: The WISDM Dataset

To demonstrate these pitfalls, the authors conducted a case study using the Wireless Indoor Surveillance and Device Monitoring (WISDM) activity recognition dataset. This dataset contains participant IDs, activity labels, UNIX timestamps, and 3-axis accelerometer data.

Processing just 200 rows of this dataset through a standard tokenizer (e.g., OpenAI's) resulted in an explosion of 6,077 tokens from 10,769 characters. As shown in Figure 1, the resulting token IDs show no logical mathematical progression, illustrating the fragmentation of the signal.

Visualization of the tokenization process for raw sensor data, showing how numerical strings are fragmented into inconsistent tokens and IDs.

Bridging the Modality Gap

The study highlights that while zero-shot inference performs poorly on sensor data, prompt tuning—training a tiny fraction of parameters—dramatically improves performance. This suggests that the "knowledge" to solve these tasks exists within the LLM, but better "translators" (adapters) are needed to help the model access that knowledge using non-textual inputs. Additionally, textual context acts as an anchor; adding a string like "Classify the following heart rate sequence" significantly improves performance by allowing the LLM to leverage its broader domain knowledge.

Conclusion

The scope of current LLM applications to sensor data is often limited by heavy preprocessing (downsampling, aggregating, and rounding) to make data appear more like text. Future work must focus on longer context windows to capture meaningful behavior in high-frequency sensor data and architectural changes that allow models to natively understand the "physics" of time-series data. Ensuring that LLMs can generalize to other modalities requires addressing the fundamental nuances of their input representation.

Summary updated

Abstract

Large Language Models (LLMs) have demonstrated remarkable generalization across diverse tasks, leading individuals to increasingly use them as personal assistants and universal computing engines. Nevertheless, a notable obstacle emerges when feeding numerical/temporal data into these models, such as data sourced from wearables or electronic health records. LLMs employ tokenizers in their input that break down text into smaller units. However, tokenizers are not designed to represent numerical values and might struggle to understand repetitive patterns and context, treating consecutive values as separate tokens and disregarding their temporal relationships. Here, we discuss recent works that employ LLMs for human-centric tasks such as in mobile health sensing and present a case study showing that popular LLMs tokenize temporal data incorrectly. To address that, we highlight potential solutions such as prompt tuning with lightweight embedding layers as well as multimodal adapters, that can help bridge this "modality gap". While the capability of language models to generalize to other modalities with minimal or no finetuning is exciting, this paper underscores the fact that their outputs cannot be meaningful if they stumble over input nuances.
BibTeX
@article{spathis2024first,
  author = {Dimitris Spathis and Fahim Kawsar},
  title = {{The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Models}},
  journal = {J. Am. Medical Informatics Assoc.},
  year = {2024},
  doi = {10.1093/jamia/ocae090},
}
Figure