: pitfalls of representing and tokenizing temporal data for large language models

Dimitris Spathis, Fahim KawsarJ. Am. Medical Informatics Assoc 2024 · OpenReview

TL;DR

Large Language Models (LLMs) struggle to effectively process temporal data (like sensor readings) because their tokenizers break down numbers and patterns in ways that lose crucial temporal relationships. This paper highlights these pitfalls and explores solutions like prompt tuning and multimodal adapters to bridge the "modality gap" between text and time-series data.

What Problem Does This Paper Address?

LLMs have become incredibly versatile, acting as powerful tools for a wide range of tasks. However, a significant hurdle arises when we try to feed them data that isn't purely text, especially temporal data from sources like wearables or health records. LLMs rely on "tokenizers" to break down input into manageable units. The problem is, these tokenizers are designed for language, not for the inherent patterns and continuous nature of numerical and temporal sequences. They tend to fragment numbers and ignore the sequential relationships between data points, rendering the LLM's understanding of this data superficial and potentially meaningless. This paper aims to shed light on these tokenization issues and propose ways to make LLMs more effective with temporal data.

Methods

The paper doesn't introduce a new model or algorithm but rather analyzes the existing tokenization mechanisms within LLMs and proposes solutions. The core of their argument lies in identifying the shortcomings of standard tokenizers when faced with temporal data. They then discuss and advocate for several approaches to overcome these limitations:

  • Prompt Tuning: This technique involves adding trainable "soft prompts" (small tensors) to the input. These prompts are optimized to help the LLM better understand and process the temporal data, acting as a bridge between the raw input and the LLM's existing knowledge.
  • Multimodal Adapters: These are small, trainable modules inserted into the LLM's architecture. They are designed to translate data from one modality (like time-series) into a format that the LLM can understand, effectively grafting new capabilities onto the pre-trained model.
  • New Tokenizers: The paper also briefly touches upon the idea of designing specialized tokenizers that can better handle numerical and temporal data, citing examples where splitting digits into individual tokens improved arithmetic reasoning.

Datasets

The paper uses the WISDM activity recognition dataset for a case study. This dataset contains participant ID, activity label, UNIX timestamp, and accelerometer readings (x, y, z axes). The authors specifically chose not to pre-process this data to highlight the raw tokenization issues.

Key Insights and Ideas

  • Tokenization is the "First Step" Bottleneck: The paper strongly emphasizes that how temporal data is initially tokenized is the most critical factor determining an LLM's ability to understand it. If this step fails, subsequent processing is compromised.
  • Temporal Relationships are Lost: Standard tokenizers break down numbers and timestamps into arbitrary sub-units, destroying the sequential and contextual information vital for understanding time-series data. For example, a timestamp might be split into multiple tokens, making it impossible for the LLM to infer the passage of time.
  • "Modality Gap" is Real and Problematic: There's a fundamental mismatch between how LLMs process language and how temporal data is structured. This "modality gap" prevents LLMs from leveraging their full potential when dealing with sensor data.
  • Prompt Engineering Isn't Enough: While prompt engineering can help guide LLMs, it's insufficient on its own to overcome the deep-seated tokenization issues. More robust methods are needed.
  • Parameter-Efficient Fine-Tuning (PEFT) is Key: Techniques like prompt tuning offer a way to adapt LLMs to new data modalities without the prohibitive cost of full fine-tuning, making them more practical for real-world applications.

Key Results

The paper presents a qualitative analysis rather than quantitative results from extensive experiments. The key finding is demonstrated through the case study on the WISDM dataset:

  • Incorrect Tokenization of Temporal Data: The paper shows concrete examples of how timestamps and accelerometer values are broken down into meaningless tokens. For instance, negative signs in sensor readings are grouped with commas, and numbers are fragmented in ways that obscure their numerical value and temporal context. This fragmentation makes it impossible for LLMs to perform meaningful inference on the raw data.

Limitations and Future Work

  • Limitations: The paper acknowledges that current LLMs, due to their tokenization mechanisms, are not well-suited for direct processing of raw temporal data. The "modality gap" remains a significant challenge. The authors also note that while adapter-based methods exist for image-to-text, time-series to text is less explored.
  • Future Work: The authors suggest several avenues for future research:
  • Developing more sophisticated tokenizers that can better represent numerical and temporal data.
  • Further exploring and refining parameter-efficient fine-tuning techniques like prompt tuning.
  • Investigating and developing multimodal adapters specifically for time-series data.
  • Rethinking LLM architectures to better accommodate longer context windows, which are crucial for handling extensive temporal datasets.
  • Generalizing these approaches to other non-language tasks.

Summary updated

Abstract

Large language models (LLMs) have demonstrated remarkable generalization and across diverse tasks, leading individuals to increasingly use them as personal assistants due to their emerging reasoning capabilities. Nevertheless, a notable obstacle emerges when including numerical/temporal data into these prompts, such as data sourced from wearables or electronic health records. LLMs employ tokenizers in their input that break down text into smaller units. However, tokenizers are not designed to represent numerical values and might struggle to understand repetitive patterns and context, treating consecutive values as separate tokens and disregarding their temporal relationships. This article discusses the challenges of representing and tokenizing temporal data. It argues that naively passing timeseries to LLMs can be ineffective due to the modality gap between numbers and text.
BibTeX
@article{spathis2024pitfalls,
  author = {Dimitris Spathis and Fahim Kawsar},
  title = {{: pitfalls of representing and tokenizing temporal data for large language models}},
  journal = {J. Am. Medical Informatics Assoc.},
  year = {2024},
  url = {https://openreview.net/forum?id=Vsr0PlMOKF},
}
Figure