TL;DR
Large Language Models (LLMs) are increasingly being applied to sensor and health data, but they face a fundamental "modality gap": their tokenizers are designed for words, not numbers. This paper identifies how standard tokenization mangles temporal data—breaking integers and floats into arbitrary, meaningless chunks—and argues that bridging this gap requires specialized adapters and numerical-aware encoding rather than just clever text prompting.
What Problem Does This Paper Address?
As LLMs evolve into "universal computing engines," researchers are feeding them data from wearables, medical records, and mobile sensors. However, these models were primarily trained on natural language. The authors point out a critical, often overlooked bottleneck: Tokenization.
In natural language, tokenizers (like Byte-Pair Encoding) group frequent character clusters into meaningful units. When applied to time-series or sensor data, this process fails spectacularly. A number like "480" might be a single token, while "481" is split into two. This inconsistency destroys the mathematical and temporal relationships inherent in the data, forcing the model to "memorize" arbitrary patterns rather than "understand" the underlying signal.
Methods
The paper functions as a critical analysis and position paper. It evaluates the three most common subword tokenization methods—Byte-Pair Encoding (BPE), WordPiece, and SentencePiece—through the lens of numerical and temporal data.
The authors propose a hierarchy of solutions to better integrate this data:
- Prompt Engineering (Hard Prompts): Using manual text templates to provide context.
- Prompt Tuning (Soft Prompts): Adding a small, trainable embedding layer to the input that "learns" how to represent the sensor data for the frozen LLM.
- Model Grafting & Adapters: Using specialized encoders (e.g., for ECG or motion data) and mapping their outputs into the LLM’s embedding space.
- Specialized Tokenizers: Moving toward digit-level splitting (as seen in LLaMA) to ensure consistent numerical representation.
Datasets
To demonstrate these pitfalls, the authors conduct a case study using the WISDM (Wireless Indoor Surveillance and Device Monitoring) activity recognition dataset. This dataset contains:
- Participant IDs
- Activity labels (e.g., walking, running)
- UNIX timestamps
- 3-axis accelerometer data (x, y, z)
They processed the first 200 rows of this dataset through the OpenAI tokenizer (used for ChatGPT) to visualize how raw sensor data is fragmented.
Key Insights and Ideas
- The "Modality Gap": There is a fundamental mismatch between how LLMs "see" text and how they "see" numbers. Without addressing this at the input level, the model's output for sensor tasks is essentially "hallucinated" based on fragmented patterns.
- The Negative Sign Trap: In their case study, the authors found that negative signs in sensor readings were often grouped with the preceding comma separator (e.g.,
,-). This effectively strips the directional information from the sensor value, making it invisible to the model. - Context as an Anchor: Previous research cited in the paper shows that adding a simple text string like "Classify the following heart rate sequence" significantly improves performance. Textual context helps "anchor" the numerical data, allowing the LLM to leverage its broader domain knowledge.
- Inconsistent Chunking: The most "aha" moment is the demonstration of digit chunking. Because tokenizers prioritize frequent strings, common years (like "1998") might be one token, while a specific sensor value (like "3.14159") is shattered into four unrelated IDs (
3,.,14,159).
Key Results
The WISDM case study revealed that just 200 rows of sensor data exploded into 6,077 tokens and 10,769 characters. The resulting token IDs showed no logical mathematical progression.
The authors highlight that while Zero-shot inference (asking the model to predict without training) performs poorly on sensor data, Prompt Tuning (training a tiny fraction of parameters) dramatically improves performance. This suggests that the "knowledge" to solve these tasks exists within the LLM, but we need better "translators" (adapters) to help the model access that knowledge using non-textual inputs.
Limitations and Future Work
The authors acknowledge that their scope is limited to temporal/numerical data and that most current "successes" in this field rely on heavy preprocessing (downsampling, aggregating, and rounding) to make the data look more like text.
Future work should focus on:
- Longer Context Windows: Sensor data is high-frequency; LLMs need to handle much longer sequences to capture meaningful behavior.
- Architectural Constraints: Moving beyond just "fixing" tokenization to creating models that natively understand the "physics" of time-series data.
- Generalization: Testing if these numerical-aware approaches can translate to other non-language tasks, such as financial forecasting or climate modeling.
Summary updated