By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Prompting

Hyungjun Yoon, Biniyam Aschalew Tolera, Taesik Gong, Kimin Lee, Sung-Ju Lee

Introduction

"By My Eyes" introduces a visual prompting framework that transforms raw, numeric sensor data into images for analysis by Multimodal Large Language Models (MLLMs). By "showing" rather than "telling" the data to the model, the researchers achieved a 10% average accuracy boost while reducing token costs by a staggering 15.8x.

The Problem: Numeric Data as a Bottleneck

While Large Language Models (LLMs) are incredibly versatile, they have a "kryptonite": long sequences of numbers. Ubiquitous sensing—data from smartwatches, heart rate monitors, or IoT devices—generates massive streams of numeric data. Feeding these raw numbers into an LLM as text creates two major hurdles:

  1. Performance Degradation: LLM accuracy drops sharply as numeric sequences grow longer. They struggle to "see" the patterns in a wall of text.
  2. Prohibitive Costs: Every number in a sequence consumes tokens. A single minute of accelerometer data can translate into 90,000 tokens, costing hundreds of dollars per hour to process via high-end APIs.
LLM performance (Error Rate and MAE) degrades significantly as the length of numeric sequences increases.

Methodology: Visual Prompting

The authors propose Visual Prompting, where sensor data is converted into a visual plot (like a waveform or spectrogram) before being sent to an MLLM like GPT-4o. This approach bypasses the limitations of text-based numeric processing by leveraging the model's inherent visual pattern recognition capabilities.

Comparison between text-based prompting (high token cost, incorrect result) and the proposed visual prompting method (low token cost, correct result).

The core of their approach is a two-stage Visualization Generator:

  1. Visualization Tool Filtering: The MLLM reviews a library of plotting tools (e.g., Matplotlib, Scipy) and the task description to suggest a shortlist of effective visualization methods.
  2. Visualization Selection: The system generates actual plots using the shortlisted methods. The MLLM then "looks" at these samples and selects the one that most clearly distinguishes between different classes (e.g., the difference between "walking" and "running" waveforms).

The final Visual Prompt consists of a few-shot setup: labeled example images of different activities followed by the "target" image the model needs to classify, all wrapped in specific textual instructions.

Experimental Setup and Datasets

The researchers evaluated their method across nine sensory tasks and four modalities:

  • Accelerometer: HHAR (daily activities), UTD-MHAD (complex actions), and Swim (swimming styles).
  • ECG (Heart): PTB-XL dataset for arrhythmia detection (CD, MI, HYP, STTC).
  • EMG (Muscle): Gesture recognition.
  • Respiration: WESAD dataset for stress detection.

Key Insights

  • Pattern Recognition > Arithmetic: LLMs are better at identifying visual patterns in a plot than performing the implicit "mental math" required to understand a list of coordinates.
  • The "Single Image" Advantage: An MLLM treats an image as a fixed number of tokens regardless of how much data is plotted. This allows you to condense thousands of data points into a single "visual token," bypassing the context window limits of text.
  • Aggregated vs. Subplots: Interestingly, the authors found that plotting multiple channels (like X, Y, and Z axes of an accelerometer) in a single overlapping plot with different colors worked better than using separate subplots. This suggests MLLMs are better at correlating features when they are visually overlaid.
  • Automated Domain Expertise: The Visualization Generator allows the system to act as its own data scientist, choosing the best way to represent data (e.g., picking a spectrogram for frequency-heavy tasks) without human intervention.

Key Results

The "By My Eyes" approach consistently outperformed traditional text-based prompting:

  • Accuracy: Achieved an average of 10% higher accuracy across all tasks. In the UTD-MHAD complex activity task, the accuracy gain was as high as 33%.
  • Cost Efficiency: Reduced token consumption by an average of 15.8x. For some datasets like WESAD, the reduction was nearly 50x.
  • Robustness: Visual prompts remained effective even when the number of "shots" (examples) was increased, whereas text prompts often hit token limits or confused the model.

Limitations and Future Work

While images are great for patterns, they discard exact numerical values. Tasks requiring precise retrieval (e.g., "What was the exact voltage at 5.02 seconds?") still favor text. Additionally, visualizing data with hundreds of channels (like high-density EEG) in a single plot remains a challenge, as the resulting image becomes too cluttered for the model to interpret. Finally, the benefits were most pronounced in top-tier models like GPT-4o; smaller, open-source MLLMs struggled to interpret the complex plots, suggesting a need for better vision-language alignment in smaller models.

Summary updated

BibTeX
@inproceedings{yoon2024by,
  author = {Hyungjun Yoon and Biniyam Aschalew Tolera and Taesik Gong and Kimin Lee and Sung-Ju Lee},
  title = {{By My Eyes: Grounding Multimodal Large Language Models with Sensor Data via Visual Prompting}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.133},
}
Figure