Limited Linguistic Diversity in Embodied AI Datasets

Selma Wanna, Agnes Luhtaru, Jonathan Salfity, Ryan Barron, Juston Moore, Cynthia Matuszek, Mitch PryorarXiv 2026 · arXiv

TL;DR

While Vision-Language-Action (VLA) robotics models are rapidly advancing, the datasets used to train them are linguistically starved. This paper presents a systematic audit of popular VLA corpora, revealing extreme command repetition, shockingly small vocabularies, and a near-total absence of complex logical structures like negations or conditionals.

What Problem Does This Paper Address?

Modern generalist robots rely on VLA models to translate natural language instructions into physical actions. However, despite being backed by powerful Large Language Models (LLMs), these robots often remain remarkably brittle—failing when faced with simple paraphrases or distractor objects.

The authors argue that the root of this problem lies in the training data. While current robotics dataset documentation meticulously tracks diversity in physical environments, objects, and robot embodiments, the linguistic diversity of the instructions has been largely ignored. This paper addresses this critical gap by auditing the language signal present in widely used VLA datasets to understand what these models are actually learning.

Methods

The authors developed a comprehensive evaluation framework to analyze dataset language across three core dimensions:

  1. Duplication and Lexical Diversity: Quantifying the number of unique commands, sentence lengths, vocabulary size, and text compression ratios.
  2. Semantic Diversity: Using sentence embeddings (via encoders like USE, SBERT, CLIP, and SONAR) and Principal Component Analysis (PCA) to determine the "intrinsic dimensionality" of the instructions. They also mapped the co-occurrence of verbs, direct objects, and adverbs.
  3. Structural Diversity: Analyzing grammatical syntax by extracting Part-of-Speech (POS) patterns and evaluating higher-level logical structures (negations, conditionals, multi-step sequencing, and loops) using an LLM-guided dependency parsing pipeline (DeepSeek-R1-Distill-Qwen-32B).
Figure 2: We analyze linguistic diversity in embodied AI datasets across three categories: Analysis 1 (A.1) Duplication & Lexical diversity, Analysis 2 (A.2) Semantic diversity, and Analysis 3 (A.3) Structural diversity.

Datasets

The study analyzed a diverse set of datasets categorized into three groups:

  • VLA Datasets: RT-1, BRIDGE, TacoPlay, Language Table, and LIBERO.
  • Language-Focused Robotics Datasets: ALFRED and SCOUT (used as robotics baselines with higher language emphasis).
  • Instruction-Tuning Datasets: OASST2, Alpaca, and LLaVA-Instruct (used as non-robotics baselines to anchor text complexity).
Overview of the audited datasets, including their focus and language styles.

Key Insights and Ideas

  • The 2% Uniqueness Trap: In almost all VLA datasets, fewer than 2% of the total language instructions are uniquely worded. The massive size of these datasets comes from pairing the exact same text command with thousands of different action trajectories across multiple trials.
  • Shockingly Tiny Vocabularies: Some of the most popular datasets rely on a minuscule set of words. For example, the massive RT-1 dataset contains a total vocabulary of only 49 unique words.
  • Superficial Semantic Correlations: Because commands are highly templated, robots are exposed to narrow verb-object pairings. For instance, in RT-1, a "banana" is frequently something to be "picked," but it is never "moved" or "closed," creating artificial blind spots in the robot's understanding of object affordances.
  • A Total Lack of Logic: Real-world human instructions are full of constraints ("don't touch the hot pan") and contingencies ("if the cup is full, stop pouring"). However, VLA datasets are almost exclusively composed of flat, linear, step-by-step commands. Negations and conditionals appear in fewer than 2% of instructions across the board.

Key Results

The quantitative audit highlighted a stark contrast between VLA datasets and standard language datasets:

  • Lexical Redundancy: RT-1 features over 3.7 million sentences but only 577 unique variations (0.02% uniqueness). Even BRIDGE, which showed the highest lexical variety among VLA datasets, only achieved 1.4% uniqueness.
  • Semantic Skew: The heatmap analysis of verb-object co-occurrences revealed heavy clustering. In RT-1, specific verbs are rigidly tied to specific shapes (e.g., the verb "knock" is almost exclusively paired with can-shaped objects).
Figure 3: Analysis 2: Semantic Diversity Verb and direct object co-occurrence frequencies in the manually annotated RT-1 dataset. The heatmap highlights lim- ited verb diversity across plausible actions; for instance, banana is frequently “picked” but never “moved.” The rare verb knock appears mostly with can-shaped objects, despite being equally applicable to others like an upright sponge.
  • Syntactic Monotony: Structural diversity is heavily constrained by templates. In TacoPlay, a single POS pattern (VERB → DET → ADJ → NOUN → ADP → DET → NOUN) accounts for a staggering 24% of all unique sentences.
Figure 4: Analysis 3: Structural Diversity Most fre- quent POS patterns per dataset on unique sentences and their relative frequency and example.
  • Missing Logical Structures: As shown below, while multi-step sequencing is common, negations, conditionals, and cyclic loops are virtually non-existent in VLA training data compared to language-focused datasets like SCOUT.
Figure 5: Analysis 3: Structural Diversity Percent- age of instructions exhibiting four structural phenom- ena: negation, conditionality, multi-step sequencing, and cyclic repetition. For datasets containing fewer than 600 unique sentences, annotations were performed manually. For those with more than 600 unique sen- tences, annotations were generated using an automated pipeline. Standard error bars reflect labeling uncertainty estimated from a manually reviewed subset of 500 ran- domly sampled commands per dataset.

Limitations and Future Work

A primary limitation acknowledged by the authors is that this study focuses exclusively on the textual component of these datasets, ignoring cross-modal alignment (how well the language syncs with visual inputs and action trajectories). Additionally, expanding object diversity in robotics is inherently bottlenecked by the physical cost of acquiring real-world props.

To fix the linguistic deficit, the authors suggest three paths forward:

  1. Targeted Augmentation: Using LLMs to generate paraphrases and introduce syntactic variations (like phrase inversion or synonym replacement) into existing datasets.
  2. Cross-Domain Transfer: Mixing linguistically rich text corpora (like procedural manuals or human dialogues) into VLA training pipelines.
  3. Interactive Data Collection: Designing user interfaces that actively prompt human annotators in real-time to use synonyms, adverbs, and varied phrasing during data collection.

Summary updated

Abstract

Language plays a critical role in Vision-Language-Action (VLA) models, yet the linguistic characteristics of the datasets used to train and evaluate these systems remain poorly documented. In this work, we present a systematic dataset audit of several widely used VLA corpora, aiming to characterize what kinds of instructions these datasets actually contain and how much linguistic variety they provide. We quantify instruction language along complementary dimensions--including lexical variety, duplication and overlap, semantic similarity, and syntactic complexity. Our analysis shows that many datasets rely on highly repetitive, template-like commands with limited structural variation, yielding a narrow distribution of instruction forms. We position these findings as descriptive documentation of the language signal available in current VLA training and evaluation data, intended to support more detailed dataset reporting, more principled dataset selection, and targeted curation or augmentation strategies that broaden language coverage.
BibTeX
@misc{wanna2026limited,
  author = {Selma Wanna and Agnes Luhtaru and Jonathan Salfity and Ryan Barron and Juston Moore and Cynthia Matuszek and Mitch Pryor},
  title = {{Limited Linguistic Diversity in Embodied AI Datasets}},
  year = {2026},
  eprint = {2601.03136},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2601.03136},
}
Figure