The Text Uncanny Valley: Non-Monotonic Performance Degradation in LLM Information Retrieval

Zekai Tong, Ruiyao Xu, Aryan Shrivastava, Chenhao Tan, Ari HoltzmanarXiv 2026 · arXiv

TL;DR

When text contains moderate word-boundary corruption (like spurious spaces introduced by OCR), LLM performance unexpectedly plummets but then recovers when the text becomes completely character-separated. This non-monotonic, U-shaped performance curve—termed the Text Uncanny Valley—reveals a hidden failure mode where LLMs get caught in a chaotic transition zone between word-level and character-level processing.

What Problem Does This Paper Address?

Real-world text inputs to Large Language Models (LLMs) are rarely pristine. Text extracted from PDFs via optical character recognition (OCR), scraped from web pages, or passed through automated pipelines frequently contains formatting artifacts, such as spurious spaces inserted within words (e.g., international $\rightarrow$ int er nation al).

While standard benchmarks evaluate LLMs on clean, syntactically perfect text, we know little about how word-boundary corruption impacts performance. Intuitively, one might expect a monotonic decline: more noise equals worse performance. This paper challenges that assumption by systematically studying how continuous word-boundary fragmentation affects an LLM's ability to retrieve and align information.

Methods

The authors design a controlled framework to perturb text by inserting spaces into intra-word slots (the spaces between characters in a word) based on a tunable probability, defined as the word_fragmentation_rate ($p \in [0, 1]$). A rate of $0.0$ represents perfectly clean text, while $1.0$ represents fully character-separated text (every single character separated by a space).

They evaluate models on tasks requiring exact line-level span matching using two primary perturbation strategies:

  1. Random-slot fragmentation: Spaces are inserted stochastically across any available intra-word slot.
  2. Fixed-slot fragmentation: Spaces are predictably inserted only at the first slot of a contiguous character sequence to test if regularity mitigates performance degradation.

Performance is measured using case-insensitive Micro-F1 scores to strictly penalize both missed lines and false positives.

Datasets

The framework is evaluated across three distinct textual domains:

  • Legal Documents (CUAD-QA): Expert-annotated commercial contracts characterized by dense, natural-language prose that is highly sensitive to word-boundary disruption.
  • GitHub Pull Requests: Code diffs containing long, non-spaced identifier runs (e.g., getUserByEmailAndPassword), making the text structurally dense before any perturbation.
  • Mathematical Reasoning (GSM8K): Grade-school math word problems. To challenge long-context retrieval, the authors concatenate 80 problems into single documents.

Key Insights and Ideas

  • The Mode Transition Hypothesis: The authors explain the U-shaped curve by proposing that LLMs process text in two distinct regimes. In the word-level regime (low fragmentation), subword tokenization is stable and the model processes coherent lexical units. In the character-level regime (extreme fragmentation), tokenization collapses entirely to single characters, and the model falls back on character pattern matching to recover semantics. The "Text Uncanny Valley" is the chaotic transition zone where words are inconsistently shattered, leaving the model unable to effectively deploy either strategy.
  • Irregularity, Not Noise Magnitude, Drives the Failure: When spaces are inserted predictably via fixed-slot fragmentation, the U-shape is substantially flattened. This proves that the model's failure is driven by the structural disorder of the text rather than the sheer volume of added whitespace.
  • Tokenization Entropy Misalignment: By analyzing Shannon entropy across token distributions, the authors found that tokenization heterogeneity peaks before the performance minimum. This indicates that the worst performance occurs in the "no-man's-land" where word-level processing has broken down but character-level processing has not yet stabilized.
  • Prompting Can't Save It: Providing few-shot examples via In-Context Learning (ICL) does not rescue performance at the bottom of the valley, indicating that this bottleneck occurs at the fundamental tokenization layer rather than at the level of high-level task comprehension.

Key Results

The "Text Uncanny Valley" was consistently observed across three major model families: Gemini 3.0 Flash, GPT-5.2, and Claude 4.5 Haiku.

The Text Uncanny Valley observed in absence detection across three domains.
  • The Valley Depth: In the absence detection task on legal documents, Claude 4.5 Haiku's Micro-F1 drops from a clean baseline of ~0.70 down to a valley bottom of ~0.30 at a moderate fragmentation rate ($0.6$), before rebounding back up to ~0.65 when the text is fully character-separated ($1.0$).
  • The GitHub PR Anomaly: On GitHub PRs, low levels of fragmentation actually improved baseline performance across all models. Because code diffs contain long, dense variable names, a small number of inserted spaces acts as a pseudo-tokenizer, breaking them into shorter, more parseable units before heavy fragmentation eventually degrades performance.
  • Model Robustness: Across both retrieval tasks, GPT-5.2 consistently produced the flattest curves and achieved the highest absolute Micro-F1, demonstrating superior robustness to word-boundary corruption compared to Gemini and Claude.

Limitations and Future Work

  • Indirect Mechanistic Evidence: The two-modes hypothesis relies on behavioral observations and tokenization statistics. A definitive mechanistic account would require probing internal model states (such as attention patterns or layer-wise activations), which is impossible through black-box APIs. Future work with open-weight models is required.
  • Proprietary Model Bias: The phenomenon was clearly defined in only three of the eight evaluated model families. It remains unclear whether other models avoid the U-shape due to training data composition, architectural choices, or safety filtering.
  • Narrow Perturbation Scope: This study focused exclusively on intra-word space insertion. Real-world OCR noise includes character substitutions, deletions, and transpositions, which may interact with subword tokenizers in entirely different ways.

Summary updated

Abstract

Existing Large Language Model (LLM) benchmarks primarily focus on syntactically correct inputs, leaving a significant gap in evaluation on imperfect text. In this work, we study how word-boundary corruption affects how LLMs detect targeted information. By inserting whitespace characters within words to break them into fragments, LLMs' detection accuracy follows a U-shaped curve with the increase in insertion rate. We refer to this curve as the Text Uncanny Valley. To explain such observation, we propose a mode transition hypothesis: LLMs operate in a word-level mode for near-normal text and a character-level mode for heavily fragmented text, with the valley marking the disordered transition where neither mode is effective. Four experiments and one analysis are consistent with this account: in-context learning fails to rescue valley-bottom performance; regularizing the perturbation substantially reduces the U-shape; a math reasoning task replicates the U-shape for Gemini 3.0 Flash but not for stronger models, suggesting the effect is attenuated when tasks rely less on exact lexical alignment; and tokenization entropy peaks before the F1 minimum, consistent with a regime-conflict interpretation. These findings reveal a failure mode invisible to clean-text benchmarks yet directly relevant to any deployment scenario involving noisy or uncurated text inputs.
BibTeX
@misc{tong2026text,
  author = {Zekai Tong and Ruiyao Xu and Aryan Shrivastava and Chenhao Tan and Ari Holtzman},
  title = {{The Text Uncanny Valley: Non-Monotonic Performance Degradation in LLM Information Retrieval}},
  year = {2026},
  eprint = {2605.07186},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.07186},
}
Figure