Introduction
As Large Language Models (LLMs) evolve, their ability to process vast amounts of information—the "long-context" capability—has become a primary differentiator. However, current evaluation frameworks like the "Needle-In-A-Haystack" (NIAH) test are predominantly English-centric. This creates a significant gap in our understanding of how these models perform across the global linguistic landscape. To address this, researchers have introduced ONERULER, a comprehensive benchmark designed to evaluate multilingual long-context performance across 26 languages, ranging from high-resource languages like English and Chinese to low-resource ones like Sesotho and Swahili.
The ONERULER Benchmark
ONERULER adapts the English-only RULER benchmark into a multilingual framework. It utilizes seven synthetic tasks to stress-test model retrieval and aggregation capabilities, focusing on identifying specific facts ("needles") within massive backgrounds ("haystacks").
A critical innovation in ONERULER is the introduction of the "None-NIAH" task. Unlike standard benchmarks that assume an answer is always present, ONERULER modifies instructions to include: "If no such numbers exist, please answer 'none'." This addition tests whether models are truly processing the text or simply hallucinating plausible answers based on the prompt's structure. To ensure linguistic accuracy, the researchers employed native speakers for 25 languages to localize instructions, ensuring prompts were natural and grammatically sound across diverse linguistic structures.
Experimental Results
The study yielded several surprising insights that challenge common assumptions about LLM performance.
English is Not the "One Ruler"
Contrary to expectations, English was not the top-performing language, ranking 6th overall. Polish emerged as the leader, with Slavic, Romance, and Germanic languages generally showing superior performance. Surprisingly, Chinese ranked as the 4th worst language on the benchmark, even for models like Qwen 2.5 72B that were specifically tuned for it.
The "None" Trap and Context Scaling
The introduction of the "none" option revealed a significant vulnerability in modern LLMs. As context length increases, models become increasingly prone to "over-caution," frequently predicting "none" even when the information is present. For instance, in English, adding the "none" option caused S-NIAH accuracy to drop by 32% at a 128K token context.
Furthermore, while models perform consistently across languages at shorter context lengths (8K tokens), a massive performance gap opens as the context scales to 128K. High-resource languages maintain relative stability, whereas low-resource languages suffer a sharp decline in retrieval accuracy.
Reasoning Models and Tokenizer Bias
The researchers also evaluated reasoning-focused models like OpenAI’s o3-mini-high. They found that these models often "overthink" simple retrieval tasks, generating extensive reasoning traces that occasionally lead them to incorrect conclusions.
A significant technical hurdle identified is tokenizer bias. Because different models tokenize languages with varying efficiency, a 128K token limit does not represent a uniform amount of information across languages. For example, 128K tokens might contain significantly more semantic information in English than in Tamil. The authors suggest that future benchmarks should move toward "bits-per-byte" normalization to ensure a truly equitable multilingual evaluation.
Conclusion
ONERULER highlights the critical need for diverse, multilingual long-context evaluation. The findings suggest that model performance is highly sensitive to both the language of the context and the specific phrasing of instructions. As LLMs continue to expand their context windows, benchmarks like ONERULER will be essential for ensuring that these capabilities are robust and accessible across all the world's languages, not just a select few.
Summary updated