SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models

Lichen Li, Ziping Dong, Sicheng Feng, Qi Li, Xinchao WangarXiv 2026 · arXiv

TL;DR

SO-Arena is a new, rigorous benchmark designed to evaluate "System One" AI models—those that make fast, single-pass decisions without the overhead of lengthy chain-of-thought reasoning. The benchmark reveals a surprising "reliability gap": a model's raw capability often fails to predict its decision stability, with the top-performing models frequently ranking poorly in reliability.

What Problem Does This Paper Address?

In the dual-process theory of cognition, "System One" refers to fast, intuitive thinking, while "System Two" is slow and deliberative. In AI, most current research focuses on System Two—models that "think" by generating reasoning tokens. However, many real-world tasks like request routing, tool selection, and safety filtering require the speed and structured output of System One models.

Existing benchmarks for these models are often inadequate. They frequently rely on ambiguous labels, fail to distinguish between easy and hard tasks, and ignore decision reliability. A reliable model shouldn't change its mind just because you reorder the multiple-choice options, and it should be able to recognize when the correct answer isn't provided at all. SO-Arena fills this gap by providing a systematic framework to measure both the capability spectrum and the reliability of these fast decision-makers.

Highlights of the SO-Arena benchmark, showing its focus on verified labels, stratified difficulty, interactive evaluation, and reliability testing.

Methods

The authors introduce several novel evaluation strategies:

  • Evidence-Grounded Evaluation: Unlike many benchmarks that use potentially noisy web-scraped labels, SO-Arena uses manually verified reference labels that are strictly supported by the information visible to the model.
  • Fine-Grained Difficulty Tiers: Tasks are stratified into Easy, Medium, and Hard tiers based on domain-specific criteria (e.g., number of calculation steps in math or depth of history in agentic tasks).
  • Interactive Sandboxes: For agentic capabilities, models are tested in realistic, iterative environments where they must execute actions (like "click" or "type") in a sandbox to reach a goal.
  • Reliability Metrics: The paper defines two key components of reliability:
  • Option Order Stability ($S$): Measures if the model consistently picks the same content regardless of its position in a list.
  • Missing-Answer Recognition ($N$): Tests if a model can identify when a "Null" or "None of the above" state is appropriate because the correct option has been removed.
Overview of the SO-Arena pipeline, from data sourcing and quality control to the final capability and reliability scoring.

Datasets

SO-Arena aggregates and adapts 2,919 tasks from 40 distinct public sources across 11 domains (e.g., Math, Code Understanding, Safety, Web Interaction). These are organized into five capability families:

  1. Semantic Understanding
  2. Knowledge and Reasoning
  3. Safety and Policy Compliance
  4. Evidence Assessment
  5. Agent Task Completion

The benchmark includes 2,300 static tasks and 619 environment-based tasks, all unified under a single decision interface.

Key Insights and Ideas

  • Capability $\neq$ Reliability: This is the most striking finding. The best-performing System One model (GLiDE) ranks only 7th in reliability. Conversely, JevAny-27B ranks 11th in capability but 1st in reliability. This suggests that training for raw accuracy doesn't automatically yield robust decision-making.
  • The "Thinking" Dividend is Domain-Specific: For tasks like sentiment analysis or basic knowledge, adding System Two reasoning (thinking) provides negligible gains. However, for "multi-step derivation" tasks like Math, thinking can lift a model's score from 50.8 to 92.5.
  • Scaling Benefits Grounded Decisions: Larger backbones significantly improve performance in "fine-grained grounded decision" domains (like tool use and web interaction), even without explicit reasoning steps.
  • Instability Follows Failure: Models are far more likely to flip their answers when options are reordered on questions they were going to get wrong anyway. Stability is high (95.7%) for correct answers but drops significantly (to 74.3%) for incorrect ones.

Key Results

The authors evaluated 21 models, including 16 open-weight System One models and several closed-source references.

  • Latency: System One models are incredibly fast. Every open-weight model tested answered in under 0.3 seconds per task on a single H100 GPU—tens of times faster than System Two models like GPT-6 Luna.
  • Performance Pareto Frontier: GLiDE (27B) and KnowLine-31B emerged as the strongest open-weight System One models, though they still trail the best System Two models by about 10 points on the hardest tasks.
  • Reliability Leaders: JevAny-27B and the Jev API 1.14 are the most reliable models, showing high stability and a strong ability to recognize missing answers.
Performance vs. Latency for 21 models. System One models (left) offer massive speed advantages, while System Two models (right) still lead in raw capability.

Limitations and Future Work

The authors acknowledge that while SO-Arena is comprehensive, it focuses on models that output structured decisions rather than free-text. Future work could explore how to bridge the gap between System One speed and System Two reasoning more effectively, and how to specifically train for the "missing-answer recognition" that many current models lack. The divergence between capability and reliability rankings highlights a major area for future architectural and alignment research.

Summary updated

Abstract

System One models promise fast, general-purpose decision-making without lengthy token generation. However, understanding their capability spectrum and decision reliability requires more than aggregate performance on current benchmarks. Existing evaluations often provide limited insight into the breadth of model capabilities and the reliability of their decisions. We introduce SO-Arena, a benchmark characterizing the capability spectrum and decision reliability of System One models across 11 domains and five capability families. For the capability spectrum, SO-Arena offers evidence-grounded evaluation through reliable sources and manual verification of reference labels against model-visible inputs; fine-grained capability assessment through domain-specific difficulty criteria; and realistic interactive evaluation, where models iteratively execute actions in sandboxes. Reasoning-intensive tasks further enable system one-two comparison. Beyond accuracy, SO-Arena assesses decision reliability through two complementary measures: decision stability under controlled option-set changes and missing-answer recognition when no valid option is available. Evaluations across 21 models reveal that the benefits of scaling and additional reasoning vary substantially across domains. More importantly, strong overall performance does not necessarily imply reliable decision-making: the best-performing System One model ranks only seventh in reliability. Together, these findings highlight the need to characterize capability spectrum and decision reliability as complementary dimensions, providing a systematic view of the strengths and limitations of System One models.
BibTeX
@misc{li2026so,
  author = {Lichen Li and Ziping Dong and Sicheng Feng and Qi Li and Xinchao Wang},
  title = {{SO-Arena: Benchmarking the Capability Spectrum and Reliability of System One Models}},
  year = {2026},
  eprint = {2611.15777},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.15777},
}
Figure