TL;DR
SO-Arena is a new, rigorous benchmark designed to evaluate "System One" AI models—those that make fast, single-pass decisions without the overhead of lengthy chain-of-thought reasoning. The benchmark reveals a surprising "reliability gap": a model's raw capability often fails to predict its decision stability, with the top-performing models frequently ranking poorly in reliability.
What Problem Does This Paper Address?
In the dual-process theory of cognition, "System One" refers to fast, intuitive thinking, while "System Two" is slow and deliberative. In AI, most current research focuses on System Two—models that "think" by generating reasoning tokens. However, many real-world tasks like request routing, tool selection, and safety filtering require the speed and structured output of System One models.
Existing benchmarks for these models are often inadequate. They frequently rely on ambiguous labels, fail to distinguish between easy and hard tasks, and ignore decision reliability. A reliable model shouldn't change its mind just because you reorder the multiple-choice options, and it should be able to recognize when the correct answer isn't provided at all. SO-Arena fills this gap by providing a systematic framework to measure both the capability spectrum and the reliability of these fast decision-makers.
Methods
The authors introduce several novel evaluation strategies:
- Evidence-Grounded Evaluation: Unlike many benchmarks that use potentially noisy web-scraped labels, SO-Arena uses manually verified reference labels that are strictly supported by the information visible to the model.
- Fine-Grained Difficulty Tiers: Tasks are stratified into Easy, Medium, and Hard tiers based on domain-specific criteria (e.g., number of calculation steps in math or depth of history in agentic tasks).
- Interactive Sandboxes: For agentic capabilities, models are tested in realistic, iterative environments where they must execute actions (like "click" or "type") in a sandbox to reach a goal.
- Reliability Metrics: The paper defines two key components of reliability:
- Option Order Stability ($S$): Measures if the model consistently picks the same content regardless of its position in a list.
- Missing-Answer Recognition ($N$): Tests if a model can identify when a "Null" or "None of the above" state is appropriate because the correct option has been removed.
Datasets
SO-Arena aggregates and adapts 2,919 tasks from 40 distinct public sources across 11 domains (e.g., Math, Code Understanding, Safety, Web Interaction). These are organized into five capability families:
- Semantic Understanding
- Knowledge and Reasoning
- Safety and Policy Compliance
- Evidence Assessment
- Agent Task Completion
The benchmark includes 2,300 static tasks and 619 environment-based tasks, all unified under a single decision interface.
Key Insights and Ideas
- Capability $\neq$ Reliability: This is the most striking finding. The best-performing System One model (GLiDE) ranks only 7th in reliability. Conversely, JevAny-27B ranks 11th in capability but 1st in reliability. This suggests that training for raw accuracy doesn't automatically yield robust decision-making.
- The "Thinking" Dividend is Domain-Specific: For tasks like sentiment analysis or basic knowledge, adding System Two reasoning (thinking) provides negligible gains. However, for "multi-step derivation" tasks like Math, thinking can lift a model's score from 50.8 to 92.5.
- Scaling Benefits Grounded Decisions: Larger backbones significantly improve performance in "fine-grained grounded decision" domains (like tool use and web interaction), even without explicit reasoning steps.
- Instability Follows Failure: Models are far more likely to flip their answers when options are reordered on questions they were going to get wrong anyway. Stability is high (95.7%) for correct answers but drops significantly (to 74.3%) for incorrect ones.
Key Results
The authors evaluated 21 models, including 16 open-weight System One models and several closed-source references.
- Latency: System One models are incredibly fast. Every open-weight model tested answered in under 0.3 seconds per task on a single H100 GPU—tens of times faster than System Two models like GPT-6 Luna.
- Performance Pareto Frontier: GLiDE (27B) and KnowLine-31B emerged as the strongest open-weight System One models, though they still trail the best System Two models by about 10 points on the hardest tasks.
- Reliability Leaders: JevAny-27B and the Jev API 1.14 are the most reliable models, showing high stability and a strong ability to recognize missing answers.
Limitations and Future Work
The authors acknowledge that while SO-Arena is comprehensive, it focuses on models that output structured decisions rather than free-text. Future work could explore how to bridge the gap between System One speed and System Two reasoning more effectively, and how to specifically train for the "missing-answer recognition" that many current models lack. The divergence between capability and reliability rankings highlights a major area for future architectural and alignment research.
Summary updated