TL;DR
The MMDM (Massive Multitask Decision Making) benchmark introduces a rigorous testbed for "System-1" decision models—AI that makes direct, probabilistic choices in a single forward pass without slow, autoregressive reasoning. By evaluating 28 models across 5,448 complex, long-context questions, the paper reveals a significant "capability gap": models that excel at common knowledge often fail miserably when faced with interdependent options, strict logical constraints, and multi-select tasks.
What Problem Does This Paper Address?
As Large Language Models (LLMs) are increasingly integrated into agentic workflows, they are often asked to act as "deciders"—routing intents, selecting tools, or scoring rubrics. Ideally, these decisions should be fast and structured. "System-1" decision models aim to provide this by outputting probabilities over a set of options in one go.
However, existing benchmarks for these models suffer from three main issues:
- Data Contamination: Many rely on widely circulated public datasets that models may have seen during training, leading to inflated scores.
- Lack of Complexity: Real-world decisions involve long contexts (e.g., reading a whole manual before picking a tool) and large option spaces (e.g., choosing from 1,000+ APIs), which current benchmarks often ignore.
- Logical Interdependence: Most benchmarks treat options as independent, whereas real decisions often require understanding how one choice might invalidate another.
Methods
The authors formalize the decision-making task as a mapping from a state (context), a query, and a set of options to a probability distribution. They evaluate models across three distinct answer types:
- Hard Single-select: Choosing the one correct option.
- Hard Multi-select: Identifying the complete set of valid options.
- Soft-label: Predicting a distribution that matches a reference (useful for uncertain outcomes or rubric scoring).
To measure performance, they use standard accuracy for single-select, but introduce the Separation-Balance F-score (SB-F) for multi-select tasks. This metric rewards models for both separating correct options from distractors and balancing the probability mass across all correct choices. They also use Brier error to evaluate the quality of predicted probability distributions.
Datasets
The core of the paper is the MMDM benchmark, consisting of 5,448 high-quality questions across 16 categories.
The dataset is a mix of:
- Source-grounded adaptations: Rewriting existing datasets (like QuALITY or BIRD) to prevent memorization while keeping the core logic.
- Original constructions: New tasks specifically designed to test logic, constraint planning, and agent state tracking.
- Long-context inputs: Reaching up to ~65,000 tokens.
- Variable option spaces: Ranging from simple binary choices to 1,024 options.
Key Insights and Ideas
- The "World Knowledge" Mirage: Models like Perplexity Decider-V1.2 show impressive accuracy on world knowledge (~79%) but drop to ~48% on constraint planning. This suggests that "smart" models might just be well-read rather than good at reasoning through novel constraints.
- Multi-select is the New Frontier: Even the best models struggle to identify all valid options in a set. The highest multi-select accuracy was only 36.81%, and SB-F scores were universally low (below 0.20). Models often find one right answer and stop "looking."
- Calibration vs. Accuracy: A model can be frequently right but dangerously overconfident when it's wrong. The paper shows that high single-select accuracy does not guarantee good probability calibration.
- Scaling Reversals: Performance doesn't scale linearly. Some models that dominate in small option spaces (2-4 options) fall behind significantly as the catalog grows to 100+ options. Similarly, a model's ability to handle long contexts doesn't always predict its ability to handle large option sets.
Key Results
The authors evaluated 28 models, including closed-source heavyweights and open-source contenders:
- Top Performer: WeMM-JEV 27B achieved the highest overall Hard Single-select accuracy (76.00%) and Multi-select accuracy (36.81%).
- Category Leaders: Strengths vary wildly. For instance, KnowLine 31B excels at workflow decisions and policy handling, while Quyet-1.0-Large leads in evidence validation.
- The Logic Gap: Across the board, "Option Logic" and "Constraint Planning" remain the most difficult categories, highlighting a clear area for future model development.
Limitations and Future Work
The authors identify several areas for expansion:
- Modality: MMDM is currently text-only. Real-world decisions often involve visual or auditory states.
- Interactivity: The benchmark uses a fixed state-query-options format. It doesn't test "System-2" thinking (multi-step reasoning) or interactive trajectories where a model can ask clarifying questions.
- Subjectivity: While rubric scoring provides clear labels, broader evaluation of subjective human judgments remains a challenge for future iterations.
By releasing MMDM and its evaluation toolkit, the authors provide a much-needed "stress test" for the next generation of AI decision-makers.
Summary updated