MMDM: Measuring Massive Multitask Decision Making

Junhao Shen, Qing Zong, Yue Guo, Zhanqiu Zhang, Yiwen Guo, Hong ChengarXiv 2026 · arXiv

TL;DR

The MMDM (Massive Multitask Decision Making) benchmark introduces a rigorous testbed for "System-1" decision models—AI that makes direct, probabilistic choices in a single forward pass without slow, autoregressive reasoning. By evaluating 28 models across 5,448 complex, long-context questions, the paper reveals a significant "capability gap": models that excel at common knowledge often fail miserably when faced with interdependent options, strict logical constraints, and multi-select tasks.

What Problem Does This Paper Address?

As Large Language Models (LLMs) are increasingly integrated into agentic workflows, they are often asked to act as "deciders"—routing intents, selecting tools, or scoring rubrics. Ideally, these decisions should be fast and structured. "System-1" decision models aim to provide this by outputting probabilities over a set of options in one go.

However, existing benchmarks for these models suffer from three main issues:

  1. Data Contamination: Many rely on widely circulated public datasets that models may have seen during training, leading to inflated scores.
  2. Lack of Complexity: Real-world decisions involve long contexts (e.g., reading a whole manual before picking a tool) and large option spaces (e.g., choosing from 1,000+ APIs), which current benchmarks often ignore.
  3. Logical Interdependence: Most benchmarks treat options as independent, whereas real decisions often require understanding how one choice might invalidate another.

Methods

The authors formalize the decision-making task as a mapping from a state (context), a query, and a set of options to a probability distribution. They evaluate models across three distinct answer types:

  • Hard Single-select: Choosing the one correct option.
  • Hard Multi-select: Identifying the complete set of valid options.
  • Soft-label: Predicting a distribution that matches a reference (useful for uncertain outcomes or rubric scoring).

To measure performance, they use standard accuracy for single-select, but introduce the Separation-Balance F-score (SB-F) for multi-select tasks. This metric rewards models for both separating correct options from distractors and balancing the probability mass across all correct choices. They also use Brier error to evaluate the quality of predicted probability distributions.

Datasets

The core of the paper is the MMDM benchmark, consisting of 5,448 high-quality questions across 16 categories.

Figure 2: Task coverage and answer semantics. The colors of the 16 task categories are fixed across figures. Labels show the total count followed by (Hard Single- select / Hard Multi-select / Soft-label) counts.

The dataset is a mix of:

  • Source-grounded adaptations: Rewriting existing datasets (like QuALITY or BIRD) to prevent memorization while keeping the core logic.
  • Original constructions: New tasks specifically designed to test logic, constraint planning, and agent state tracking.
  • Long-context inputs: Reaching up to ~65,000 tokens.
  • Variable option spaces: Ranging from simple binary choices to 1,024 options.

Key Insights and Ideas

  • The "World Knowledge" Mirage: Models like Perplexity Decider-V1.2 show impressive accuracy on world knowledge (~79%) but drop to ~48% on constraint planning. This suggests that "smart" models might just be well-read rather than good at reasoning through novel constraints.
  • Multi-select is the New Frontier: Even the best models struggle to identify all valid options in a set. The highest multi-select accuracy was only 36.81%, and SB-F scores were universally low (below 0.20). Models often find one right answer and stop "looking."
  • Calibration vs. Accuracy: A model can be frequently right but dangerously overconfident when it's wrong. The paper shows that high single-select accuracy does not guarantee good probability calibration.
Figure 3: Confidence calibration on 3,537 shared single-select questions with native, full-state predictions and unique maxima. The left panel groups predictions into ten equal-width confidence bins. The gray diagonal marks perfect calibration. Above it, accuracy exceeds confidence (underconfidence); below it, confidence exceeds accuracy (overconfidence). The right panel reports ECE (↓) on the same questions.
  • Scaling Reversals: Performance doesn't scale linearly. Some models that dominate in small option spaces (2-4 options) fall behind significantly as the catalog grows to 100+ options. Similarly, a model's ability to handle long contexts doesn't always predict its ability to handle large option sets.

Key Results

The authors evaluated 28 models, including closed-source heavyweights and open-source contenders:

  • Top Performer: WeMM-JEV 27B achieved the highest overall Hard Single-select accuracy (76.00%) and Multi-select accuracy (36.81%).
  • Category Leaders: Strengths vary wildly. For instance, KnowLine 31B excels at workflow decisions and policy handling, while Quyet-1.0-Large leads in evidence validation.
  • The Logic Gap: Across the board, "Option Logic" and "Constraint Planning" remain the most difficult categories, highlighting a clear area for future model development.
Figure 1: MMDM capability profiles for five representative decision models. The left panel shows Hard Single-select accuracy across 16 task categories (%, higher is better). The right panel shows soft Brier error over all soft targets, soft Score targets, and other soft tasks (lower is better).

Limitations and Future Work

The authors identify several areas for expansion:

  • Modality: MMDM is currently text-only. Real-world decisions often involve visual or auditory states.
  • Interactivity: The benchmark uses a fixed state-query-options format. It doesn't test "System-2" thinking (multi-step reasoning) or interactive trajectories where a model can ask clarifying questions.
  • Subjectivity: While rubric scoring provides clear labels, broader evaluation of subjective human judgments remains a challenge for future iterations.

By releasing MMDM and its evaluation toolkit, the authors provide a much-needed "stress test" for the next generation of AI decision-makers.

Summary updated

Abstract

System-1 Decision Models directly make structured probabilistic decisions over query-specified options given a state in a single forward pass, without autoregressively generating reasoning trajectories or textual responses. Recent evaluations cover diverse tasks but often rely on existing benchmarks, where prior exposure to test material can obscure generalization. Moreover, real-world decision making often involves long contexts, large option spaces, and complex logical dependencies, challenges that existing benchmarks do not adequately capture together. Thus, we introduce MMDM (Massive Multitask Decision Making), a benchmark of 5,448 high-quality questions across 16 task categories, combining newly constructed tasks with source-grounded recompositions and rewrites for single-select, multi-select, and soft-label decision making in fresh contexts. It covers realistic scenarios such as agent state tracking, tool and skill selection, preference judgments, and rubric scoring, with option spaces ranging from 2 to 1,024 and inputs extending to 64,817 tokens. With MMDM, we conduct a comprehensive quantitative evaluation of 28 decision models, showing that many perform well on common knowledge yet struggle with interdependent options, constrained decisions, and multi-select tasks. For example, Perplexity Decider-V1.2 reaches 78.75% Hard Single-select Accuracy on world knowledge, but only 53.95% on option logic and 47.95% on constraint planning, while the best Hard Multi-select Accuracy across all models is 36.81%. These results highlight joint reasoning, constraint-aware selection, and complete multi-select recovery as key priorities for decision model development, with MMDM providing a testbed for measuring this progress.
BibTeX
@misc{shen2026mmdm,
  author = {Junhao Shen and Qing Zong and Yue Guo and Zhanqiu Zhang and Yiwen Guo and Hong Cheng},
  title = {{MMDM: Measuring Massive Multitask Decision Making}},
  year = {2026},
  eprint = {2611.14309},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14309},
}
Figure