MKA: Evaluating Multimodal Knowledge Agents on end-to-end professional tasks

Anima Singh, Suhas Yogin, Weicheng Kuo, Ivor Rendulic, Ivan Jurin, Antoine Yang, Oran Kelly, Rohan Doshi, Hammad Ayyubi, Ioana Bica, Sergi Caelles, Matthew Conniff, Dan Graur, Anastasija Ilić, Ahmet Iscen, Chao Jia, Tomasz Kępa, Chunyuan Li, Noah Ó Donnaile, Filip Pavetic, Goran Petrovic, Khalid Salama, Zhicheng Wang, Polina Zablotskaia, Yiqian Zhang, Ming Zhao, Benjamin Caine, Mario Lučić, Melvin Johnson, Radu Soricut, Jean-Baptiste AlayracarXiv 2026 · arXiv

TL;DR

The Multimodal Knowledge Agent (MKA) benchmark is a new, rigorous evaluation for AI agents performing complex, end-to-end professional tasks that require genuine visual reasoning. It reveals that while frontier models like Claude Fable 5.2 and Gemini 4 Carbon are leading the pack, they still struggle with "load-bearing" visual tasks—like interpreting CAD drawings or geospatial maps—leaving significant room for the next generation of multimodal AI.

What Problem Does This Paper Address?

Current AI agents suffer from a "text-and-code bias." When faced with a folder of files, they instinctively reach for OCR or text-extraction scripts rather than "looking" at the data. Existing benchmarks often fail to expose this because they either focus on text-heavy domains or allow models to "cheat" by extracting critical information via code.

The authors argue that real-world professional work—architecture, engineering, medicine, and GIS—is deeply visual. If you remove an agent's ability to see, its performance on a truly multimodal task should collapse. MKA was designed to be a benchmark where vision is load-bearing: essential for both understanding complex inputs and verifying the layout of professional-grade outputs.

Methods

The researchers evaluated models using a standardized, sandboxed environment called the Stirrup harness. Agents are given a prompt and a set of files, then allowed to use tools like code_exec (Python/Bash), view_image (to see renders), and web_search.

To grade the results, they developed an Agentic Autorater. Unlike simple scripts that check for keywords, this is a high-reasoning agent (GPT-6 Luna) that "visually" inspects the deliverables. It can render a PDF to a PNG to check formatting or run code to inspect the formulas inside a submitted Excel sheet. This autorater was validated against human experts, achieving a 96.2% agreement rate.

Datasets

The core of the paper is the MKA Dataset, which consists of 200 professional-grade tasks.

  • Breadth: Covers 10 economic sectors (from Healthcare to Construction) and 83 different occupations.
  • Inputs: Tasks include a mix of 1,459 files, including CAD blueprints, radiographs, and multi-tab spreadsheets.
  • Outputs: Agents must produce standalone deliverables like .xlsx, .pptx, .pdf, or even interactive .html dashboards.
Distribution of input files by category

The authors also released MKA-Showcase, a public set of 10 tasks (one per sector) to allow the community to test their own agents.

Key Insights and Ideas

  • Vision is Load-Bearing: In a "blind" ablation study where the view_image tool was removed, Gemini 4 Carbon’s performance on the "Climb" (frontier) tasks dropped significantly. Interestingly, the model tried to compensate with "coding workarounds," such as rendering images as ASCII art or Braille blocks in its terminal to "see" shapes—a creative but brittle and expensive strategy.
  • The "Climb" vs. "Regression" Split: The benchmark distinguishes between tasks that prior-generation models (like GPT-4 or Claude 3.5) could already solve ("Regression") and those that represent the current frontier ("Climb"). While models hit >95% on regression tasks, they all struggle on the "Climb" set.
  • Efficiency Matters: There is a massive disparity in how many tokens models use to reach the same result. Gemini 4 Carbon achieved top-tier quality while using 5.7x fewer tokens and costing significantly less than Claude Fable 5.2.

Key Results

The evaluation compared four frontier models: Gemini 4 Carbon, Claude Fable 5.2, Claude Opus 5.5, and GPT-6 Astra.

  • Top Performers: Claude Fable 5.2 (56.6% Climb-WSR) and Gemini 4 Carbon (54.3% Climb-WSR) were the strongest, performing statistically similarly on the hardest tasks.
  • The Ceiling: Even the best models stayed below 57% on "Climb" tasks, showing that autonomous multimodal reasoning is still an unsolved problem.
  • Visual Necessity: Removing vision tools caused a ~6 percentage point drop in performance and a massive inflation in "turns" and tokens as models struggled to use code to interpret visual data.
Deliverable quality across different rubric splits

Limitations and Future Work

The authors acknowledge several limitations:

  • Sample Variance: Each model was only run once per task due to the high cost of these long-running agentic trajectories.
  • Sandbox Access: The agents had internet access, which might allow them to retrieve some information externally rather than extracting it visually from the provided files.
  • Human Study Scale: The human-agreement study was limited to 50 tasks and focused on high-weight criteria.

Future work will likely focus on closing the 40%+ gap remaining on the "Climb" tasks and improving the efficiency of these multimodal trajectories.

Summary updated

Abstract

We introduce the Multimodal Knowledge Agent (MKA) benchmark, which measures how well autonomous agents perform end-to-end professional knowledge work that depends on visual understanding. Each task requires an agent to extract and combine information from multiple visually dense files, such as CAD blueprints, electronics schematics, radiographs, complex charts and geospatial maps, and to produce a standalone deliverable in .xlsx, .pptx, .pdf, .docx, .html or .png format. Existing benchmarks leave this capability largely untested: removing image-viewing tools leaves Gemini 4 Carbon's score on GDPval unchanged, and single-format evaluations such as GDP$.$pdf and GDP$.$xlsx do not require reasoning across files or modalities. MKA contains 200 tasks across 10 economic sectors. Deliverables are graded against expert-weighted rubrics by a sandboxed agentic autorater that agrees with human domain experts on 96.2% of criterion judgments in our expert-graded study (Cohen's $κ= 0.91$), scored by weighted satisfaction rate (WSR) overall and on frontier criteria unsaturated by prior-generation models (Climb-WSR). Across the frontier models we evaluate (Gemini 4 Carbon, Claude Fable 5.2, Claude Opus 5.5, and GPT-6 Astra), Claude Fable 5.2 (56.6% Climb-WSR, 73.3% Overall WSR) and Gemini 4 Carbon (54.3% Climb-WSR, 71.9% Overall WSR) achieve the highest scores, while Carbon uses $5.7\times$ fewer tokens. Even for the strongest models, Climb-WSR remains below 57%, leaving substantial headroom for multimodal agents.
BibTeX
@misc{singh2026mka,
  author = {Anima Singh and Suhas Yogin and Weicheng Kuo and Ivor Rendulic and Ivan Jurin and Antoine Yang and Oran Kelly and Rohan Doshi and Hammad Ayyubi and Ioana Bica and Sergi Caelles and Matthew Conniff and Dan Graur and Anastasija Ilić and Ahmet Iscen and Chao Jia and Tomasz Kępa and Chunyuan Li and Noah Ó Donnaile and Filip Pavetic and Goran Petrovic and Khalid Salama and Zhicheng Wang and Polina Zablotskaia and Yiqian Zhang and Ming Zhao and Benjamin Caine and Mario Lučić and Melvin Johnson and Radu Soricut and Jean-Baptiste Alayrac},
  title = {{MKA: Evaluating Multimodal Knowledge Agents on end-to-end professional tasks}},
  year = {2026},
  eprint = {2611.25228},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25228},
}
Figure