TL;DR
The Multimodal Knowledge Agent (MKA) benchmark is a new, rigorous evaluation for AI agents performing complex, end-to-end professional tasks that require genuine visual reasoning. It reveals that while frontier models like Claude Fable 5.2 and Gemini 4 Carbon are leading the pack, they still struggle with "load-bearing" visual tasks—like interpreting CAD drawings or geospatial maps—leaving significant room for the next generation of multimodal AI.
What Problem Does This Paper Address?
Current AI agents suffer from a "text-and-code bias." When faced with a folder of files, they instinctively reach for OCR or text-extraction scripts rather than "looking" at the data. Existing benchmarks often fail to expose this because they either focus on text-heavy domains or allow models to "cheat" by extracting critical information via code.
The authors argue that real-world professional work—architecture, engineering, medicine, and GIS—is deeply visual. If you remove an agent's ability to see, its performance on a truly multimodal task should collapse. MKA was designed to be a benchmark where vision is load-bearing: essential for both understanding complex inputs and verifying the layout of professional-grade outputs.
Methods
The researchers evaluated models using a standardized, sandboxed environment called the Stirrup harness. Agents are given a prompt and a set of files, then allowed to use tools like code_exec (Python/Bash), view_image (to see renders), and web_search.
To grade the results, they developed an Agentic Autorater. Unlike simple scripts that check for keywords, this is a high-reasoning agent (GPT-6 Luna) that "visually" inspects the deliverables. It can render a PDF to a PNG to check formatting or run code to inspect the formulas inside a submitted Excel sheet. This autorater was validated against human experts, achieving a 96.2% agreement rate.
Datasets
The core of the paper is the MKA Dataset, which consists of 200 professional-grade tasks.
- Breadth: Covers 10 economic sectors (from Healthcare to Construction) and 83 different occupations.
- Inputs: Tasks include a mix of 1,459 files, including CAD blueprints, radiographs, and multi-tab spreadsheets.
- Outputs: Agents must produce standalone deliverables like
.xlsx,.pptx,.pdf, or even interactive.htmldashboards.
The authors also released MKA-Showcase, a public set of 10 tasks (one per sector) to allow the community to test their own agents.
Key Insights and Ideas
- Vision is Load-Bearing: In a "blind" ablation study where the
view_imagetool was removed, Gemini 4 Carbon’s performance on the "Climb" (frontier) tasks dropped significantly. Interestingly, the model tried to compensate with "coding workarounds," such as rendering images as ASCII art or Braille blocks in its terminal to "see" shapes—a creative but brittle and expensive strategy. - The "Climb" vs. "Regression" Split: The benchmark distinguishes between tasks that prior-generation models (like GPT-4 or Claude 3.5) could already solve ("Regression") and those that represent the current frontier ("Climb"). While models hit >95% on regression tasks, they all struggle on the "Climb" set.
- Efficiency Matters: There is a massive disparity in how many tokens models use to reach the same result. Gemini 4 Carbon achieved top-tier quality while using 5.7x fewer tokens and costing significantly less than Claude Fable 5.2.
Key Results
The evaluation compared four frontier models: Gemini 4 Carbon, Claude Fable 5.2, Claude Opus 5.5, and GPT-6 Astra.
- Top Performers: Claude Fable 5.2 (56.6% Climb-WSR) and Gemini 4 Carbon (54.3% Climb-WSR) were the strongest, performing statistically similarly on the hardest tasks.
- The Ceiling: Even the best models stayed below 57% on "Climb" tasks, showing that autonomous multimodal reasoning is still an unsolved problem.
- Visual Necessity: Removing vision tools caused a ~6 percentage point drop in performance and a massive inflation in "turns" and tokens as models struggled to use code to interpret visual data.
Limitations and Future Work
The authors acknowledge several limitations:
- Sample Variance: Each model was only run once per task due to the high cost of these long-running agentic trajectories.
- Sandbox Access: The agents had internet access, which might allow them to retrieve some information externally rather than extracting it visually from the provided files.
- Human Study Scale: The human-agreement study was limited to 50 tasks and focused on high-weight criteria.
Future work will likely focus on closing the 40%+ gap remaining on the "Climb" tasks and improving the efficiency of these multimodal trajectories.
Summary updated