Route First, See Later: Enabling Vision Agents to Reason Scientifically by Separating Routing Policy from Action Spaces

Shubhabrata Mukherjee, Jeffrey Donatelli, Daniela Ushizima

TL;DR

Researchers have introduced ZETA, a new architecture for scientific vision agents that adopts a "Route First, See Later" approach. By explicitly separating the decision of which tool to use (routing) from the execution of that tool, ZETA significantly improves the reliability and auditability of AI agents in complex scientific workflows, boosting segmentation routing accuracy for some models from a meager 6.2% to 75.0%.

What Problem Does This Paper Address?

In scientific image analysis, a single image can prompt many different questions: "Where is the cell?", "How many are there?", or "What is the exact area of the boundary?" Current AI agents often conflate intent interpretation (understanding what the scientist wants) with visual execution (performing the task).

This creates a "black box" problem. If an agent provides an incorrect measurement, it’s often impossible to tell if the failure was due to flawed reasoning (picking the wrong tool) or failed perception (the tool itself performed poorly). This lack of auditability is a major roadblock for deploying AI in high-stakes scientific automation where provenance and precision are non-negotiable.

Figure 1: Effect of constrained action space v routing policy specification

Methods

The authors propose the ZETA architecture, which operationalizes a strict decoupling of the action space (the set of available tools) from the routing policy (the rules for selecting those tools).

Key features include:

  • Text-First Routing: The agent resolves the scientific intent using a text-only router before any visual tools are invoked. This ensures the agent "thinks" about the requirement before it "looks" at the data.
  • Deterministic Measurement: Unlike generative models that might "hallucinate" a number, ZETA treats measurement as a deterministic operation. It calculates values (like area or length) directly from explicit geometric artifacts, such as pixel masks generated during segmentation.
  • Auditable Execution Traces: ZETA produces independent logs for the routing decision, the tool invoked, and the resulting artifacts, allowing researchers to pinpoint exactly where a failure occurred.
Figure 3: ZETA architecture with text-first routing, specialized visual execution, and optional deterministic measurement.

Datasets

The study utilized a zero-shot benchmark consisting of 51 distinct scientific-intent queries. These queries were designed to test the agent's ability to distinguish between semantically adjacent tasks, such as object localization (detection) versus boundary recovery (segmentation). The evaluation was performed across five different language models: Qwen 3.8, Nemotron 3.5, Qwen 3.6, Qwen3 1.7B, and muse-glimmer 30b.

Key Insights and Ideas

  • The Specification Gap: Simply giving an agent a "menu" of tools (the action space) is insufficient. Without explicit semantic rules (the routing policy), agents often struggle to choose the correct tool for the job, even if the tool is available.
  • Semantic Adjacency is a Trap: Agents frequently confuse tasks that seem similar but require different computational pathways. For example, a request for "size" requires segmentation for accuracy, but agents often default to simpler object detection (bounding boxes), which cannot measure true irregular shapes.
  • Auditability as a First-Class Citizen: By separating routing from execution, ZETA allows for "failure attribution." We can now distinguish between an agent that is "smart but blind" (correct route, failed execution) and one that is "seeing but confused" (wrong route, successful execution of the wrong task).

Key Results

The intervention of adding explicit routing semantics yielded dramatic improvements, though the results varied by model:

  • Qwen 3.8: Segmentation routing accuracy skyrocketed from 6.2% to 75.0%, while detection routing reached a perfect 100% (up from 60%).
  • Nemotron 3.5: Saw a significant jump in segmentation accuracy from 37.5% to 87.5%.
  • Model Sensitivity: Interestingly, the smallest model tested (Qwen3 1.7B) actually saw performance decline with explicit guidance, suggesting that the ability to follow complex routing policies is tied to the underlying model's reasoning capacity.

Limitations and Future Work

The authors acknowledge that this is a preliminary evaluation. The benchmark is relatively small (51 queries), and the study focuses primarily on routing correctness rather than the downstream accuracy of the visual tools themselves.

Future work will likely involve scaling the benchmark to more diverse scientific domains and investigating how to make routing policies even more robust for smaller, more efficient models. The ultimate goal is to move toward fully autonomous, yet completely transparent, scientific discovery pipelines.

Summary updated

Abstract

Scientific image-analysis workflows require agents to select among diverse capabilities like description, detection, counting, and segmentation. Current multi-tool architectures conflate intent interpretation with visual execution, masking whether a failure arises from flawed reasoning (misrouting) or failed perception (invalid tool execution). This lack of auditability renders agents unreliable for high-stakes scientific automation. We introduce ZETA, an extensible architecture that operationalizes the decoupling of routing policy from action execution. ZETA resolves scientific intent before invoking specialized visual tools, treating measurement as a deterministic operation grounded in explicit geometry rather than generative output. This architecture exposes the "black box" of agent decision-making, allowing routing correctness and downstream execution to be evaluated as independent failure modes. To our knowledge, we provide the first controlled study in scientific vision-agent routing that isolates explicit routing-policy specification while holding the legal action interface fixed. Evaluation on a zero-shot benchmark of 51 scientific-intent queries shows that explicit routing semantics can substantially improve routing accuracy, particularly at semantically adjacent action boundaries, although improvements vary across models. For Qwen 3.8, explicit routing-policy specification improves segmentation routing accuracy from 6.2% to 75.0% and detection routing accuracy from 60.0% to 100%. ZETA's execution traces make routing decisions and downstream artifacts separately observable, providing a basis for distinguishing intent-selection errors from visual execution failures. ZETA establishes the necessary infrastructure to isolate execution errors from interpretive ones, a critical requirement for deploying reliable scientific vision agents.
BibTeX
@misc{mukherjee2026route,
  author = {Shubhabrata Mukherjee and Jeffrey Donatelli and Daniela Ushizima},
  title = {{Route First, See Later: Enabling Vision Agents to Reason Scientifically by Separating Routing Policy from Action Spaces}},
  year = {2026},
  eprint = {2611.06760},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06760},
}
Figure