TL;DR
AI agents often make silent, arbitrary assumptions to complete underspecified tasks, leading to results that are technically correct but potentially useless to the user. This paper introduces Agentic Path Maps, a framework for visualizing the "untaken roads" an agent considered, demonstrating that exposing these alternatives significantly reduces user overreliance and encourages more critical engagement with AI outputs.
What Problem Does This Paper Address?
When you ask an AI agent to "find the best price for aquarium tickets," it has to make a dozen micro-decisions: Are you a state resident? Are you going on a weekend? How many kids are with you? If the task is underspecified, the agent fills in the gaps with assumptions.
The authors describe this as the "Garden of Forking Paths." While an agent might return a "task-valid" output (a real price found on a website), it might not be "user-valid" (the price for your specific situation). Currently, we face a binary choice: either the agent executes silently (risking incorrect assumptions) or it interrupts the user constantly for clarification (creating "interruption fatigue").
This paper explores a third way: post-hoc transparency. How can agents communicate the impactful decisions they made—and the alternatives they ignored—after the task is done?
Methods
The researchers approached this in two stages:
- Design Space Distillation: They reviewed over 2,400 papers across domains like multiverse analysis, counterfactual explanations, and decision trees to identify eight key dimensions for communicating alternatives (e.g., Objectives, Structure, Timing, Quantity).
- Empirical User Study: They conducted a study with 48 participants in the domain of data analysis. Using a data agent (DataVoyager), they manipulated the quantity of paths shown to the user:
- Baseline: A single "standard" output (often an extreme value).
- Treatment: An Agentic Path Map showing either 3 or 9 alternative analysis pathways.
To measure impact, they used Weight of Advice (WoA)—a metric quantifying how much a user shifts their prior belief toward the AI's suggestion—and a validated Cognitive Load questionnaire.
Datasets
The study utilized the BLADE benchmark, which provides expert-validated decision spaces for data-driven science. Specifically, they used two datasets:
- Affairs: Investigating if having children affects the rate of extramarital affairs.
- Crofoot: Measuring how the size and location of capuchin monkey fights affect the likelihood of winning.
These datasets were chosen because they allow for "multiverse analysis," where different defensible statistical choices (like which variables to control for) can lead to wildly different results.
Key Insights and Ideas
- The Anchoring Trap: Without seeing alternatives, users almost never consider other analysis choices on their own. They anchor heavily to the first "task-valid" result the AI provides.
- Productive Effort (Germane Load): While showing more paths increases the mental effort required (Intrinsic and Extraneous Cognitive Load), it also increases Germane Cognitive Load. This is "good" mental effort—the kind associated with actual learning and critical reflection.
- The "Three-Path" Sweet Spot: Increasing the number of shown alternatives from three to nine showed diminishing returns. A sample of three paths (e.g., minimum, median, and maximum outcomes) provided a strong middle ground between transparency and information overload.
- Agency through Alternatives: Participants reported feeling more agency when seeing Path Maps. Instead of just "relying on the AI," they felt they could "critically think about why a method was chosen."
Key Results
The study found strong statistical support for the benefits of Path Maps:
- Reduced Overreliance (H1): There was a 99.92% probability that participants placed a lower Weight of Advice on extreme outputs when shown alternatives (Effect size: -0.63, very large).
- Better Sensemaking (H2): Participants' final beliefs ended up significantly closer to the "multiverse median" (the consensus of all possible paths) rather than being pulled toward a single extreme AI output (96.45% probability).
- Increased Engagement (H4): Users spent significantly more time reviewing the interface when shown alternatives (100% probability), indicating they were actually auditing the agent's work rather than rubber-stamping it.
Limitations and Future Work
- Computational Cost: Generating 9 or 100 paths is significantly more expensive than generating one. Future work needs to address how to select the most "impactful" paths to generate.
- Agent Reliability: The study used expert-validated paths. In the real world, an AI agent might generate invalid paths, which could further confuse users.
- Domain Specificity: The results are grounded in data analysis. The authors note that for irreversible tasks (like making a purchase), the cost-benefit analysis of showing alternatives might change.
- P-Hacking Risks: Providing too many alternatives might inadvertently encourage users to "cherry-pick" the result that matches their existing bias (p-hacking).
Summary updated