When a language-model agent fails at a multi-step task, we usually cannot say which step caused the failure. Current benchmarks settle that question by asking humans or another model to read the agent's log and point to the culprit. Both judges are unreliable on long, tool-heavy logs. On AgentHallu, the best model finds the responsible step only 41.1% of the time, and only 11.6% of the time when the error is a tool-use error (AgentHallu). We propose a different source of ground truth. Physically re-run the failed plan with one step repaired, and see whether the outcome flips.
Ultrafast laser ablation is unusually well suited to this. Femtosecond pulses remove material before heat can spread into the surrounding solid, so-called "cold ablation." As a result, a test site a few tens of micrometers away is physically unaffected by its neighbor. One silicon wafer can therefore host hundreds of independent, nearly identical trials of an agent's plan and its repaired variants. Each trial is a short, irreversible physical experiment with a measurable outcome. The proposal is to build a benchmark of language-model agents that plan femtosecond laser patterning jobs. Every failure in it is labeled by counterfactual re-execution. We then use those labels to test whether existing taxonomies of agent hallucination, and existing model judges, get the blame right when the world, not a transcript, is the referee.
Core Claim
Step-level blame labels obtained by counterfactual physical re-execution will disagree substantially with labels from model judges and with labels from human readers of the trajectory. The disagreement will be concentrated in errors where the agent's plan is consistent with the user's specification but contradicts laser–matter physics. Examples include treating a picosecond pulse on a metal as "cold," or ignoring the threshold drop from repeated pulses on the same spot (incubation). Ignoring plasma-plume shielding at high repetition rate is another.
If this holds, two things follow. Transcript-based attribution systematically mislabels a whole family of hallucinations, the ones that a survey of multimodal hallucination calls factuality errors as opposed to faithfulness errors (A Survey of Multimodal Hallucination Evaluation and Detection). And a step-level supervisor trained on counterfactual labels should correct agents better than one trained on judge labels.
The precise label is defined as follows. A failed trajectory has actions $a_1,\dots,a_T$ and outcome $y(a_{1:T})=0$. Let $\tilde a_t$ be an expert's minimal correction of step $t$, with later steps re-planned by the same agent from the repaired state. The responsible step is
$$ t^\star = \min \{\, t : \Pr[y(a_{1:t-1}, \tilde a_t, \hat a_{t+1:T}) = 1] \ge \tau \,\}, $$
where $\hat a_{t+1:T}$ are the agent's re-planned later actions and $\tau$ is a success-probability threshold estimated from replicate sites. This is the earliest step whose repair makes the job succeed. Because a repair at step $t$ usually stays effective when the repair point moves later, the success curve is close to monotone. Binary search over $t$ therefore needs about $\log_2 T$ fabrication runs per failure rather than $T$.
Why ultrafast ablation, and what prior work supplies
The laser literature supplies both the physics that makes the testbed possible and the failure modes that make it interesting.
On the physics side, femtosecond patterning of WS₂ monolayers leaves Raman and photoluminescence spectra unchanged, while 350 ns pulses cause wrinkling and debris (Atomic ribbon formation in the pulsed laser patterning of WS₂ layers). This is the locality we rely on for independent neighboring sites.
On the failure side, work on picosecond machining of diamond micro-drills shows that ultrashort pulses still convert diamond to graphite-like carbon, which must be planned around (PCD micro drill). That is a textbook "physically unfaithful" plan an agent could make. A review of periodic surface ripples (LIPSS) documents ripple splitting, debris redeposition and pulse-train shaping. It also notes that the formation mechanism is still debated, so no analytical simulator can serve as a complete oracle (Uniformity Control of LIPSS). That gives us both outcome metrics (ripple orientation divergence, reported below 5° for etching-assisted methods) and a reason why physical execution, not simulation, must provide ground truth.
Prior machine learning in laser ablation is discriminative monitoring. Examples are classifying tissue from acoustic waves during Er:YAG ablation (Deep-Learning Approach for Tissue Classification) and monitoring laser osteotomy (Machine learning monitoring for laser osteotomy). Neither involves planning agents or blame assignment.
Metasurface inverse design is the closest machine learning work. Its knowledge-inherited networks beat a monolithic network, 86.7% versus 20.0% (knowledge-inherited learning). That work, however, scores designs by aggregate accuracy and never fabricates inside the loop. We borrow its lesson that modular checks help, but our object is the multi-step physical plan, not the unit-cell geometry.
The hallucination side supplies the method and the motivation. AgentHallu defines step localization with five categories (planning, retrieval, reasoning, human interaction, tool use) and finds that longer trajectories hurt badly, with accuracy falling from 29.9% to 11.4%. Its labels, though, come from human annotation of transcripts, which is exactly what we propose to audit. The SWE-PRM study shows that taxonomy-based, descriptive mid-trajectory feedback raises SWE-bench Verified resolution from 40.0% to 50.6% and beats rigid action directives (When Agents go Astray). This is the intervention we will retrain on physical labels. VCBench finds that about 62% of Gemini 2.0 Flash's errors are visual-perception errors and that multi-image inputs are fragile (VCBench). This warns that our agents' microscope-reading steps will be a major error source, so the taxonomy must keep perception separate.
MedCalc-Bench's split of knowledge retrieval, information extraction and arithmetic (MedCalc-Bench) maps directly onto the physics side of our tasks. The corresponding laser errors are recalling a material's ablation threshold, extracting the specification, and computing fluence from power, repetition rate and spot size.
Research Program
Agent task suite. We first build a laser-processing agent harness on a femtosecond setup with a motion stage and an in-line optical microscope. The tasks are restricted to two well-characterized material systems: crystalline silicon and silicon-on-insulator. Each task gives the agent a written and drawn specification, such as "a 200 µm × 200 µm LIPSS grating with orientation divergence below 10° and no visible redeposition," or "a 20 µm wide, 2 µm deep groove with no cracking." The agent has tools for material-property lookup, fluence calculation, scan-path generation, pulse-train configuration, stage execution on a test site, and image capture with measurement. A typical trajectory runs 5 to 15 steps, including at least one inspect-and-adjust loop.
Taxonomy. The taxonomy keeps AgentHallu's five top-level categories so results are comparable. It crosses them with the faithfulness/factuality axis, so that, for instance, a tool-use error is either "violates the specification" or "violates laser–matter physics." It adds a perception category for misreading microscope images, motivated by VCBench.
Counterfactual labeling. Every failed trajectory receives counterfactual labels. A laser expert writes the minimal correction for a candidate step without seeing which step is suspected to be guilty. The repaired prefix is executed and re-planned on fresh sites, with three replicate sites per run because small fluctuations in pulse energy or focus can flip outcomes. Binary search then finds $t^\star$.
Independently, two groups annotate the same trajectory from its transcript and images alone. Human experts annotate it AgentHallu-style, and frontier models annotate it with both whole-trajectory and step-by-step prompting.
Supervisor training. We train two step-level supervisors in the SWE-PRM style. One is trained on counterfactual labels and one on judge labels. Both are deployed every few steps on held-out tasks.
Baselines
Outcome-only reference. This is the naive point: the agent runs without any supervisor, and its failures receive no step label. It anchors the success rate. For blame, it is equivalent to a "last step" or "uniform" guess, which we report as chance-level localization. It is run on the same tasks and substrates with the same agent model and step budget.
Model-judge attribution. This is the strongest existing method for blame, run exactly as specified by AgentHallu with step-by-step verification prompting. The reported reference is 41.1% step localization for Gemini-2.5-Pro on AgentHallu, and 11.6% on tool-use hallucinations. Step-by-step prompting adds an average of 12.3%. We give the judges the full transcript plus all captured images and the same token budget across models. Beating this baseline does not mean our labels "score higher," since they are the reference. It means the judges' agreement with counterfactual labels is significantly lower than their agreement with human transcript labels, and that the gap is concentrated in physics-factuality errors.
Human transcript annotation. Laser experts annotate from the log alone. This is the strongest human competitor and the source of most existing benchmark labels. They get the same materials as the model judges and a fixed time per trajectory.
Taxonomy-guided supervisor. For intervention, the strongest baseline is SWE-PRM-style guidance with labels from model judges (When Agents go Astray), which reported 40.0% → 50.6% on SWE-bench Verified. No reported result was found on Lacuna for laser-processing agents. We match supervisor model size, training examples, intervention frequency and tuning budget. Beating it means a higher task success rate on held-out specifications at equal laser time.
Physics rule checker. This is a deterministic checker written by an expert. It flags fluence below threshold or far above it, pulse duration versus electron–phonon coupling time for the material, pulse overlap, and repetition rate above a plume-shielding heuristic. It serves both as a supervisor and as an attributor ("blame the first flagged step"). No reported result was found on Lacuna for such a checker.
Ablation of the key ingredient. We replace counterfactual labels with labels from a two-temperature-model simulation of the repaired plan. This isolates whether physical execution matters or whether a simulator would give the same blame.
Why not just…?
Why not just use a physics rule checker and skip attribution altogether? This is the most serious threat. Many failures will be gross parameter errors that a fluence-threshold check catches, and if the checker localizes nearly all counterfactual blame, the elaborate labeling is unnecessary. We do not argue around this; the rule-checker baseline measures it directly. Our expectation that it falls short rests on two points. The LIPSS review states that ripple formation has no settled mechanism (Uniformity Control of LIPSS), and incubation, redeposition and perception errors are not threshold violations. If the checker matches counterfactual blame on more than roughly 80% of failures, we would report that as the main finding and re-scope toward the residual errors.
Why not just trust the model judges, since AgentHallu already validated its human labels? AgentHallu validates judges against human readers of transcripts. Whether those readers are right about causation in a physical process has never been checked. The 11.4% accuracy on long trajectories suggests that context accumulation already confuses readers. The judge-versus-counterfactual agreement experiment answers this directly, and a high agreement would be a clean negative result worth publishing.
Why not just optimize laser parameters with Bayesian optimization and drop agents? For a single fixed recipe, Bayesian optimization may well reach better outcomes with less laser time. We include it as an outcome reference. Our question is different: language-model agents are being deployed to plan open-ended physical jobs from natural-language specifications, and we need to know whether our tools for diagnosing them are trustworthy. Bayesian optimization cannot read a specification or explain a failure.
Why not just simulate the counterfactuals instead of fabricating? That is the ablation above. If simulated blame agrees with physical blame, the method becomes cheap and transfers to other domains, which would be a positive outcome. We expect disagreement precisely on the debated-mechanism cases.
Proposed Experiments and Decisive Checks
Site independence. The prerequisite is to show that cold ablation actually gives independent sites. We run one fixed control recipe at 100 sites with varying spacing and measure outcome variance against spacing. We pick the smallest spacing at which outcomes are indistinguishable from isolated sites, and we report replicate flip rates, which set $\tau$.
Dataset and attribution. We collect roughly 300 agent trajectories from three model families, about half failed. We label the failures counterfactually, then measure step-localization accuracy of each judge and of human annotators against $t^\star$, broken down by taxonomy cell and trajectory length. The decisive attribution result is the agreement gap between judges and counterfactual labels on factuality versus faithfulness errors. We also report how often the counterfactual relabels the taxonomy category itself, not just the step.
Intervention. We compare the counterfactual-trained supervisor against the judge-trained supervisor, the rule checker and no supervisor on held-out specifications, including a second material (SOI) unseen in supervisor training. Metrics are task success, LIPSS orientation divergence, groove depth error and laser time used.
What would falsify the claim. The central claim fails if judge–counterfactual agreement is as high as judge–human agreement, or if the counterfactual-trained supervisor does no better than the judge-trained one. Either outcome would tell the field that transcript-based blame is adequate even for irreversible physical processes, which is itself a useful finding.
References
- AgentHallu: Benchmarking Automated Hallucination Attribution of LLM-based Agents
- A Survey of Multimodal Hallucination Evaluation and Detection
- Atomic ribbon formation in the pulsed laser patterning of WS 2 layers
- The fabrication of polycrystalline diamond micro drill based on ultrashort pulsed laser
- Uniformity Control of Laser-Induced Periodic Surface Structures
- Deep-Learning Approach for Tissue Classification using Acoustic Waves during Ablation with an Er:YAG Laser (Updated)
- Machine learning monitoring for laser osteotomy
- A knowledge-inherited learning for intelligent metasurface design and assembly
- When Agents go Astray: Course-Correcting SWE Agents with PRMs
- Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency
- MedCalc-Bench: Evaluating Large Language Models for Medical Calculations