When a language model writes out its reasoning step by step (chain-of-thought prompting), a wrong final answer usually traces back to one bad step, often a wrong fact, that later steps faithfully build on. People who check these chains to produce training data tend to flag the step where the contradiction becomes visible, not the step where it started. We propose that a checker's eye movements carry the missing information. When a reader reaches a step that does not add up, their eyes jump backward to the earlier step they suspect. If those backward jumps (regressions) land reliably on the erroneous step, an eye tracker can point to the root error even when the person's explicit rating misses it, and even when the step looks fluent to an automatic checker.
The claim is narrow and testable. The pattern of cross-step regressions, meaning which earlier step a reader's gaze returns to while reading a later one, localizes the first faulty step in a model-generated reasoning chain. It does this better than the reader's own ratings, better than per-step reading time from a cheap self-paced interface, and better than a retrieval-backed language-model verifier. It also catches a meaningful share of errors that the human rater marked as correct. If any of the cheaper signals matches it, the study says so. That outcome would also tell annotation teams something useful: they would not need the eye tracker.
Core Claim and Positioning
Three findings motivate the proposal. OASST-ETC recorded 24 evaluators reading LLM responses and found that rejected responses impose a reading cost, with longer total reading time ($t(325) = -3.76$) and longer first fixations ($t(325) = -3.37$). That study works at the level of whole responses and never looks inside intermediate reasoning. Measuring Sentiment Annotation Complexity of Text showed that annotators can agree on a label while their fixations reveal large hidden effort, so explicit agreement hides cognitive conflict. Scanpath Complexity showed that the structure of eye movements, measured as how unlikely the saccade pattern is under a model of an ideal reader, separates hard from easy text more sharply than total fixation time ($p = 1.4\times10^{-5}$ versus $2.0\times10^{-4}$). Taken together, these suggest that structure carries information that duration does not. This is exactly what is needed to locate a problem rather than merely detect one.
The method we build on is CoTEVer. It breaks GPT-3 explanations on StrategyQA into sub-question and answer steps, retrieves web evidence for each step, and has humans rate and fix each step on a 1–5 scale. It reports that 43% of faulty explanations stem from a wrong fact. CoTEVer relies only on what annotators consciously report. We keep its decomposition and evidence retrieval, and we add the annotator's gaze as an implicit signal on each step.
Two papers act as constraints rather than precedents. RT2H documents 2–3 cm gaze error against 3–5 mm line spacing. This rules out word-level analysis on ordinary screens, so we lay out each step as a separate, widely spaced block and analyze gaze at step granularity. Adding eye-tracking AOI data does not improve prediction accuracy warns that mapping gaze onto regions does not automatically help prediction. For this reason we treat "gaze adds nothing over timing" as a live outcome, not a strawman.
The proposal is clearly separate from the other gaze-plus-reasoning work in the file. GazeLLM uses chain-of-thought to predict where people in images are looking, so gaze is the output there. MedEyes trains simulated "gaze" trajectories over medical images and records no human eyes. In our setting, real human gaze is the input, and the task is curating reasoning data.
Research Program
Stimuli. We use model-generated chains of 4–8 steps from two sources. The first is the 300 incorrect StrategyQA explanations annotated through CoTEVer, mixed with an equal number of correct chains. The second is GSM8K solutions, where errors are easy to verify. For GSM8K we create controlled variants of correct chains. The first variant injects a wrong fact or wrong number into one step, so that later steps inherit it and the inconsistency only surfaces downstream. The second variant inserts a verbose but correct filler step that is long and hedging. These filler steps are modeled on the "Wait, alternatively" loops that Excessive Reasoning Attack shows can inflate chain length roughly ninefold without changing the answer. The filler condition exists to break duration-based signals: it adds reading time without adding an error. We also control step length throughout, given the length paradox in the sentiment-complexity study.
Recording. About 30 annotators each verify around 80 chains on a 1000 Hz desktop tracker, using recalibration between blocks as in OneStop. They work in a CoTEVer-style interface that gives a rating per step, offers optional evidence, and asks them to name the first wrong step. Steps occupy separate regions spaced several centimeters apart.
Gaze features. From each trial we build a step-to-step regression matrix, whose entry $(k, j)$ counts saccades launched while reading step $k$ that land on an earlier step $j$. We also compute per-step first-pass time, re-reading time, and step-local scanpath improbability as defined by the Scanpath Complexity paper. Our main gaze-only score is the "incoming regression mass" of each step, normalized by the step's length and position. This score needs no learning. A second variant is a small logistic model over all gaze features, with annotators held out during training.
Downstream use. If localization works, we test whether it pays off in annotation. With a fixed number of human repair minutes, chains triaged by gaze-flagged root steps should produce more fully corrected chains. Distilling those chains into a small model should then give higher StrategyQA accuracy than triage by explicit ratings. This follows the pattern in Is GPT-3 a Good Data Annotator?, where labels are judged by the downstream model they produce.
Baselines
The naive reference is the explicit rating. The step the annotator names as first wrong, or failing that their lowest-rated step, is the CoTEVer workflow itself. Any gaze method must beat it at localization, and it must also recover errors among chains the annotator rated as fully correct. No reported result found on Lacuna for step-level error localization accuracy of CoTEVer ratings. The file gives only task accuracy (Self-Ask 70.52% versus standard CoT 65.4% on StrategyQA) and the error-type breakdown, so we measure this baseline ourselves on the same trials.
The strongest gaze-free human signal is per-step reading time from a self-paced interface, where annotators click to reveal each next step while earlier steps stay visible. This is the cheap version of our idea. It needs no hardware and captures most of what fixation duration captures. To keep the comparison fair, it runs on the same annotators and chains in a counterbalanced block. Gaze beats it only if gaze localizes better overall, with a margin preregistered at 5 percentage points of top-1 localization accuracy, and specifically wins in the filler-step condition, where timing should point at the long innocent step.
The strongest automatic competitor is a retrieval-backed LLM step verifier. It uses CoTEVer's evidence retrieval and has a current LLM judge each step against the evidence, reasoning step by step before giving a per-step error probability. It gets the same chains and a prompt-tuning budget of several hundred labeled chains held out from evaluation. No reported result found on Lacuna for this verifier's step-localization accuracy. Gaze beats it if gaze alone wins, or if combining gaze with the verifier significantly beats the verifier alone. The combination matters most on "fluent wrong fact" steps, where the hate-speech annotation-bias study and the sentiment-complexity results lead us to expect confident machine errors.
The key ablation strips the cross-step regression matrix and keeps only per-step durations and fixation counts from the same recordings. If this ablation matches the full gaze model, our mechanism, which is backward checking, is wrong even if gaze "works." We also run a text-only predictor of reader friction trained in the style of the sentiment-complexity paper, which reached Pearson 0.57 in its own task. This checks whether gaze reflects step difficulty rather than step error.
Why not just…?
Why not just time each step with a click-to-reveal interface? This is the most serious threat. Under ordinary chains it may perform close to gaze. The Scanpath Complexity result suggests that duration cannot tell slow careful reading from confusion. But that result concerns reading effort, not error location, so it does not settle the question. The filler-step manipulation is built to separate the two signals: timing should be drawn to the long correct step, while regressions should still land on the injected wrong fact. If timing matches gaze on filler chains too, we will report that cheap timing suffices and recommend it.
Why not just let an LLM verifier with retrieval check every step? CoTEVer's own breakdown shows that most errors are wrong or outdated facts, which retrieval can catch. A verifier may therefore dominate on StrategyQA. The proposal does not claim gaze replaces it. It claims gaze adds information on the residual errors the verifier rates as confident and correct. The combined-versus-verifier comparison, restricted to verifier false negatives, decides this. If the gain is not significant, gaze is not worth recording for this purpose.
Why not just trust the annotator's explicit "first wrong step" judgment? The sentiment-complexity study found that explicit agreement hides measurable conflict, but it never tested whether that conflict marks errors. Our analysis of chains the annotator rated as fully correct tests this directly. If gaze-flagged steps in those chains are no more likely to be wrong than other steps, the implicit-signal premise fails.
Decisive Experiments
The primary measure is top-1 and top-2 root-step localization accuracy on erroneous chains. We report it separately for StrategyQA natural errors, GSM8K injected errors, and GSM8K injected-error-plus-filler chains, using annotator-grouped bootstrap intervals. Secondary measures are step-level error-detection AUROC; recall of errors missed by the annotator's explicit judgment at a fixed flag budget of one step per chain; and downstream StrategyQA accuracy of a small model distilled from chains repaired under equal human-minute budgets.
The decisive pattern has four parts. Incoming regression mass should localize root errors above explicit ratings. It should beat self-paced timing on filler chains. It should add significant information over the retrieval verifier on that verifier's misses. And the regression-free ablation should lose most of the gain. If localization works only with the regression matrix, backward checking is the mechanism. If timing ties gaze everywhere, the practical result is that annotation tools should log per-step reveal times, and eye trackers are unnecessary for this task.
References
- OASST-ETC Dataset: Alignment Signals from Eye-tracking Analysis of LLM Responses
- Measuring Sentiment Annotation Complexity of Text
- Scanpath Complexity: Modeling Reading Effort Using Gaze Information
- CoTEVer: Chain of Thought Prompting Annotation Toolkit for Explanation Verification
- See Where You Read with Eye Gaze Tracking and Large Language Model
- Adding eye-tracking AOI data to models of representation skills does not improve prediction accuracy
- GazeLLM: a plug-and-play zero-shot LLM reasoning framework for boosting gaze target detection
- MedEyes: Learning Dynamic Visual Focus for Medical Progressive Diagnosis
- Excessive Reasoning Attack on Reasoning LLMs
- OneStop: A 360-Participant English Eye Tracking Dataset with Different Reading Regimes
- Is GPT-3 a Good Data Annotator?
- Large Language Model Annotation Bias in Hate Speech Detection