Debating Only the Disputed Edits: Predicting Human Post-Editing Time from Where LLM Editors Disagree

When a translator, doctor, or engineer corrects a machine draft, the cost comes in two parts. One part is typing: inserting, deleting, and moving words. The other is deciding: reading, spotting what is wrong, and judging whether a suspicious phrase is actually a mistake. Standard effort measures such as edit rate only see the typing. Human-targeted Translation Edit Rate (HTER) counts the minimum edits between a draft and its human-corrected version. The deciding part is often larger. Long-distance reorderings and idiomatic errors cost translators a lot of time while needing few keystrokes (Post-editing time as a measure of cognitive effort). We propose an automatic judge that predicts both parts without a human reference. Several LLMs first post-edit the draft independently. Spans where they all agree are settled at once. Only the spans where they split go to a short, evidence-bound debate between an agent that defends the draft and an agent that argues for the edit. The edits that survive estimate typing effort. How hard each span was to settle estimates deciding effort.

This matters for two reasons. First, it gives a practical estimate of human workload, which is the quantity that governs whether a draft is worth anything in production, and it needs no human reference. Second, it uses multi-agent debate in a way the literature suggests should work rather than fail. Debate breaks down when agents argue over a scalar score. It works when they argue over a concrete artifact and must cite evidence. Our design asks agents to argue over individual edits, which is the second case, and it never asks them for a score.

Core Claim

Disagreement among independent LLM post-editors, followed by adversarial adjudication of only the disputed spans, yields two separable signals. The first is a conservative edit set, which predicts physical effort and correctly leaves acceptable drafts untouched. The second is a contestation profile, which predicts the human's detection-and-planning time beyond what edit distance or neural quality metrics capture.

The falsifiable core is the second signal. After controlling for predicted edit rate and COMET, debate-derived contestation should still explain variance in human seconds per word. It should also explain more of that variance than the plain disagreement among independent samples, measured at the same token budget.

Why this design, given prior work

Three findings motivate the gap. LLM-based post-editing as reference-free GEC evaluation uses a single LLM to post-edit grammar-correction outputs and treats the edit distance as a quality score. This reaches system-level correlation above 0.90 with human rankings, but the LLM over-edits sentences humans leave alone, is unstable sentence by sentence, and agrees with human post-edits at only ρ = 0.58 on Swedish. We keep its core idea that the edit is the measurement, but we target sentence-level effort and attack over-editing directly. Effort-Aware Neural Automatic Post-Editing shows that telling a post-editor whether a sentence needs no, light, or heavy editing helps when the label comes from an oracle. The gain shrinks once a classifier predicts the label, and no system improved high-quality English-Russian drafts. This constrains us: deciding to make zero edits is the hard case, so our default for an unresolved dispute is to keep the draft. Evaluating Automatic Subtitling: Correlating Post-editing Effort and Automatic Metrics finds a split. Neural metrics track editing speed, while edit-distance metrics track keystrokes and clicks. That split is the decomposition we want an LLM judge to reproduce from one process.

Two findings constrain the debate design. Judging with Many Minds shows a "debate trap": when a judge and a critic argue sequentially over scores, consistency collapses in round one and does not recover. Parallel meta-judging resists verbosity and position bias better. Voting or Consensus? Decision-Making in Multi-Agent Debate shows accuracy drifts down over extra rounds, and that having all agents draft independently first helps. We respond in three ways. Every first pass is parallel and independent. Debate is never over a score. Debate is capped at very few rounds per span.

One paper supplies the debate mechanism. Adversarial Review: Structured Disagreement for Grounded Agentic Code Review runs a reviewer and a critic over a frozen artifact with typed moves; for example, a disagreement must cite specific lines as evidence. Unconstrained versions fall into false consensus. We reuse its typed moves and its frozen-artifact inner loop. The Adversarial Review paper also reports scope creep, where the adversarial loop invents speculative problems, and it spends about 4.5× the tokens of a single agent. Our defender role and our debate-only-the-split-spans rule are aimed at those two costs.

Finally, A Diagnostic Study of Multi-Agent LLMs for Real-World Debates shows that measures of the debate process predict human preference better than the debate's outcome. Its process measures include whether agents revise beliefs in response to good arguments and whether one agent dominates. That paper never links process to human workload. We do, at the level of individual spans.

Research Program

Stage 1: independent editing. Given a source and a draft, $K$ independent post-editors produce corrected versions under a minimal-edit instruction. The editors are different open models, or one model sampled at temperature. Edits are aligned to draft spans, which are contiguous token ranges, using the alignment that underlies HTER. Each span then falls into one of three groups. It may be unanimously kept, unanimously edited (we take the majority rewrite), or split.

Stage 2: adjudication of split spans only. For each split span, a proponent argues for the most common proposed edit. A defender argues that the draft is acceptable. Both see the source and full context, and the draft is frozen. Moves are typed, following Adversarial Review. An agent may concede. It may object with evidence, which requires quoting the source token or grammatical constraint violated. It may raise a concern without evidence, but such a concern cannot by itself justify an edit. The debate lasts at most three rounds. A span with no evidence-backed resolution keeps the draft wording. The defender's role is what pushes back on over-correction and scope creep.

Signals. The predicted edit rate $E$ is the HTER between the draft and the draft with all accepted edits applied. The contestation score $C$ sums, over split spans, a per-span difficulty. That difficulty combines three things: the initial split ratio, the number of rounds before resolution, and whether resolution required a cited source token (a crude proxy for long-range or meaning-level errors). We predict seconds per word with a small linear model fit on a training split:

$$\widehat{\text{SPW}} = \alpha E + \beta C + \gamma L + \delta,$$

where $\widehat{\text{SPW}}$ is predicted seconds per word, $L$ is sentence length, and $\alpha,\beta,\gamma,\delta$ are fitted coefficients. The linear form is deliberate. It keeps the two signals interpretable, and the hypothesis is simply $\beta > 0$ with a significant gain in held-out fit.

Baselines

The naive reference point is a length-only regression for seconds per word, plus "edit nothing" for zero-edit detection. Any method that does not beat sentence length is not measuring effort.

The strongest existing method on reference-free post-edit judging is the single-LLM post-editor of LLM-based post-editing as reference-free GEC evaluation. It reports system-level Pearson and Spearman correlation above 0.90 on SEEDA and ρ = 0.58 agreement with human post-edits on Swedish. We run it with its best-reported setting, character 6-gram overlap distance, using the same backbone models we use. We beat it if we match its system-level SEEDA correlation and improve sentence-level correlation with human HTER and time on DivEMT and WMT11 English-Spanish. We also give it a self-consistency variant, averaging the distance over $K$ samples, so that it receives the same tokens as our full pipeline.

The strongest metric baselines for time are BLEURT and COMET. Evaluating Automatic Subtitling reports task-level correlations with post-editing speed of 0.676 for BLEURT and 0.641 for COMET, and correlations with total interactions of 0.810 for HTER and 0.804 for SubER. We compute BLEURT and COMET against references, which gives them an advantage our method does not have. We also use COMET as a covariate in the regression. On DivEMT we found no reported result on Lacuna for automatic-metric correlation with per-word editing time, so we compute these baselines ourselves.

The key ablation is Stage 1 alone, using raw split ratio as $C$, at a matched token budget. Matching means raising $K$ until Stage 1 alone spends what the full pipeline spends. This ablation isolates whether adversarial exchange adds anything beyond independent disagreement. A second ablation replaces span-level adjudication with the judge-plus-critic sequential score debate from Judging with Many Minds, asked to output an effort rating. It tests whether the output format, edits rather than scores, is what avoids the debate trap. A third removes the defender, leaving a proponent and a neutral arbiter, to measure its effect on over-editing.

All systems share backbones, prompts tuned on the same development split with the same number of tuning trials, and the same train/test splits for the regression.

Why not just…?

Why not just prompt a single LLM to make minimal edits and estimate the time itself? The evidence says single post-editors over-edit and are unstable at the sentence level (LLM-based post-editing as reference-free GEC evaluation). Predicted effort labels also lose most of their value relative to oracle labels (Effort-Aware Neural Automatic Post-Editing). Still, a strong modern model with a careful no-edit instruction is a real competitor. It is run as a baseline with an explicit "leave acceptable text unchanged" instruction and a direct time estimate. If it matches us on zero-edit detection and time prediction, the multi-agent machinery is unjustified.

Why not just vote among independent post-edits and use their disagreement as the difficulty signal? This is the most serious threat, which is why we built it into the method as Stage 1 rather than arguing against it. Parallel aggregation avoids the debate trap (Judging with Many Minds). Against it, the same paper finds that parallel pools dilute a lone correct judge. Static majorities also rarely recover from a wrong start: high-process debates recovered from an incorrect initial majority 74% of the time versus 40% for low-process ones (A Diagnostic Study of Multi-Agent LLMs for Real-World Debates). The matched-budget Stage-1-only ablation settles this directly. If the debate adds no held-out explanatory power for time and no edit precision, we report that and recommend the cheaper parallel estimator.

Why not just use COMET or BLEURT, which already track editing speed? They need references, which do not exist at deployment time. They also give one number per sentence, which cannot say which span will cost the human time. The subtitling correlations above are task-level, and on DivEMT, speedups range from +119% (Dutch) to +32% (Vietnamese) (DivEMT), a variation a single sentence score must somehow absorb. The partial-correlation test below asks whether contestation predicts time after COMET is already in the model. If it does not, COMET suffices for time and our contribution narrows to reference-free edit prediction.

Proposed Experiments

Primary test: DivEMT. DivEMT provides per-sentence human post-edits, HTER, and editing time across six typologically diverse languages. We measure sentence-level Spearman correlation of $E$ with human HTER and of $\widehat{\text{SPW}}$ with time per word. We also measure span-level precision and recall of accepted edits against the human edits. Zero-edit detection is especially telling here, because about 25% of Dutch and Italian sentences need no edit versus about 3% for Ukrainian and Vietnamese. A method that over-edits will be visibly wrong in the high-quality languages.

Replication on other effort logs. WMT11 English-Spanish, from Post-editing time as a measure of cognitive effort, is the cleanest test of the cognitive claim. There, time correlates with human quality rankings at 0.952 and HTER only at 0.738. The question is whether $C$ recovers part of that gap. On MuST-Cinema subtitling logs, we test whether $E$ tracks keystrokes and clicks while $C$ tracks speed, which would mirror the division between the two metric families.

System-level sanity check. On SEEDA we test whether the pipeline preserves the above-0.90 system-ranking correlation while reducing over-editing on sentences humans left unchanged.

Decisive checks. First, $\beta$ must be positive and must add significant held-out explained variance in seconds per word after $E$, length, and COMET are included. If it does not, the cognitive-effort claim fails. Second, the full pipeline must beat Stage-1-only disagreement at a matched token budget on both time prediction and edit precision. If it does not, debate is unnecessary. Third, removing the defender should increase edits on human zero-edit sentences. If it does not, the minimal-intervention role is not doing the work we claim. We also report tokens per sentence and the fraction of spans that reach debate, so the cost can be compared with the roughly 4.5× overhead of Adversarial Review. Because FLORES-derived text may be in pretraining data, we additionally report results on WMT11, which predates the backbones' training data cutoffs less ambiguously, to check that contamination does not drive the findings.