AFVE-Bench: Measuring Abandoned-Frame Verbalization Errors in Multi-Turn LLM Revision

Jiaming ChuarXiv 2026 · arXiv

TL;DR

AFVE-Bench introduces a new benchmark to measure Abandoned-Frame Verbalization Errors (AFVE)—a failure where an LLM correctly follows a revision instruction but "leaks" the superseded logic into the final output (e.g., saying "Unlike the previous version..."). The study finds that nearly 20% of otherwise successful revisions fail to keep the abandoned frame silent, primarily through unnecessary contrastive explanations.

What Problem Does This Paper Address?

When we ask an LLM to revise a document—say, switching a technical guide from "client-side sessions" to "server-side tokens"—we usually want a clean, standalone final product. However, LLMs often suffer from "contextual inertia." They might implement the change correctly but include "editorial residue" like, "Unlike the earlier persistent-login approach, we now use..."

The authors define this as an Abandoned-Frame Verbalization Error (AFVE). A "frame" here isn't just a keyword; it's the governing representation—the organizing logic, evaluative criteria, or perspective of the document. Existing benchmarks focus on whether a model follows the new instruction ($B$), but they don't check if the model successfully silences the old one ($A$). This is critical for professional workflows where such residue reduces usability or carries stale assumptions into downstream processing.

Figure 1: History-conditioned (ℎ= 2) schematic of silent frame replacement, not an ℎ= 0 prompt. The turn-3 request abbreviates the explicit abandonment-plus-𝐵form used for ℎ> 0. Clean (𝐶𝐵= 1∧𝑉𝐴= 0∧𝑈= 1) adopts 𝐵, keeps 𝐴silent, and is directly usable; TC-AFVE (𝐶𝐵= 1∧𝑉𝐴= 1) adopts 𝐵but leaves 𝐴observable, with 𝑈judged independently. The response-level schematic is illustrative and has no empirical denominator.

Methods

The researchers developed AFVE-Bench, a controlled multi-turn dialogue framework.

  1. Dialogue Structure: Each task is a four-turn dialogue. The "history" ($h$) is varied:
  2. $h=0$: The model is asked for the target frame ($B$) directly (baseline).
  3. $h=1$ or $2$: The model first generates content under the source frame ($A$) before being told to switch to $B$.
  4. Evaluation Endpoints: Instead of a single quality score, they use three distinct human-annotated metrics:
  5. Target Adoption ($C_B$): Did the model follow the new instructions?
  6. Source-frame Verbalization ($V_A$): Is the old frame still visible?
  7. Direct Usability ($U$): Is the artifact ready to use without further editing?
  8. Annotation: Two independent annotators scored over 4,200 responses, with a senior adjudicator resolving disagreements.
Figure 2: AFVE-Bench workflow. a, Seeds define 𝐴, 𝐵, and invariant artifact requirements; LLM-assisted drafts are manually naturalized. b, Matched dialogues vary ℎ∈{0, 1, 2}, the number of pre-target user requests under 𝐴; turns 3–4 request artifacts under 𝐵and are scored. c, Eight domains, four frame pairs per domain, two languages, and three ℎconditions yield 192 dialogues; 11 model configurations over two scored turns yield 4,224 post-target responses. d, Two annotators independently score target adoption 𝐶𝐵(𝑦), source-frame verbalization 𝑉𝐴(𝑦), and direct usability 𝑈(𝑦) with model and condition metadata hidden; an independent adjudicator resolves disagreements. Panels a, b, and d are schematic; panel c reports frozen counts.

Datasets

AFVE-Bench consists of:

  • 32 conceptual frame pairs across eight writing domains (e.g., legal revision, clinical summaries, technical docs).
  • Bilingual support: Matched seeds in both English and Chinese.
  • 192 matched dialogues resulting from crossing 64 seeds with three history conditions ($h \in \{0, 1, 2\}$).

Key Insights and Ideas

  • Target Adoption $\neq$ Silence: A model can be 100% compliant with new instructions while still failing to be "silent" about the past. This "Target-Consistent AFVE" (TC-AFVE) is the central failure identified.
  • Contrastive Framing is the Main Culprit: Almost 90% of TC-AFVE errors occur because the model uses the old frame as a "foil" to explain the new one (e.g., "Instead of X, we now do Y").
  • Implicit Residue: In some cases, the model doesn't even name the old frame but retains its organizing logic or criteria, proving that simple "forbidden word" lists are insufficient for evaluation.
  • Usability is a High Bar: Even when models adopt the target and stay silent about the source, only about 24% of those responses were judged "Directly Usable" ($U=1$), often failing due to minor structural or stylistic issues.
Prevalence of different AFVE subtypes; contrastive framing is the most common

Key Results

The study evaluated 11 model configurations (including Claude 3.5 Sonnet, GPT-4o/5, and Gemini 1.5 Flash).

  • AFVE Frequency: AFVE occurred in 25.54% of all history-conditioned responses.
  • The "Leakage" Rate: Among responses that successfully adopted the target frame, 19.77% still exposed the abandoned source frame.
  • Turn Transitions: If a model verbalized the source frame in Turn 3, it was highly likely (~66%) to continue doing so in Turn 4, suggesting that once an error is made, the model struggles to self-correct in the same context.

Limitations and Future Work

  • Observable Behavior Only: The benchmark measures what the model writes, not its internal "hidden" reasoning or private states.
  • Synthetic Nature: While naturalized by humans, the scenarios are controlled and may not fully capture the complexity of real-world professional editing.
  • Causality: The authors note that while they vary history ($h$), the results are descriptive snapshots. They don't claim a strictly calibrated causal scale for "activation" of the source frame.
  • Future Directions: The authors suggest that future work could explore whether "reasoning" models (like the "Thinking" versions of Qwen or GPT) are more or less prone to these errors compared to standard chat models.

Summary updated

Abstract

Multi-turn revision often asks a model to replace an earlier organizing frame while preserving still-valid facts and artifact requirements. Existing instruction-following evaluations mainly assess the latest target and may therefore score an artifact as successful even when it adopts the new frame but exposes the superseded one. We call this failure an abandoned-frame verbalization error (AFVE). AFVE-Bench evaluates silent frame replacement in English and Chinese using 192 matched four-turn dialogues across eight writing domains. Each seed has zero, one, or two pre-target user requests to generate under the source frame. AFVE eligibility depends on the user-request history and applies to the one- and two-request conditions; the zero-request condition serves as a frame-bearing baseline. Eleven model configurations yielded 4,224 expected post-target response units. Two annotators independently scored 4,204 scorable responses for target adoption, source-frame verbalization, and direct usability; model and condition metadata were hidden, and disagreements were adjudicated. AFVE occurs in 25.54% (715/2,799) of history-conditioned responses. Of 2,388 target-passing history-conditioned responses, 472 (19.77%) still expose the abandoned source frame, most often through contrastive framing (423/472). For these standalone revision tasks, target adoption does not ensure source-frame silence.
BibTeX
@misc{chu2026afve,
  author = {Jiaming Chu},
  title = {{AFVE-Bench: Measuring Abandoned-Frame Verbalization Errors in Multi-Turn LLM Revision}},
  year = {2026},
  eprint = {2611.10541},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10541},
}
Figure