Iterative Refinement via Multi Pass Decoding

26 concepts20 papers1 proposal

When humans write, edit code, or translate text, we rarely produce a flawless final product on our first attempt. Instead, we draft, review, spot errors, and refine. Yet, traditional sequence-to-sequence models are often forced into a "one-shot" paradigm. Autoregressive models generate text token-by-token from left to right, leaving no room to revise past mistakes, while non-autoregressive models generate everything in a single parallel burst, often sacrificing structural coherence. As Elgohary (2020) points out, treating minor near-misses as total failures is a major bottleneck in sequence generation tasks.

To bridge this gap, a powerful paradigm has emerged: multi-pass iterative refinement. Instead of treating sequence generation as a single forward pass, these architectures unroll the decoding process into multiple sequential stages. By feeding a model's own output back into itself, the system can progressively correct errors, resolve complex token interdependencies, and dramatically improve final quality.

Clearing the Fog: Breaking Contextual Blind Spots

Why can't a model fix all errors at once? In tasks like Grammatical Error Correction (GEC) or spelling check, errors frequently occur in clusters or close proximity. When a sentence contains multiple overlapping mistakes, one typo can obscure the semantic context required to identify and fix an adjacent error.

Multi-pass decoding solves this via an "easy-first" strategy. As demonstrated by Ge et al. (2018), an initial pass can resolve the most obvious, high-confidence errors. By fixing these prominent mistakes early on, the model effectively reduces noise and uncovers a clearer semantic context. In subsequent rounds, the subtle or complex errors that were previously obscured become easy to identify and correct.

This mechanism is particularly vital when dealing with consecutive errors. Li et al. (2022) utilize a sliding window to refine text over multiple passes, showing that correcting a single character in an early pass provides the necessary context to resolve adjacent errors later. Similarly, the Denoising Decoding Correction ($D^2C$) approach proposed by Jiang et al. (2024) iteratively masks low-confidence tokens and replaces them with high-confidence contextual predictions until the sequence reaches a stable state.

Bridging the Non-Autoregressive Independence Gap

Non-autoregressive (NAR) generation breaks the sequential bottleneck of standard Transformers, enabling massive inference speedups by generating all tokens in parallel. However, because NAR models assume tokens are conditionally independent given the source text, they struggle to capture internal token-to-token dependencies, often leading to repetitive words or grammatical hallucinations.

As surveyed by Xiao et al. (2022), iterative refinement acts as the ultimate bridge between the speed of fully parallel generation and the accuracy of sequential models. By generating a coarse initial sequence in parallel and updating it over multiple passes, the decoder can dynamically capture internal dependencies.

This "coarse-to-fine" approach is highly effective across modalities. For instance, Yang et al. (2021) first predict key visual words to form a rough template for video captions, then use multi-pass refinement to fix low-confidence predictions and eliminate hallucinations. In text editing, Zhang et al. (2023) feed the output of an initial parallel editing step back into the system, allowing the model to resolve complex structural dependencies that a single parallel pass would miss.

Stagewise Edits vs. Full Rewrites

When faced with highly corrupted text, attempting to rewrite the entire sequence in one go often causes the model to over-correct, lose track of the original meaning, or introduce entirely new errors. Iterative refinement mitigates this by shifting the objective from global rewriting to incremental editing.

Lichtarge et al. (2019) show that making incremental, high-confidence changes over multiple passes is significantly more stable than single-pass correction. To make this process explicit, Sorokin (2022) proposed a framework that ranks elementary edits, applying only the single highest-confidence change in each pass before re-evaluating the modified sentence. This sequential evaluation ensures that early fixes correctly inform subsequent editing decisions.

Even in sequence labeling tasks like Chinese word segmentation, a two-phase architecture can be highly beneficial. King et al. (2005) demonstrate that using the predictions of an initial tagging phase as contextual features for a secondary correctional phase allows the model to recognize and eliminate linguistically impossible patterns that a single-pass model might generate. To ensure the model doesn't drift too far from the original text during these multiple passes, Wang et al. (2024) introduce a "source fusion" method that continuously injects the original input context into the intermediate decoding passes.

Progressive Context for Retrieval and Reranking

Multi-pass decoding also fundamentally changes how models interact with external knowledge bases and complex scoring functions. In retrieval-augmented generation, querying a database with a noisy or error-ridden input sequence yields poor results.

By incorporating iterative refinement, the model can clean up the input text in an initial pass, creating a significantly higher-quality query for subsequent database lookups. Song et al. (2023) demonstrate this multi-stage process in domain-adaptive spelling check, where initial corrections prevent the model from accidentally replacing rare technical terms with common words during retrieval. Similarly, the Bi-$k$NN framework by You et al. (2024) uses the output of each iteration to provide progressively improved context for subsequent database queries, allowing non-autoregressive models to adapt to specialized domains without retraining.

Beyond retrieval, this progressive refinement helps guide search space exploration. Marie et al. (2015) use a high-complexity reranker to provide feedback to a simpler, faster decoder across multiple passes, identifying discrepancies to iteratively prune and refine the translation search space.

Integrating Vision and Semantics

The power of unrolling the decoding process extends to multimodal sequence tasks like scene text recognition. When reading text in the wild, humans combine visual signals with language intuition. Bhunia et al. (2021) mimic this by unrolling the decoding process into multiple stages: the model makes an initial visual guess and then iteratively refines it using joint visual and semantic context, correcting early character errors by evaluating the global structure of the word.

In a similar vein, the ABINet++ architecture by Fang et al. (2022) passes initial text predictions through a bidirectional language model multiple times. This progressive refinement allows the model to resolve discrepancies between visual features and linguistic context, effectively correcting both character-level typos and sequence length misalignments.

Key Foundational Work

Multi-Pass Decoding With Complex Feature Guidance for Statistical Machine Translation (2015) — contributed 1 concept elements to this direction.
Two-Phase LMR-RC Tagging for Chinese Word Segmentation (2005) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_iterative_sequence_refinement_90535257,
  title = {iterative sequence refinement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/iterative-sequence-refinement/rdr_9053525719814660ad2e4d0328f87f58},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_iterative_sequence_refinement_90535257_v1,
  title = {iterative sequence refinement},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/iterative-sequence-refinement/rdr_9053525719814660ad2e4d0328f87f58/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake