The Road to Quality is Paved with Good Revisions: A Detailed Evaluation Methodology for Revision Policies in Incremental Sequence Labelling

Brielen Madureira, Patrick Kahardipraja, David Schlangen

Introduction

Incremental sequence labelling is a critical component of real-time natural language processing systems, such as live captioning, simultaneous translation, and interactive dialogue agents. In these settings, a model must produce output labels as input arrives, word by word. However, language is inherently ambiguous; a word like "bank" may refer to a financial institution or a riverbank until further context is provided. Consequently, incremental models often need to "change their minds," revising previous predictions as more information becomes available.

While various "revision policies" have been developed to manage these updates, the field has lacked a standardized framework to evaluate their quality. This paper addresses this gap by introducing a formal evaluation methodology and a suite of metrics designed to measure the efficiency, effectiveness, and stability of revision policies.

The Problem: The "Naive" Tax and Output Flicker

The standard approach to incremental processing is restart-incremental, where the model re-processes the entire input sequence every time a new token arrives. This approach is computationally expensive, scaling at $O(n^2)$, and often leads to "flicker"—a phenomenon where the output labels for previous tokens oscillate back and forth unnecessarily.

While researchers have proposed policies to decide when a model should recompute or revise its output, there has been no rigorous way to determine if these revisions are actually beneficial. Are they correcting genuine errors, or are they merely replacing one incorrect guess with another?

Proposed Methodology

The authors propose a three-part evaluation framework to characterize the behavior of incremental models:

1. Formalization: The Incremental Chart (IC)

To track predictions over time, the authors define the Incremental Chart (IC), a lower triangular matrix. This structure records every prediction made by the model at every time step, allowing for a precise distinction between an edit (a change to a specific label) and a revision (a change to the output prefix).

2. Characterization Dimensions

The framework introduces 10 dimensions to describe the nature of edits, including:

  • Effectiveness: Does the edit successfully transform an incorrect label into a correct one?
  • Distance: How far back into the past does the model reach to modify a label?
  • Oscillation: Does the model repeatedly flip a specific label between different values?

3. New Metrics: Pertinence and Appropriateness

Drawing on the concepts of precision and recall, the authors introduce four key metrics to evaluate revision quality:

  • R-Pertinence (Revision Precision): What fraction of the model's revisions actually targeted prefixes that were incorrect?
  • R-Appropriateness (Revision Recall): What fraction of incorrect prefixes were successfully identified and revised by the model?
  • A-Pertinence/Appropriateness (Addition): Similar measures applied to the model's decision not to revise (i.e., when it simply appends a new label to a correct prefix).

Experimental Setup

The methodology was tested by profiling three Transformer-based architectures across three standard sequence labelling tasks:

  • Slot Filling: Using the Snips dataset (voice assistant commands).
  • Part-of-Speech (POS) Tagging: Using the Universal Dependencies (UD) English Web Treebank.
  • Named Entity Recognition (NER): Using the CoNLL-2003 dataset.

Key Insights and Results

The Inefficiency of the Naive Approach

The study reveals a significant "Naive Tax" in restart-incremental models. In some tasks, only 7.7% of recomputations actually resulted in a change to the output, meaning the vast majority of computation was wasted on redundant processing.

Revision Quality vs. Quantity

The authors argue that high final accuracy is an insufficient metric for incremental systems. A "jittery" model that eventually reaches the correct answer but changes its mind dozens of times is difficult for downstream systems or human users to process. Advanced policies, such as TAPIR, demonstrate a "waiting" strategy: they are more patient, resulting in fewer revisions and higher Pertinence. When these models do choose to revise, they are usually correct, though this sometimes comes at the cost of Appropriateness (waiting too long to fix an error).

Performance Benchmarks

  • Efficiency: Policy-based models (like TAPIR) reduced the number of restarts to between 10% and 25% of the naive baseline.
  • Stability: The number of revisions was 2 to 3 times lower in policy-based models compared to the baseline.
  • Metric Trends: Most models achieved high A-Appropriateness (~0.9), indicating they rarely revise prefixes that are already correct. However, R-Appropriateness remained generally low, suggesting that identifying and fixing all internal mistakes incrementally remains a significant challenge for current architectures.

Conclusion and Future Directions

This research provides the tools necessary to move beyond simple accuracy and toward a more nuanced understanding of incremental model behavior. The authors acknowledge a limitation in current evaluation: the use of "incrementalized" gold standards that treat final ground truth as the target for every step. This can penalize models for failing to resolve true local ambiguities.

Future work will focus on developing incremental gold standards that account for linguistic ambiguity and creating more granular revision policies that can target specific labels for correction rather than recomputing entire prefixes.

Summary updated

Abstract

Incremental dialogue model components produce a sequence of output prefixes based on incoming input. Mistakes can occur due to local ambiguities or to wrong hypotheses, making the ability to revise past outputs a desirable property that can be governed by a policy. In this work, we formalise and characterise edits and revisions in incremental sequence labelling and propose metrics to evaluate revision policies. We then apply our methodology to profile the incremental behaviour of three Transformer-based encoders in various tasks, paving the road for better revision policies.
BibTeX
@inproceedings{madureira2023road,
  author = {Brielen Madureira and Patrick Kahardipraja and David Schlangen},
  title = {{The Road to Quality is Paved with Good Revisions: A Detailed Evaluation Methodology for Revision Policies in Incremental Sequence Labelling}},
  booktitle = {SIGDIAL},
  year = {2023},
  url = {https://openreview.net/forum?id=YGd7QVTmvz},
}
Figure