Spending Beam Search Only Where It Teaches: Disagreement-Gated Calibration for Generative NER

Generative named entity recognition (NER) treats entity extraction as text generation: a model such as BART reads a sentence and writes out its entities, or a tagged copy of the sentence. Models trained the standard way rank their own outputs badly. Among the top five outputs from beam search (a decoder that keeps several candidate outputs alive at once), the best one often scores much higher on entity F1 than the one the model ranks first. The most effective fix is to run beam search during training and teach the model to rank candidates by their true quality. It works, but it is expensive: search-in-the-loop training is reported to be 2–4x or more slower than ordinary training. Most of that cost is wasted on NER data. The great majority of tokens are "outside" any entity, and many sentences contain no entity or only an obvious one. On those sentences every beam candidate is the same, so the ranking loss has nothing to compare and contributes no gradient.

We propose to decide, sentence by sentence, whether training-time beam search is worth running. The decision uses statistics that training already produces for free: how much the beam candidates disagreed the last time a sentence was searched, and whether the ranking loss was active then. The same span-level disagreement also weights the ranking loss, so that gradient goes to the contested spans rather than to agreed-upon background text. The central question is empirical and falsifiable. At a fixed compute budget, does disagreement-gated search recover the accuracy of full search-in-the-loop calibration, and does it beat simply searching a random subset of sentences?

Core Claim

The closest existing work is Debiasing Generative Named Entity Recognition by Calibrating Sequence Likelihood (RerankNER). After standard maximum-likelihood training, it fine-tunes BART with beam-search candidates produced during training. It combines the usual likelihood loss with a margin ranking loss, which forces candidates with higher F1 to receive higher likelihood, and a gold-sequence term. It reports 93.48 F1 on CoNLL-2003 and 86.22 on ACE 2005, and finds that the best top-5 candidate is 8.1 F1 points above the top-ranked one. RerankNER searches every training sentence uniformly and weights every candidate pair equally.

Our claim has two parts. First, the sentences that produce a useful ranking signal are a small and predictable subset, and they can be identified from cached beam statistics rather than from new search or Monte Carlo dropout. Monte Carlo dropout estimates uncertainty by running many forward passes with dropout switched on. Second, weighting the ranking loss by span-level beam disagreement concentrates learning on the spans where the model is still undecided.

This connects three threads that have so far stayed separate. Beam-derived confidence is known to be well calibrated: How Does Beam Search improve Span-Level Confidence Estimation in Generative Sequence Labeling? aggregates probability across beam candidates (AggSpan, AggSeq) and reports an expected calibration error of 0.011 on ATIS, against 0.014 for the single-span baseline. That work uses the signal only at inference time. Confidence gating of search is known to save compute: Search or Accelerate switches between parallel decoding and beam search based on confidence, and is training-free. Background-class imbalance explains why the savings should be large here. We move the gate into the training loop and feed the beam confidence back into the loss.

Research Program

The base system is a RerankNER-style calibration stage on BART. We keep two formats: the entity-only linearization that RerankNER uses, and a context-aware tagging format following Rethinking Negative Instances for Generative Named Entity Recognition (GNER). GNER shows that writing out the full tagged sentence, background tokens included, improves boundary decisions. That format makes generated sequences longer and search costlier, which makes it the regime where gating should matter most.

For each training sentence $i$, we keep a small cache from the last time it was searched. The cache holds the top-$K$ candidates' entity sets, their normalized probabilities, and that round's ranking loss. From these we compute a span disagreement score $u_i$. For each span proposed by any cached candidate, we take the AggSpan probability $q_s$, which is the summed normalized probability of the candidates that contain span $s$. We then sum the binary entropy $H(q_s)$ over those spans. The score is zero when all candidates agree and grows with the number and severity of contested spans.

At each epoch, sentence $i$ is searched with probability

$$ \pi_i = \min\left(1,\; \beta \cdot \frac{u_i + \lambda\, r_i}{\bar{u} + \lambda \bar{r}}\right), $$

Here $r_i$ is the cached ranking loss, $\bar{u}$ and $\bar{r}$ are corpus means, $\lambda$ trades disagreement against past loss, and $\beta$ sets the expected fraction of the corpus searched. That fraction is our compute knob. A floor on $\pi_i$ keeps stale "agreed" sentences from never being revisited. Sentences seen for the first time use a single greedy-decode confidence until their first search.

The likelihood loss is still applied to every sentence. It is cheap because it uses the gold prefix rather than search. Searched sentences add a ranking loss in which each candidate pair $(a,b)$ is weighted by $1 + \alpha \cdot \mathrm{disagree}(a,b)$, where $\mathrm{disagree}(a,b)$ is the summed span entropy over spans on which $a$ and $b$ differ. Pairs that differ only in uncontested background structure therefore count less.

We will analyze three things. The first is the skip structure: what fraction of sentences have fully agreeing beams, and how this varies with the share of background tokens across CoNLL-2003, OntoNotes 5.0, ACE 2005, GENIA and CADEC. The second is staleness: how often a skipped sentence would have produced a nonzero ranking loss if searched. This is measured on a held-out audit sample searched every epoch. The third is confident errors, where all beams agree on a wrong answer. Disagreement cannot catch these cases; the past-loss term and the gold-sequence loss are meant to cover them.

Baselines

The naive reference point is BART trained with maximum likelihood alone, with no calibration stage. Its compute serves as the 1x unit. It establishes how large the calibration gain is in the first place.

The strongest existing method is full RerankNER calibration, which searches every sentence every epoch. RerankNER reports 93.48 F1 on CoNLL-2003 and 86.22 on ACE 2005 (Debiasing Generative NER). We found no reported training-time cost for RerankNER on Lacuna, so we will measure it ourselves. The general overhead reports for search-in-the-loop training are about 3.3x slower than teacher forcing (Sequence-to-Sequence Learning as Beam-Search Optimization) and 2–4x (CoNT). We will rerun RerankNER with the same backbone, beam size, number of epochs and tuning budget as our method, measuring GPU-hours and the number of decoder calls.

Beating it means one of two outcomes. Either we match its F1 within seed noise using at most half its search compute, or we exceed it at equal compute. In the equal-compute case, the savings go into more calibration epochs.

The decisive competitor is random gating. It searches a uniformly random subset of sentences with the same expected fraction $\beta$ and the same total compute. If disagreement gating does not beat random gating at matched $\beta$, the uncertainty ingredient has no value.

A further baseline is loss-only gating, with $\lambda \to \infty$ so that only past ranking loss drives selection. It isolates whether beam disagreement adds anything beyond "search where the loss was nonzero last time."

The ablation that isolates the loss-weighting ingredient runs disagreement gating with uniform pair weights ($\alpha = 0$). A second ablation reweights selected sentences by $1/\pi_i$. This tests whether deliberately focusing on contested sentences is better than an unbiased estimate of the full-search objective.

We also run a teacher-forced token classifier with Focal Loss, a loss that down-weights tokens the model already gets right. This places our numbers against the cheap imbalance fix. Its strongest reported number is on a different task: 81.6 F1 versus 78.9 for cross-entropy on IWSLT2011 punctuation prediction (Focal Loss for Punctuation Prediction). We found no reported Focal-Loss result for CoNLL-2003 NER on Lacuna, so we run it ourselves.

Why not just…?

Why not just search a random subset of sentences? This is the real threat. If useful sentences are spread uniformly, random subsampling at the same budget will match us, and the proposal reduces to "calibrate on less data." We test it directly at three budgets (searching 10%, 25% and 50% of the corpus) with five seeds each. The audit sample gives a mechanism-level check: it measures how much better disagreement predicts nonzero ranking loss than chance does. If that predictive edge is small on CoNLL-2003 but large on nested or discontinuous data, the honest conclusion is that gating pays only where ambiguity is concentrated.

Why not just gate on the cached ranking loss and skip the uncertainty signal? The ranking loss is zero once candidates are correctly ordered by the margin, so past loss is a natural gate. But it is a lagging signal. A sentence whose candidates are currently well ordered can become misordered as the shared parameters move, and a sentence whose candidates all agree gives zero loss whatever its status. Disagreement tracks how close the model is to changing its mind. The loss-only baseline settles which signal matters. If it ties our method, we will report past-loss gating as the result and treat disagreement as unnecessary.

Why not just use Focal Loss under teacher forcing and avoid search altogether? Focal Loss addresses gradient swamping at no extra cost. However, it acts on single-token probabilities and cannot fix sequence-level misranking. The 8.1-point top-5 oracle gap reported by RerankNER is a ranking failure, not a token-weighting failure. The Focal-Loss baseline above shows how much of the calibration gain is attainable without search.

Why not just apply beam-confidence aggregation at inference time? AggSeq-style aggregation (How Does Beam Search improve…) improves confidence estimates but leaves the model's ranking unchanged. It also adds decoding cost at every prediction. We include it applied to the maximum-likelihood model, reporting both F1 and calibration error, to separate "better confidence scores" from "better predictions."

Why not just use Monte Carlo dropout for uncertainty, as UANet and SeqUST do? Uncertainty-Aware Label Refinement for Sequence Labeling and Uncertainty-Aware Self-Training for Low-Resource Neural Sequence Labeling both need several stochastic forward passes. Spending those passes to save beam search largely defeats the point. Our cached beam statistics cost nothing beyond the searches already performed. We include one Monte Carlo dropout gate, charged its full compute, to check whether its uncertainty estimates are good enough to pay for themselves.

Proposed Experiments

The main experiment plots span F1 against measured GPU-hours for every method on CoNLL-2003, OntoNotes 5.0, ACE 2004/2005, GENIA and CADEC. Each method is run in both the entity-only and the GNER-style tagged format, with five seeds. The plot gives a full cost–accuracy curve rather than single points.

We expect the gap between disagreement gating and random gating to widen as two things increase: the share of sentences whose beams agree, and the length of the generated sequence. This would show up most in the tagged format and on nested data. The reason is that random gating wastes its budget in proportion to how many sentences carry no signal.

A decisive negative result would be any of three outcomes. Random gating could match disagreement gating at every budget. Loss-only gating could match it. Or the audit sample could show that skipped sentences produce ranking loss at the same rate as searched ones. Any of these would show that the uncertainty signal does not identify where search pays off.

Secondary checks cover four things: per-entity-type recall on rare classes, calibration error of the final model on ATIS and SNIPS using the evaluation protocol of the beam-confidence paper, the fraction of confident errors that the gate never revisits, and sensitivity to $\beta$, $\lambda$ and $\alpha$ under the same tuning budget given to RerankNER.