Record-Level Backward Compatibility for Universal Information Extraction

Many deployed language systems now use one model to pull structured records (entities, relations, events) out of text from very different domains: news, biomedical abstracts, clinical notes, receipts. When the team retrains that model and the new version scores higher on average, users in some domains may find that outputs which used to be correct are now wrong. The literature calls these "negative flips," and has shown that accuracy and compatibility come apart in structured prediction (Measuring and Reducing Model Update Regression in Structured Prediction for NLP). That work's remedy samples whole candidate outputs from the new model and lets the old model pick among them. A document in universal information extraction can contain dozens of independent records, though. A small sample of whole outputs will rarely contain one that keeps every record the old model got right and also adds every record the new model fixed.

We propose to make backward compatibility a property of individual records rather than of whole outputs. The method pools records from several samples of the new model. It then searches over combinations of those records, scoring each combination by the new model's likelihood plus an agreement term from the old model, while enforcing schema validity, so that a relation's arguments must themselves be extracted entities. We also propose to measure where regressions land across domains. Our hypothesis is that average gains from a model update hide flips concentrated in specialized, low-resource schemas such as biomedical and clinical ones.

Core Claim

Record-level recombination reduces negative flips in universal extraction more than whole-output re-ranking does, at the same number of new-model forward passes, and without lowering the new model's F1. The reason is structural. Whole-output re-ranking can only choose among the $k$ candidates it sampled. Recombination can assemble an output that appears in none of them, for example legacy-correct entities from candidate 3 combined with a newly fixed relation from candidate 7.

The three directions in this proposal play different roles. Structured-prediction theory supplies the mechanism: a search with a well-defined objective over a combinatorial output space. Practical deployment supplies the problem, which is that model updates break users' working outputs. Domain diversity is the regime that makes the problem acute, because one shared model serves schemas with very different amounts of training data.

Research Program

We first establish the phenomenon. We need a legacy/updated model pair of the kind seen in practice. The legacy model is a T5-based text-to-structure extractor in the style of Unified Structure Generation for Universal Information Extraction, which casts entity, relation, event and sentiment extraction as generating a structured string. The updated model is a stronger code-style extractor in the spirit of KnowCoder, which casts extraction as generating Python class instances and reports large zero-shot and low-resource gains. Both papers evaluate only static accuracy. Neither measures what an upgrade from one to the other breaks.

To measure that, we define a record-level negative flip as a gold record that the legacy model extracted correctly and the updated model misses or corrupts. For each dataset, the negative flip rate (NFR) is the number of such records divided by the number of gold records. We report it separately for each dataset in the UIE 13-benchmark suite and the KnowCoder 33-dataset suite, and separately on CoNLL 2003 and NCBI Disease to contrast news with biomedical text. The descriptive question is whether NFR correlates with low training-set size or with domain distance from the update's training data. If it does, average F1 is a misleading signal for deciding whether to ship an update.

The method follows. For an input $x$ we draw $k$ outputs from the updated model, using dropout sampling exactly as the regression paper does. We parse each output into records and pool them into a candidate set $R$. We then search for a subset $S \subseteq R$ that maximizes

$$ J(S) = \log p_{\text{new}}(y_S \mid x) + \lambda \log p_{\text{old}}(y_S \mid x), $$

subject to $S$ being schema-valid. Here $y_S$ is the canonical linearization of the record set $S$, each model's log-probability is computed by scoring that fixed string under the model, and $\lambda \ge 0$ trades off fidelity to the new model against agreement with the old one. Each model has its own linearization format; a deterministic converter maps between them.

The search is best-first over single-record additions and deletions, starting from the new model's top output. It borrows the lazy exploration of next-best alternatives from Lazy-k Decoding. That paper finds constraint-satisfying token assignments for encoder taggers and reports 38x to 182x speedups over integer-programming search. We reuse its search strategy, but move it from token labels to records and from hard arithmetic constraints to a compatibility objective.

Scoring a candidate costs one teacher-forced forward pass per model, so we cap the search at a fixed number of scored candidates and report wall-clock latency. We then tune $\lambda$ per domain on development data. This tests whether low-resource schemas need a stronger pull toward the old model.

Baselines

The naive reference point is to ship the updated model's top output unchanged. This defines the regression we are trying to remove, and it is the F1 ceiling we must not fall below by more than a small, pre-declared margin of 0.5 F1.

The strongest existing method is Backward-Congruent Re-ranking from Measuring and Reducing Model Update Regression in Structured Prediction for NLP. It draws $k=10$ dropout samples from the new model and returns the one the legacy model scores highest. That paper reports that it reduced Negative Flip Impact, the share of errors caused by regressions, by 58% in dependency parsing and by 73% in conversational semantic parsing. It also reports that it beat distillation and ensembling. No reported result found on Lacuna for BCR on information extraction.

We run BCR with the same $k$, the same sampled candidates, the same tuning data and the same budget of scored forward passes as our method. Any difference therefore comes from recombination alone. Beating BCR means a lower record-level NFR averaged over domains, a lower NFR on the worst domain, and F1 at least equal to BCR's, all with a paired bootstrap confidence interval that excludes zero.

The key ablation removes recombination: we run our search restricted to the $k$ whole candidates. In that form it reduces to BCR with a joint score. A second ablation sets $\lambda = 0$, which isolates schema-valid recombination under the new model alone.

We also include two training-time baselines. One is knowledge distillation from the legacy model during the update. The other is a simple record-level vote, described below.

Why not just…?

Why not just apply BCR as-is with a larger $k$? This is the real threat. If whole candidates already cover the useful combinations, recombination adds nothing. We settle it by sweeping $k$ from 5 to 40 for BCR and comparing at matched forward-pass budgets. We also directly measure an "oracle coverage" quantity: how often any single candidate contains every legacy-correct and new-correct record. If coverage is high, we will report that BCR suffices and the contribution shrinks to the measurement study.

Why not just take the union of legacy and new records, or vote between them? A union keeps old correct records cheaply, but it also keeps old errors and can produce relations whose arguments disagree with each other. Whether that matters is empirical. We implement both the union and a confidence-thresholded vote with schema repair, and expect them to reduce flips while losing precision. This is analogous to the overprediction that rule-based post-processing had to fix in Stanford MLab at SemEval-2021 Task 8, where cleanup heuristics raised quantity precision from 0.564 to 0.8153. If the vote matches our method on both NFR and F1, the search is unnecessary, and we will say so.

Why not just distill the old model into the new one during retraining? The BCR paper found that distillation and ensembles were costly and less effective on large structured output spaces. Distillation also has to be redone for every update and every domain. We nonetheless train a distilled updated model at equal training compute, to check whether that finding transfers to generative extraction.

Why not just prompt a large general model zero-shot? That swaps one update for a more disruptive one. Prompting also lags on schema-following: XNLP reports its structure-tuned 13B model at 90.2% on entity recognition versus ChatGPT's 87.6%. We include zero-shot prompting only as an additional "updated model" in the regression study.

Decisive Experiments

The first check concerns the phenomenon. Across the UIE and KnowCoder suites, if per-domain NFR is roughly uniform and small, the domain-concentration hypothesis fails.

The second check is the method comparison. Our method versus BCR, the vote and distillation, at matched budgets, on record-level NFR and F1 for every domain, with worst-domain NFR as the primary metric.

The third check is a sanity transfer to the original setting. We run our method on English EWT dependency parsing and on TOP, treating arcs and frame nodes as records, to see whether it reproduces or exceeds BCR's reported reductions in Negative Flip Impact. This tells us whether recombination is general or specific to extraction.

Finally, the oracle-coverage analysis states plainly when whole-output re-ranking is already enough, so that practitioners know when to use the simpler method.