Many deployed information-extraction systems pair hand-written rules (regular expressions, dictionaries) with a learned model, and the learned part is often an off-the-shelf component whose training data nobody can examine. Once these systems ship, they are rarely measured again. Teams report a test-set score from before deployment, or report usage counts, and then assume nothing has changed. We propose a cheap and statistically honest way to keep measuring them. The rule path and the learned path of a hybrid extractor already produce two independent-ish opinions on every input. Sorting inputs by whether those two opinions agree divides live traffic into groups with very different error rates. Spending a small human-labeling budget unevenly across those groups gives precision, recall, and F1 estimates with confidence intervals, using far fewer labels than random spot checks. In addition, shifts in the size of the groups flag degradation before any label arrives.
The approach needs only the outputs of each component. It therefore works when the speech recognizer, the named-entity recognizer (a model that tags spans such as names, drugs, or dates), or the language model is a black box trained on undisclosed data. That is exactly the case where pre-deployment numbers are least trustworthy, because nobody can predict which accents, lexicons, or noise conditions the component never saw.
Core Claim
In a hybrid extractor, the agreement pattern between the rule path and the learned path is a free stratification variable that is strongly predictive of error. Each prediction falls into one of five groups, which we call strata: both paths fire and agree; both fire but disagree on the span or the type; only the rule fires; only the model fires; neither fires. Errors concentrate in the disagreement strata, and the estimator exploits that.
We use stratified sampling, meaning a fixed labeling budget is split across these strata rather than spent on a uniform random sample. With per-stratum weights $W_h$ (the fraction of extracted spans in stratum $h$) and per-stratum precision estimates $\hat p_h$ from labeled samples, overall precision is estimated as
$$\hat P = \sum_h W_h \hat p_h, \qquad n_h \propto W_h \sqrt{p_h(1-p_h)},$$
where $n_h$ is the number of labels allocated to stratum $h$. The allocation rule on the right is the standard variance-minimizing (Neyman) allocation. When strata differ sharply in $p_h$, its variance falls well below that of uniform sampling, and that gap is the source of every gain we expect.
Recall needs the same treatment for missed entities. The "neither fires" stratum is sampled at the sentence level, so it catches entities both paths missed. The resulting estimate stays unbiased even if both paths fail together; correlated failure only makes it less efficient, never wrong. Separately, with zero labels, the vector of stratum proportions $W_h$ serves as a drift signal. We test whether it moves when true F1 moves.
What is new is not stratified sampling itself. It is the observation that a hybrid architecture produces an error-predictive stratification for free and needs no access to confidence scores, training data, or model internals. We then test when that holds and when it breaks.
Where This Sits
The motivating evidence comes from three places.
First, deployed systems drift out of measurement. Implementation of large language models in electronic health records deployed a Qwen3-235B assistant in Epic for 28 physicians and tracked 64% daily use. It reported no extraction accuracy at all, and physician feedback faded as the tool became "good enough." Hear Me Out audits speech foundation models with undisclosed training data, but through interactive exploration rather than a reproducible metric. We target the step these papers skip: a number with an error bar, maintained after deployment.
Second, hybrid systems built on opaque components already fall back to expensive manual review. Extending Off-the-Shelf NER Systems to Personal Information Detection combined regular expressions with spaCy and RoBERTa models. Privacy rules barred the authors from inspecting either the model training data or the target dialogues. They reached 0.99 recall and 0.64 precision only through repeated rounds of annotator review. Their result also suggests that the "rule-only" stratum is where false positives cluster, which is the kind of structure our estimator relies on.
Third, two papers constrain the design. Extracting Periodontitis Diagnosis in Clinical Notes with RoBERTa and Regular Expression trained RoBERTa on labels produced by regular expressions over 8,065 notes and evaluated on only 60 gold notes. When the model learns from the rules, the two paths stop being independent witnesses, and agreement may carry little information. We treat this as a failure mode to measure, not ignore. Medical Spoken Named Entity Recognition shows that the shift we must detect is large: F1 falls from 0.74 on clean text to 0.58 on transcripts with about 29% word error rate. Its gazetteer-based annotation workflow supplies a ready-made rule path for our experiments.
We deliberately do not pursue the more obvious direction suggested by lattice-based confidence estimation for black-box ASR, which raised confidence AUC from 0.7496 on 1-best output to 0.9231 using word lattices (graphs of alternative transcriptions). Our premise is that the speech recognizer is a black box, and black-box recognizers typically do not expose lattices. A method that requires them would fail in the setting we care about.
Research Program
The work has three parts.
First, we build the estimator for span-level precision, recall, and F1. F1 is a ratio of counts, so we estimate true positives, false positives, and false negatives per stratum and propagate intervals by bootstrap. Allocation is adaptive: a small pilot sample per stratum estimates $p_h$, and the rest of the budget follows the Neyman rule.
Second, we characterize when agreement predicts error. For each benchmark we measure per-stratum error rates and the correlation between rule errors and model errors. We do this under three conditions: clean versus noisy speech-recognition input, a model trained on gold labels, and a model trained on rule-generated labels in the style of the periodontitis paper. This produces a usable diagnostic, namely the ratio of between-stratum to within-stratum variance, that tells a practitioner in advance whether the method will help.
Third, we test the label-free drift signal. We ask whether changes in stratum proportions track the true change in F1 across known shifts.
Baselines
The do-nothing reference reports the pre-deployment test F1 as if it still held. On VietMed-NER this would claim 0.74 while the true score on speech-recognized input is 0.58 (Medical Spoken Named Entity Recognition). On SLURP it would claim 78.19 SLU-F1 against a true 70.84 (SLURP). Any method should at least detect these gaps.
The strongest real competitor is uniform random annotation at the same label budget. It is unbiased, simple, and what careful teams already do, as the 60-note gold set in the periodontitis paper illustrates. We run it with identical budgets (25 to 400 labeled sentences), identical span-matching rules, and 1,000 repeated draws. We count our method as beating it if, at equal budget, it either reduces confidence-interval width by at least 30% while keeping coverage near the nominal 95%, or reaches a ±0.03 F1 interval with half the labels. No reported result was found on Lacuna for label-efficient F1 estimation on any of these benchmarks, so the comparison numbers will be our own reproductions.
A second baseline stratifies by model confidence, binning spans by the learned model's softmax score. We run it only where scores are available. It isolates the key ingredient of the proposal: whether rule–model agreement carries information beyond what the model already knows about its own uncertainty.
A third baseline is a label-free proxy: the raw rule–model agreement rate, reported directly as an accuracy stand-in. The fourth is an ablation that reuses our strata but allocates labels proportionally to stratum size rather than by the Neyman rule, which separates the value of the strata from the value of the allocation.
Why not just…?
Why not just annotate a random sample each month? This is the real threat, and it is unbeatable on bias. Our estimator only helps if the strata actually differ in error rate. If rule and model errors are nearly independent of agreement, uniform sampling ties our method and the proposal fails. The variance comparison in every experiment decides this directly. The labeling-cost argument rests on evidence that strata do differ: the 0.99 recall versus 0.64 precision split in the personal-information pipeline suggests rule-only predictions are far noisier than agreed ones.
Why not just use the model's confidence scores? Black-box components often expose none, as with the 1-best-only recognizers studied in the lattice confidence paper. Even where scores exist, they come from a model trained on unspecified data and may be poorly calibrated after a shift. Rules do not shift with acoustic conditions in the same way. The confidence-stratified baseline measures this. If it matches agreement stratification on clean text but loses under speech-recognition noise, that supports the claim. If it wins everywhere, we report that agreement adds nothing when scores exist.
Why not just have an LLM judge each extraction? An LLM judge is another component with unspecified training data, and its errors are unmeasured. That simply moves the evaluation problem one level up, and the EHR deployment shows how such systems escape measurement. An LLM judge could, however, serve as a third path that defines finer strata. We include it as an optional stratifier on SLURP, never as a source of ground truth.
Proposed Experiments
All experiments use benchmarks with gold labels and treat those labels as a hidden oracle that answers only the queries a method buys.
On VietMed-NER, the rule path is the medical gazetteer from the dataset's annotation workflow, and the model path is PhoBERT or XLM-R. We evaluate on clean references and on transcripts with about 29% word error rate.
On SLURP, rules cover dates, times, numbers, and email-like entities, and the model is a fine-tuned NER tagger. Shifts are from headset to distant microphone and from SLURP-synth to real audio, scored with SLU-F1.
On the French MEDIA corpus, the stochastic finite-state transducer serves as the symbolic path and the CRF as the learned path, following System Combination for Spoken Language Understanding. That paper reports CRF concept error rising from 11.8% to 29.8% under speech-recognition noise.
Three checks are decisive. First, at budgets of 50 and 100 labels, the stratified interval must be meaningfully narrower than uniform sampling while still covering the true F1 at near-nominal rates, on both clean and noisy conditions. Second, when the learned model is retrained on gazetteer-generated labels instead of gold labels, we expect the efficiency gain to shrink. The pre-computed variance-ratio diagnostic must predict that shrinkage; if it does not, the diagnostic is useless to practitioners. Third, the zero-label shift in stratum proportions must move in the right direction and roughly in proportion for every known shift: clean to noisy VietMed, headset to distant SLURP, and reference to recognizer output on MEDIA. If the "neither fires" stratum's share does not grow as recall falls, the drift signal does not work and we drop it from the claim.
References
- Implementation of large language models in electronic health records
- Hear Me Out: Interactive evaluation and bias discovery platform for speech-to-speech conversational AI
- Extending Off-the-shelf NER Systems to Personal Information Detection in Dialogues with a Virtual Agent: Findings from a Real-Life Use Case
- Extracting Periodontitis Diagnosis in Clinical Notes with RoBERTa and Regular Expression
- Medical Spoken Named Entity Recognition
- Confidence Estimation for Black Box Automatic Speech Recognition Systems Using Lattice Recurrent Neural Networks
- SLURP: A Spoken Language Understanding Resource Package
- System combination for spoken language understanding