Choosing Few-Shot Nested NER Models by Top-Down Self-Consistency

When a model must learn to find entities from five labeled sentences, the hard part is often deciding which trained model to trust, not training it. With a held-out set of five sentences, the ranking of learning rates, training lengths, and architecture settings is close to a coin flip. We propose to address this in nested named entity recognition (NER), the task of finding entity mentions that can sit inside one another, such as a protein name inside a longer DNA-region name. We use a model that refines its span predictions by sending information both upward, from short spans to the longer spans that contain them, and downward, from those longer spans back to their parts. Each refinement step is written as an expectation-maximization (EM) update. EM is an alternating procedure that first infers hidden labels and then re-fits parameters to those inferred labels. Run this way, the model produces a measurable quantity on unlabeled text: how much its bottom-up and top-down beliefs about each span disagree. Our claim is that this residual disagreement is a continuous, low-variance, label-free score. It should pick hyperparameters and decide when to stop iterating better than cross-validation on the tiny labeled set.

This matters for two reasons. First, the gains reported for feedback architectures were all tuned on large validation sets, so nobody knows whether those architectures survive honest few-shot use. Second, few-shot nested NER papers report numbers under protocols that do not rule out validation peeking. The project therefore also builds an evaluation harness for nested NER that enforces the label budget. Every comparison in this note runs inside that harness.

Core Claim

True Few-Shot Learning with Language Models showed that when the validation set is as small as the training set, leave-one-out cross-validation and minimum description length select prompts barely better than random. In that setting there is roughly a 27% chance of choosing a prompt 13% worse than average. That finding motivates this work. Feedback models such as Contextual Feedback Loops add exactly the hyperparameters that cannot be selected under those conditions. That paper fixes the number of feedback iterations globally at one, after observing a plateau on ImageNet-1k, PG-19, and Long Range Arena, all of which are large-sample benchmarks.

Our central proposal is to replace the labeled validation signal with an unlabeled structural one, and to make that signal the stopping rule for the feedback loop itself. A nested-entity model with top-down feedback makes two predictions about every span. The first comes from the span's own tokens. The second comes from what its enclosing span believes. On in-domain text, a well-fit model should make these two agree, and the agreement can be averaged over thousands of unlabeled target-domain spans rather than over a few dozen labeled ones. The claim is specific and falsifiable. Across a fixed grid of configurations, the residual disagreement on unlabeled target text should correlate with true test F1 more strongly than three label-based scores computed on the five-shot support set: leave-one-out span F1, leave-one-out continuous gold-span probability, and a span-level adaptation of unlabeled early stopping. The score should also yield lower selection regret than all three.

Method

The encoder and span scorer follow the few-shot nested setup of Few-shot nested named entity recognition (BCL), which pretrains a biaffine span model with contrastive loss on Few-NERD and adapts it to target support sets. Information in BCL flows only bottom-up. We add a top-down pass modeled on the inverse pyramid of Pyramid, which sends representations of longer phrases back to their sub-spans in a single forward pass. Pyramid used this pass to recover recall in fully supervised training, reaching 79.31 F1 on GENIA. We turn that single pass into an iterative estimation procedure.

In the E-step, each candidate span $s$ in a sentence receives two label distributions over entity types plus "none". The bottom-up distribution $q^{\uparrow}_s$ is computed from the span's own representation. The top-down distribution $q^{\downarrow}_s$ is computed from the current beliefs of the spans that contain $s$. The combined belief is

$$q_s \propto q^{\uparrow}_s \cdot q^{\downarrow}_s ,$$

and this product is projected onto non-crossing span structures with the masked inside algorithm of Nested Named Entity Recognition with Partially-Observed TreeCRFs. That paper treats unlabeled spans as latent tree nodes but trains by direct gradient ascent under full supervision. We reuse its inference routine as the structural constraint inside an alternating loop. In the M-step, the type prototypes are re-fit to the labeled support spans plus soft-labeled spans from unlabeled target sentences.

This alternating update is the neural-EM pattern of Neural Expectation Maximization, which also alternates soft assignments with representation updates. That work was confined to synthetic shape and digit grouping. Here it is applied to span hierarchies in text.

The selection score is the average disagreement on a pool $U$ of unlabeled target-domain spans:

$$C(\theta) = \frac{1}{|U|}\sum_{s \in U} \mathrm{KL}\left(q^{\uparrow}_s \,\|\, q^{\downarrow}_s\right),$$

where $\theta$ denotes one trained configuration. KL is the Kullback–Leibler divergence, a standard measure of how far one probability distribution is from another. The loop iterates until the drop in $C$ between steps falls below a fixed tolerance, so the number of iterations becomes an outcome rather than a tuned hyperparameter. Among candidate configurations, the one with the lowest converged $C$ is selected.

$C$ has an obvious failure mode: a model that labels nothing as an entity agrees with itself perfectly. To guard against this, configurations whose predicted entity density on $U$ falls outside a tolerance band around the support-set density are discarded. We also test the degenerate case directly rather than assuming the guard works.

Research Program

We first build the harness, because without it none of the comparisons is meaningful. It loads GENIA, ACE-2004, ACE-2005, GermEval, and NEREL through Datasets, with Few-NERD as the source domain. It fixes episode seeds for 1-shot and 5-shot support sets, and it holds an unlabeled target pool that is disjoint from the test sentences.

The harness extends the label-budget rules of the true few-shot codebase from prompts to structured prediction. Every selection method sees only the support set and the unlabeled pool. Test labels are touched only by an analysis-only oracle, which reports the best configuration in the grid so that we can measure regret.

The motivation for building this, rather than reusing the protocols of BCL and LPNER, comes from evaluation-methodology work. A Sober Look at the Unsupervised Learning of Disentangled Representations showed claimed gains traced back to hyperparameter and seed selection against ground truth. BARS showed reported gains disappearing under standardized pipelines.

We then train every method over the same grid of 48 configurations. The grid varies learning rate, number of adaptation steps, prototype temperature, and feedback strength. For the iterated model, the stopping tolerance is held fixed across the grid rather than tuned. For each episode, we record each method's test F1 and each selection score's value.

Baselines

The naive reference point is BCL run in the harness with the configuration and hyperparameters its authors reported, carried over unchanged with no target-side selection. Under its own protocol, BCL reports 46.06 ± 1.22 F1 on GENIA 5-shot, a 12.12% gain over NNShot (Few-shot nested named entity recognition). We expect this number to fall under true few-shot rules, and the size of the fall is itself a result.

The strongest existing accuracy competitor is LPNER, which uses label-informed prompts and prototypes. It reports 39.45 and 59.30 F1 on GermEval 1-shot and 5-shot, and 45.11 and 61.54 F1 on NEREL (LPNER). No reported LPNER number on GENIA, ACE-2004, or ACE-2005 was found on Lacuna. Neither codebase appears in the file, so both models will be reimplemented. Both receive the same encoder size, the same Few-NERD pretraining, the same 48-configuration grid, and the same selection rules as our model. Beating LPNER means higher mean query F1 than LPNER-selected-by-LOOCV, with non-overlapping 95% bootstrap intervals over at least 20 episodes per dataset and shot count.

The strongest selection competitors are scores computed on the same feed-forward or feedback model. The first is leave-one-out span F1 on the support set, which was the failing baseline in True Few-Shot Learning with Language Models. The second is leave-one-out mean probability assigned to gold spans, our span-level version of the continuous metric that DataDecide found far more stable than discrete accuracy at small budgets. The third is a span-level port of BUS-stop's confidence and class-distribution similarity on unlabeled data (Early Stopping Based on Unlabeled Samples in Text Classification), which reached 0.872 accuracy against 0.826 for a held-out split at 50 samples per class. All three selectors see the same data and choose from the same grid. Beating them means a higher Spearman rank correlation between score and test F1 across the grid, together with lower mean regret against the oracle.

Two ablations isolate the key ingredients. The first keeps iteration but fixes the iteration count at one, as in Contextual Feedback Loops, which isolates the convergence-based stopping rule. The second uses Pyramid's single top-down pass with no EM, selected by leave-one-out gold probability, which isolates whether iteration matters beyond simply having feedback.

Why not just…?

Why not just select with leave-one-out gold-span probability and skip the feedback model? This is the most serious threat. DataDecide's evidence that continuous likelihoods stabilize small-sample rankings could carry over to spans, and leave-one-out gold-span probability is cheap. Its weakness is that it is still averaged over the handful of labeled spans in five sentences, whereas $C$ is averaged over the unlabeled pool. The experiment compares the two directly on identical grids. If the continuous leave-one-out score matches $C$ on rank correlation and regret, we will report that the selection benefit comes from continuity rather than structure, and that feedback is justified only if it improves oracle accuracy.

Why not just port BUS-stop to spans? BUS-stop's signals compare output confidence and class proportions with reference profiles, so they ignore the parent-child structure that defines nested NER. It was also validated only on flat sentence classification. The span-level port is a full baseline. If it matches $C$, structural self-consistency adds nothing beyond distribution matching, and we will say so.

Why not just add Pyramid's inverse pyramid and fix one iteration, as Contextual Feedback Loops did? That paper's plateau after one iteration was measured on large datasets with large validation sets. It says nothing about whether one iteration is right when the prototypes themselves are fit to five sentences. The fixed-one-iteration and single-pass ablations test exactly this. If they match the convergence-stopped model in both accuracy and selection quality, the EM formulation is unnecessary.

Decisive Checks

Four results decide the proposal.

The first is the headline: across GENIA, ACE-2004, ACE-2005, GermEval, and NEREL at 1-shot and 5-shot, the rank correlation between $C$ and test F1 must exceed that of all three label-based or label-free alternatives. We also report the frequency of picking a configuration worse than the grid mean, the analogue of the 27% figure in Perez et al., and expect it to drop clearly.

The second is accuracy under the oracle. If top-down feedback does not raise oracle-selected F1 over oracle-selected BCL, the model has no accuracy case. In that event, the remaining contribution is the selection score and the harness.

The third is degeneracy. We deliberately include under-trained and over-regularized configurations that predict few entities. We then check whether the density guard removes them, and whether $C$ without the guard would have selected them.

The fourth is stability of the stopping rule. The number of iterations chosen by convergence should vary sensibly with domain shift, for example more iterations on GENIA than on ACE given the Few-NERD source. Its F1 should match or exceed the best fixed iteration count chosen by the oracle.

If the first check fails while the second succeeds, the contribution narrows to an honest re-evaluation of feedback in true few-shot nested NER. That result would still be worth reporting through the harness.