Named entity recognizers are now often trained on text written by a large language model (LLM): you give the LLM a few seed entities, ask it to write sentences containing them with the entities tagged, and train a smaller extractor on the result. Such models lose accuracy on real documents. One common way to build these extractors is to represent each candidate phrase by the encoder's vectors at its first and last token, the "boundary" representation. We argue that this design has a specific weak point. A boundary token's vector is shaped heavily by the words right next to it. In LLM-generated text, those neighbouring words are exactly where the generator's habits concentrate: stock framings like "the drug X was administered", "patients with X", or leftover traces of the markup it was asked to produce. So the boundary vector may learn "this phrase sits in an LLM-style slot" rather than "this phrase names a chemical".
The proposal has two parts. First, we measure where synthetic-versus-real information lives inside span representations. Second, we test a targeted fix borrowed from simulation-to-reality work in 3D perception. During training, a small learned module chooses which words flanking each span to mask in synthetic sentences. It is trained so that a discriminator can no longer tell synthetic spans from real (unlabeled) spans using the boundary vectors. If the weak point is real, it names a mechanism for part of the synthetic-to-real gap in span models and gives a cheap way to reduce it. If it is not real, the diagnostic alone rules out an assumption the field has never checked.
Core Claim
Our hypothesis is that span models trained on LLM-generated data carry the synthetic-versus-real signal mostly in their boundary vectors, specifically through the context immediately outside the span, and much less in pooled interior tokens. Perturbing that flanking context during training should therefore recover more real-domain F1 than perturbing the whole sentence, pooling the interior instead of using boundaries, or aligning features adversarially without touching the input. The claim is falsifiable at two points. The diagnostic can show no boundary-specific concentration, and random flank masking can match the learned masking. We treat both as outcomes that would change the recommendation, not as nuisances.
What Prior Work Does and Leaves Open
The span side supplies the architecture. Several works represent a span by its endpoint vectors, sometimes adding length embeddings or interior pooling: End-to-end Neural Coreference Resolution, SpanNER, S-NER, Deep Exhaustive Model for Nested NER, and Enhancing NER by Harnessing Multiple Datasets with CVAEs. Coreference Resolution without Span Representations goes further and argues that in Transformers the endpoints alone are sufficient. That claim was established on real in-domain text. Our diagnostic asks whether "sufficient" becomes "fragile" when the training text is synthetic.
Improving Span Representation for Domain-adapted Coreference Resolution is the closest precedent for a boundary-specific failure under domain shift. It shows that boundary vectors latch onto sub-word fragments when moving from general to clinical text, and it fixes this with ontology-based losses. It studies real-to-real transfer and a vocabulary cause. We study synthetic-to-real transfer and a context cause.
The synthetic-data side motivates the problem. GenLLM generates tagged sentences from seed entities with LLaMA-3.2-3B. BioSynNER paraphrases real seed sentences because text written from scratch loses structure, and it reports boundary noise and false positives. Both train sequence taggers and neither asks where the gap enters the representation. Familiarity constrains our evaluation: synthetic corpora can overlap semantically with test label sets and inflate transfer scores, so we must control label overlap before attributing any gain to our method.
The method comes from perception. Adversarially Masking Synthetic to Mimic Real (ASM) learns discrete masks with Gumbel-Softmax, a trick for sampling differentiable on/off choices, to corrupt clean synthetic LiDAR point clouds toward realistic dropout. It reports that this beats standard feature alignment on SemKITTI (22.8% versus 20.0% mIoU). PixelDA likewise finds that changing the input beats feature-level alignment. We carry ASM's idea to text and aim it at the only words the boundary vectors are hypothesized to over-rely on.
Research Program
The first stage is a diagnostic. We train three span extractors of equal size on the same synthetic corpus: boundary-only (start and end vectors plus length), interior-pooled (mean or max pooling plus length, as in S-NER), and hybrid. For each, we fit a linear probe that predicts whether a span came from synthetic or real text, using held-out synthetic spans and real unlabeled spans matched by entity string.
We then run a counterfactual swap. We keep an entity mention fixed, move it between a synthetic sentence and a real sentence, and measure how often the model's prediction flips. Finally, we localize the effect by re-running the swap with only the $k$ words on each side of the span replaced. The hypothesis predicts three things: higher probe accuracy for the boundary vectors, higher flip rates for boundary models, and flip rates that are mostly explained by the flanking words alone.
The second stage is flank-targeted adversarial masking. For a synthetic sentence with gold span $(i,j)$, a small masking network looks at the tokens in windows $[i-k, i-1]$ and $[j+1, j+k]$. For each one it samples a keep-or-mask decision through Gumbel-Softmax, and masked tokens are replaced by the encoder's mask token before encoding. A discriminator $D$ receives the boundary representation $g=[h_i; h_j]$ and tries to tell synthetic spans from real unlabeled candidate spans. The extractor $E$ is trained on the span loss while the masking network $M$ is trained to fool $D$:
$$\min_{E,M}\max_{D}\; \mathcal{L}_{\text{span}}(E; M(x_{\text{syn}})) \;-\; \lambda\, \mathcal{L}_{\text{disc}}\big(D; g_E(M(x_{\text{syn}})), g_E(x_{\text{real}})\big) \;+\; \beta\,\rho(M).$$
Here $\mathcal{L}_{\text{span}}$ is the span classification loss, $\mathcal{L}_{\text{disc}}$ is the discriminator's cross-entropy, $\lambda$ weights the adversarial term, and $\rho(M)$ penalizes the fraction of tokens masked with weight $\beta$, so the mask cannot erase all context.
Unlike standard feature alignment, the extractor itself is never asked to be domain-invariant directly. The only way it can reduce the discriminator's signal is through which flanking words are shown. This is the same design choice ASM credits for avoiding the discriminator latching onto a shortcut. At test time no masking is applied.
We control for false negatives in synthetic text, which could otherwise be confused with the effect we are studying. Every span model uses the negative sampling that S-NER reports doubles training speed, and we additionally report results with a fixed negative sampling rate.
Baselines
The naive reference point is the same boundary-only span model trained on the synthetic corpus with no adaptation. It anchors every gain. The upper reference is the same model trained on the real training split, which is the "oracle" in ASM's terms. We report the fraction of the gap between the two that each method closes.
The strongest existing reported number in our setting is GenLLM's 72.85 F1 on BC5CDR with 50 seed entities, from a sequence tagger trained on LLaMA-generated data (GenLLM). We regenerate data with the same recipe and a matched seed budget, train the tagger and all span models on that identical corpus with the same encoder, and give every system the same hyperparameter search budget. Beating it means exceeding its re-run F1 by more than the seed-to-seed standard deviation across five generation seeds. If our re-run falls short of 72.85, we report both numbers.
We also compare against the interior-pooled S-NER-style model. If pooling the interior already avoids the boundary shortcut, it is the most natural competitor to adversarial masking.
For standard feature alignment, we run DANN-style gradient reversal on the same boundary vectors with the same unlabeled real text. ASM's finding in LiDAR suggests it will do worse, but we have not assumed that holds for text.
The key ablations isolate each ingredient. Random masking of flanking tokens at the same average rate tests whether learning the mask matters. Adversarially learned masking over the whole sentence tests whether targeting the flanks matters. Applying our masking to the interior-pooled model tests whether the fix is specific to boundary vectors.
Why not just…?
Why not just use interior pooling and skip boundaries? If the diagnostic shows that pooled representations carry little synthetic signal and match the boundary model's real F1, that would be a legitimate answer: stop using boundaries when training on synthetic text. The priors conflict. Kirstain et al. find the interior redundant in Transformers, while S-NER finds max pooling best. The three-way comparison in stage one settles which applies here, and we will report pooling as the recommendation if it wins.
Why not just mask flanking words at random? This is the most serious threat, because it is almost free. If random flank masking closes as much of the gap as the learned version, the contribution becomes the localization result plus a one-line recipe, and we will say so. The learned mask should help only when the generator's habits are uneven, for example concentrated on particular framing words or on one side of the span. We test that by also reporting results on paraphrase-based synthetic data from BioSynNER, whose contexts are closer to real text. There we expect the learned-versus-random difference to shrink.
Why not just make the synthetic text more realistic by paraphrasing real seeds? BioSynNER does exactly that and still reaches only 65.62% of original performance with 20 seed sentences per label and 73.52% with 50. So a gap remains even after paraphrasing. We treat paraphrased data as a second source condition rather than a competitor, and ask whether masking adds to paraphrasing.
Why not just prompt an LLM at test time? Familiarity shows that zero-shot scores are inflated by label overlap, falling from 45.8 to 28.0 F1 on a low-overlap NuNER split. We do not claim small extractors beat prompting in general. Our question is how to train cheap extractors, which people deploy regardless of whether prompting is available.
Proposed Experiments
The main biomedical testbed is BC5CDR, using GenLLM-style generated data at seed budgets of 5, 50, and 500 entities, plus paraphrased data. We add nested GENIA, where boundary span models are the natural choice because a single tag per token cannot represent entities inside entities. For general domains, we train on the CrossNER domains with PileNER or NuNER-style synthetic sources and report the FAMILIARITY score for each source-target pair, so that gains can be read against label overlap. For GENIA, the supervised reference is the 77.1 F1 of Deep Exhaustive Model for Nested NER. For TDMSci, the file contains no reported per-class numbers, so we would establish our own references there.
Metrics are span-level micro F1 on real test sets and the fraction of the synthetic-to-oracle gap closed. For mechanism, we also report probe accuracy, counterfactual flip rate, and F1 broken down by span length and by how "LLM-typical" each span's flanking context is. We score typicality with a language model classifier trained to separate synthetic from real sentences.
Three checks decide the outcome. The hypothesis survives only if boundary vectors show clearly higher synthetic-versus-real probe accuracy than pooled vectors, and flank-only swaps account for most of the prediction flips. The method is worth keeping only if flank-targeted learned masking beats whole-sentence masking, random flank masking, and DANN on real F1 across five generation seeds. Its gains must also concentrate on spans with LLM-typical flanking context, and must not come from pairs with high FAMILIARITY. If the first check fails, we publish the negative diagnostic. If only the second fails against random masking, we publish the simpler recipe.
References
- End-to-end Neural Coreference Resolution
- SpanNER: Named Entity Re-/Recognition as Span Prediction
- S-NER: A Concise and Efficient Span-Based Model for Named Entity Recognition
- Deep Exhaustive Model for Nested Named Entity Recognition
- Enhancing NER by Harnessing Multiple Datasets with Conditional Variational Autoencoders
- Coreference Resolution without Span Representations
- Improving Span Representation for Domain-adapted Coreference Resolution
- Overcoming Data Scarcity in Named Entity Recognition: Synthetic Data Generation with Large Language Models
- BioSynNER: Synthetic Data for Biomedical Named Entity Recognition
- Familiarity: Better Evaluation of Zero-Shot Named Entity Recognition by Quantifying Label Shifts in Synthetic Training Data
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks