Language models are now routinely used to write synthetic training sentences for entity extraction in domains where expert annotation is scarce, such as biomedicine. The models trained on that text are usually span-based extractors, which score whole candidate phrases instead of tagging tokens one at a time. To stay affordable, these extractors only consider phrases up to some maximum length. This note argues that the gap between synthetic and real data, in this setting, is concentrated in one measurable place: entity length.
LLM paraphrases tend to produce compact, well-formed mentions. Real clinical text contains long, clause-like entities, such as PICO mentions (the Population, Intervention, Comparison and Outcome phrases of a clinical trial) that run a full sentence. A span model trained on synthetic text therefore rarely sees long positives. If it also caps candidate length, it cannot recover them at all. We propose to measure synthetic-to-real transfer as a function of span length. We then borrow domain randomization from robot simulation, which means deliberately spreading simulator parameters far beyond realistic values. The randomized parameter here is entity length: we rewrite synthetic entity mentions to lengths sampled from a broad distribution that is not tied to the seed data. We then test whether this closes the long-span gap better than trying to match the real length distribution.
Core Claim
Our hypothesis has two parts. First, most of the drop from real-data to synthetic-data training in biomedical span extraction occurs on entities longer than about 10 tokens. Second, length-randomized synthesis paired with a span model that has no length cap recovers that loss more reliably than length-faithful synthesis.
The second part depends on the realistic low-resource case. When only 20–50 seed sentences per label exist, the tail of the real length distribution cannot be estimated from the seeds. A generator that faithfully copies the seeds therefore under-produces exactly the long spans that real test data contains. This is the variety-versus-fidelity question applied to text. In vision, Training Deep Networks with Synthetic Data: Bridging the Reality Gap by Domain Randomization found that crude randomized renders beat photorealistic ones on real KITTI (71.5 vs 64.6 AP). For biomedical NER, BioSynNER: Synthetic Data for Biomedical Named Entity Recognition argues for faithful paraphrasing of seed sentences. We claim the right answer depends on the attribute: keep the surrounding context faithful, and randomize the entity length.
Relation to Prior Work
The closest work is BioSynNER, which mines attributes from seed sentences (including length), paraphrases them with an LLM, and expands entity vocabulary using the Unified Medical Language System (UMLS) ontology. It recovers 65.62% of real-data performance on average with 20 seeds per label and 73.52% with 50. BioSynNER motivates this proposal and supplies its base generator. However, it never checks whether paraphrasing shifts the length distribution of mentions, never breaks results down by length, and uses standard architectures with no attention to length caps. Our contribution is the length-resolved diagnosis and an intervention aimed at it.
A span-based model for extracting overlapping PICO entities from randomized controlled trial publications introduces PICOX. PICOX removes the length cap by first predicting start and end tokens and then classifying the resulting candidate spans. It adds synthetic negative "composite spans" that pair a start and end from different entities, which raised EBM-NLP precision from 51.86 to 53.49. PICOX supplies both an uncapped architecture and a negative-synthesis trick that we reuse. It also constrains our design, because it reports that long spans get merged with neighbours (e.g. "epistaxis and headache" becoming one entity). Lengthening synthetic positives could make that merging worse, so our generator also inserts adjacent short entities as distractors. All of PICOX's synthetic data is negative and built from real sentences; it never trains on synthetic positive documents.
Training on Synthetic Data Beats Real Data in Multimodal Relation Extraction uses a teacher model to filter synthetic examples by how well their label can be predicted, which gave a 24–26% F1 improvement over unfiltered synthesis. We reuse that filter to remove rewrites in which a lengthened mention no longer denotes its entity type, the "label loss" failure that paper describes.
The span-architecture papers set up a second question: which architectures suffer most when trained on synthetic data. Coreference Resolution without Span Representations scores spans only from their start and end tokens and matches full span representations at much lower memory. Deep Span Representations for Named Entity Recognition shows that such shallow boundary representations lose more than 30 absolute F1 to deep span encoders on ACE 2004 entities longer than 10 tokens. Propose-and-Refine: A Two-Stage Set Prediction Network for Nested Named Entity Recognition recovers 73.42 F1 on ACE05 entities longer than 17 tokens through boundary re-prediction, even though its length-capped proposal stage found none of them. None of these papers trains on synthetic data, so whether boundary-only or deep span models transfer better from synthetic text is open.
Research Program
The first component is a diagnostic. We generate BioSynNER-style synthetic corpora from 20 and 50 seed sentences per label for GENIA, NCBI Disease, BC5CDR and EBM-NLP. For each corpus we compare the length histogram of synthetic mentions against the real training set, reporting KL divergence and tail mass above 10 and above 17 tokens. We then train extractors on each corpus and report F1 bucketed by gold entity length on real test data. The quantity of interest is the retention ratio, synthetic-trained F1 divided by real-trained F1, plotted against length.
The second component is the generator. Seed paraphrases are produced as in BioSynNER. Each entity mention is then rewritten by an LLM to a target token length drawn from a log-uniform distribution over 1 to 40 tokens. The rewrite must stay anchored to the mention's UMLS concept and its type, for example by adding qualifiers, dosing, or a population description. The surrounding sentence is otherwise left unchanged. Two further steps follow: the teacher filter removes rewrites whose label no longer holds, and in a controlled fraction of sentences we insert coordinated short entities next to a long one. We keep PICOX's composite negatives throughout.
The third component crosses these data conditions with architectures, so we can measure whether a length cap amplifies the synthetic gap. The architectures are an exhaustive span classifier capped at L=8 and L=16, the boundary-only s2e scorer, DSpERT, and PICOX.
Baselines
The naive reference point is BioSynNER's paraphrase data with no length intervention, trained under each architecture with the same seeds, generator model, number of synthetic sentences and hyperparameter search budget. Its reported retention of 65.62% (20 seeds) and 73.52% (50 seeds) is the aggregate figure to beat. We will recompute it per length bucket, because no length-resolved number for it was found on Lacuna. We expect our method to raise long-span retention clearly without lowering the aggregate.
The strongest competitor is length-faithful synthesis. It is the same pipeline, but target lengths are sampled from the empirical seed length histogram, which is BioSynNER's own premise taken to its conclusion. We also run an oracle-faithful variant that samples from the full real training histogram, information a true low-resource user would not have. Beating seed-faithful synthesis on spans longer than 10 tokens is the main claim. Matching or beating oracle-faithful synthesis would show that randomization covers the tail at least as well as knowing the true length distribution.
The real-data references are PICOX trained on real EBM-NLP, which reports 50.87 micro F1, and DSpERT on real GENIA, which reports 81.90 F1. These serve as ceilings for the retention ratio. No reported result was found on Lacuna for any synthetic-only training on EBM-NLP.
Three ablations isolate the ingredients:
- length randomization without the adjacent-entity distractors;
- randomization truncated to the maximum seed length, which isolates the value of going beyond observed support;
- distractors and filtering without any length change.
All conditions use the same number of synthetic sentences and the same compute.
Why not just…?
Why not just prompt the LLM to match the real length distribution? This is the most serious threat, and it is why the oracle-faithful baseline exists. If oracle-faithful synthesis does as well as randomization, the honest conclusion is that the problem is estimating the length distribution, not randomizing it. The practical recommendation would then shift to better tail estimation from seeds. The seed-faithful versus oracle-faithful comparison measures how much is lost by estimating from 20 seeds, which is the premise of our argument.
Why not just remove the length cap and keep the data as is? Lifting the cap removes the structural zero-recall blind spot. However, a classifier that never sees long synthetic positives has no reason to score long candidates highly. PICOX also shows that uncapped models struggle with boundaries on long spans even when trained on real data. The architecture-by-data grid answers this directly: if uncapped PICOX trained on naive synthetic data already matches its real-trained long-span F1, our data intervention is unnecessary.
Why not just prompt GPT-4o to extract entities directly? Small Language Model Makes an Effective Long Text Extractor reports that GPT-4o reaches 21.87 F1 on long-entity Scholar-XL, against 51.56 for a dedicated span model. This suggests prompted extraction is weakest exactly where our gap lies. We include zero-shot prompting with the same seeds as in-context examples as a reference on EBM-NLP.
Why not just fine-tune on the seed sentences? We will report seed-only training and seed-plus-synthetic training. Following the domain-randomization result that synthetic pretraining followed by real fine-tuning beats real data alone (98.5 vs 96.4 AP on KITTI), we expect the combination to be best. The proposal only matters if the randomized-synthetic-plus-seeds condition beats both naive-synthetic-plus-seeds and seeds alone on long spans.
Proposed Experiments and Decisive Checks
Evaluation covers GENIA, NCBI Disease, BC5CDR and EBM-NLP, plus EBM-NLP's AD and COVID-19 PICO sets to test generalization. The metrics are micro F1, retention ratio, and F1 bucketed at 1–3, 4–10, 11–17 and more than 17 tokens. Each condition is run with five seeds and evaluated with paired bootstrap tests.
Three outcomes would decide the question. The first is whether the diagnostic actually shows synthetic mentions compressed in length, with retention falling with length. If it does not, the premise fails and we stop. The second is whether randomization beyond seed support beats the truncated ablation and seed-faithful synthesis on spans longer than 10 tokens without losing precision on short spans; BioSynNER already reports a precision drop from synthetic data, which this check must not worsen. The third is whether the gap between capped and uncapped architectures is larger under synthetic training than under real training, which would directly show that a length cap amplifies the synthetic-to-real gap.
References
- Training Deep Networks With Synthetic Data: Bridging the Reality Gap by Domain Randomization
- BioSynNER: Synthetic Data for Biomedical Named Entity Recognition
- A span-based model for extracting overlapping PICO entities from randomized controlled trial publications
- Training on Synthetic Data Beats Real Data in Multimodal Relation Extraction
- Coreference Resolution without Span Representations
- Deep Span Representations for Named Entity Recognition
- Propose-and-Refine: A Two-Stage Set Prediction Network for Nested Named Entity Recognition
- Small Language Model Makes an Effective Long Text Extractor