Small, specialized models can beat frontier models on expert tasks once they are fine-tuned on good synthetic data. A 1.6B biomedical model trained on GPT-4-written question–answer pairs outperforms few-shot GPT-4 on biomedical question answering. Producing that data is the expensive part. Pipelines either pay for the strongest available teacher model, or they let a cheaper teacher generate freely and then throw away most of its output with costly verification models. We propose a third route. First, break the domain's vocabulary into a small, closed set of atomic meaning units: shared conceptual building blocks, so that "nephrotoxicity" becomes something like {kidney, damage, drug-caused}. Then use those units to plan each synthetic example before the teacher writes it, and to check it afterward with a cheap tagger rather than a large verifier.
The claim is that planning in a closed, corpus-wide inventory of meaning units makes generation cheaper and more reliable at the same time. Because every passage and every generated question is described in the same small vocabulary of units, we can see which concepts and concept combinations the synthetic set already covers. We can spend the teacher's budget on the rare ones, and we can reject off-target generations for almost no compute. If this works, a weaker or open teacher should reach the student accuracy that currently requires GPT-4, and it should do so with a much smaller fraction of generations discarded.
Where the Pieces Come From
The meaning-unit idea comes from sememe research. Sememes are the smallest indivisible units of meaning, catalogued for Chinese and English words in the HowNet lexicon. Improved Word Representation Learning with Sememes builds stable vectors for rare words by composing the vectors of their frequent sememes. Enhancing Transformer with Sememe Knowledge injects and predicts sememes inside a Transformer. That lowers perplexity from 49.01 to 46.53 on People's Daily and cuts synonym-attack error on LCQMC from 17.6% to 15.2%. Its ablation also shows sememes beating character-level splitting. TKDP injects HowNet sememes into prompt tuning for few-shot entity recognition, including on the biomedical JNLPBA set. These papers supply the method: shared atomic units carry information to rare items. They also expose the constraint we must solve. HowNet was built for everyday language and does not cover biomedical jargon, and no prior sememe work generates training data.
The cost problem is documented in a second line of work. Promptagator runs a 137B teacher over whole corpora and then a round-trip filter that discards queries failing to retrieve their source. On the small NFCorpus collection that filter over-pruned. UDAPDR generates a million queries with Flan-T5 XXL and distills an ensemble of cross-encoders. DISCO pairs GPT-3 generation with a DeBERTa filter, and shows that 75,000 targeted examples beat over a million noisy ones. DISCO is the closest methodological relative: it also edits at the level of units, but its units are syntactic phrase chunks rather than meaning units. All of these pipelines generate first and verify afterward, and none of them controls which concepts a generation exercises.
The domain gap supplies the application and the central tension. PubMedQA shows that biomedical research questions mostly require quantitative reasoning and group comparisons. On that benchmark BioBERT reaches 68.1%, against 78.0% for human experts. Dr. LLaMA raises a LoRA-tuned BioGPT-Large to 75.4% with GPT-4-augmented data. It also found that when GPT-3.5 wrote entirely new question–answer pairs, the student got worse, because the generated clinical logic was shallow or wrong. This finding motivates the proposal. Our hypothesis is that much of the weak-teacher failure comes from the teacher choosing what to ask about, rather than from its inability to write a correct question once the target concepts are fixed. KMQA constrains us from the other side. Injecting medical knowledge-graph triples helps, but 44% of its remaining errors come from facts missing from the graph, which warns against depending on a curated resource. TheoremQA adds that pasting a definition into the prompt does not help models use it. Our plans therefore shape training examples. They are not extra prompt text at test time.
Core Claim
A closed inventory of domain meaning units, induced from the target corpus and anchored to general sememes where possible, can serve as a planning language for synthetic data. We expect plan-conditioned generation with a cheap unit-tagger check to give higher student accuracy per teacher token than either free generation with heavy filtering or a stronger teacher used without plans. We also expect it to stop the weak-teacher degradation that Dr. LLaMA reported.
Research Program
The first component induces the inventory. We collect domain terms from the unlabeled PubMedQA abstracts (the PQA-U split, 61.2k) and compute their contextual embeddings with a biomedical encoder. We then learn a sparse dictionary of roughly one to two thousand atoms, so that each term is a sparse combination of atoms. To keep the inventory connected to curated meaning, the English side of HowNet initializes the general atoms (human, cause, increase, organ), and the remaining atoms are free to capture domain primitives. A small tagger, a fine-tuned encoder of the student's size, learns to predict the unit set of any sentence. Its reliability is measured on held-out terms, and a domain expert spot-checks a sample of units for interpretability.
The second component plans and generates. Each abstract is tagged with its unit set. A plan is a small subset of those units together with a question type (a comparison between groups, a direction of effect, a dose condition), and the teacher is asked to write a yes/no/maybe question whose answer depends on exactly those units. Plans are sampled to favor combinations that are under-represented in the synthetic set so far. If $n(S)$ is the number of accepted examples whose plan contains unit pair $S$, a candidate plan $P$ is drawn with probability proportional to
$$w(P) = \sum_{S \subset P,\ |S|=2} \frac{1}{\sqrt{1 + n(S)}},$$
so rare pairs gain weight and saturated ones fade. The square root is a conventional choice that damps the weighting without ignoring counts, and we treat the exponent as a tuned hyperparameter. This targets the diversity plateau reported in Beyond Quantity.
The third component verifies cheaply. The tagger checks whether the generated question and answer rationale realize the planned units. A generation is rejected if it misses planned units or introduces units absent from the source abstract, the second case being a cheap proxy for hallucinated content. This check costs one small-encoder forward pass, not a retriever retraining or a cross-encoder ensemble. The student is BioGPT-Large with LoRA, matching Dr. LLaMA. Throughout, we log teacher tokens, verifier compute, and the discard rate, alongside accuracy.
Baselines
The naive reference point is the same BioGPT-Large LoRA student trained only on the 450 labeled training questions, with no augmentation. Two published floors sit alongside it: the 55.2% majority-guess baseline and BioBERT's 68.1%, both from PubMedQA. Beating these shows only that augmentation helps at all.
The strongest existing method is Dr. LLaMA's GPT-4 augmentation, which reports 75.4% accuracy on PubMedQA with BioGPT-Large and LoRA, against 74.4% for few-shot GPT-4 Dr. LLaMA. We rerun it with the same student, LoRA rank, training steps, and hyperparameter search budget as our method. "Beating it" has two possible meanings, and we will report which one holds. The strong version is matching or exceeding 75.4% while using a weaker or open teacher (GPT-3.5 or Flan-T5 XXL, the teacher UDAPDR used). The weaker version is matching it with GPT-4 at substantially fewer teacher tokens.
The cost-matched competitor is free generation by the same weak teacher, followed by a strong verifier: a fine-tuned DeBERTa answer-consistency filter in the style of DISCO, or round-trip retrieval in the style of Promptagator. Total compute, meaning teacher plus verifier, is equalized with our pipeline. Our method must win on accuracy at equal total compute, or tie at clearly lower compute.
The most dangerous competitor is LLM-generated atomic-fact plans. Here the teacher itself lists the key propositions of each abstract, then writes a question about some of them. This tests whether a closed, corpus-wide inventory matters, or whether any decomposition does. It is run at an equal teacher-token budget, so the cost of the fact-listing pass counts against it.
Three ablations isolate the ingredients. Plans built from random units of the source abstract, without coverage weighting, isolate the scheduling. Plan-conditioned generation without the tagger check isolates verification. A HowNet-only inventory versus the induced inventory isolates domain unit discovery.
Why not just…?
Why not just use GPT-4 as the teacher and stop worrying about cost? For PubMedQA this already works at 75.4%. The proposal targets the setting where a GPT-4-scale teacher is not affordable at corpus scale, which Unleashing Reasoning Capability of LLMs via Scalable Question Synthesis from Scratch and LMGQS describe as a real barrier. We will also run our method with GPT-4. If plans give no gain there and no rescue for weaker teachers, the idea has no use case, and we will report it that way.
Why not just generate far more data with a cheap teacher and filter hard? DISCO's 75k-versus-1M result and Promptagator's over-pruning on NFCorpus suggest that volume plus filtering wastes compute and can discard the borderline examples small domains need. This alternative is a real threat, so it is a compute-matched baseline. The discard-rate and accuracy-per-compute curves will settle it.
Why not just retrieve definitions or knowledge-graph facts into the prompt at test time? TheoremQA found that theorem definitions in the prompt gave no significant gain, and KMQA's errors are dominated by missing coverage and retrieval noise. We include a retrieval-augmented student as a check, but we expect it to address a different failure than missing training signal.
Why not just let the LLM decompose each passage into atomic facts? This is the strongest alternative. It needs no inventory induction and its units are richer. Its weaknesses are cost and comparability. It spends teacher tokens on every passage, and its facts are free text, so they cannot be counted across the corpus for coverage scheduling or checked by a small tagger. If it beats our method at equal teacher tokens, the closed-inventory claim is false and we will say so.
Why not just rely on subword tokens, which already split rare words? The character-versus-sememe ablation in Enhancing Transformer with Sememe Knowledge shows that meaning units outperform surface splits. More fundamentally, subword fragments name no concepts, so they cannot express a plan.
Proposed Experiments
The primary experiment is on PubMedQA, using the expert-labeled split for evaluation and the unlabeled abstracts as generation sources. We sweep teacher budgets and teacher strengths (GPT-4, GPT-3.5, Flan-T5 XXL) and plot student accuracy against teacher tokens and against total compute.
There are three decisive checks. First, with the weak teacher, does plan-conditioned generation improve over the no-augmentation student where Dr. LLaMA's free generation degraded it? Second, at matched compute, does it beat free generation followed by heavy filtering? Third, does it beat LLM fact-planning at matched teacher tokens?
Two diagnostics accompany these checks. We measure the tagger's agreement with expert judgments of whether a question really depends on its planned concepts. We also measure unit-pair coverage of the synthetic set over the course of generation.
A secondary transfer test applies the same planning to query generation for NFCorpus and SciFact from BEIR, the setting where Promptagator's filter over-pruned. Promptagator reports 47.8 average nDCG@10 across 11 tasks with 8 examples. No per-dataset result for NFCorpus or SciFact was found on Lacuna, so we will rerun the Promptagator-style baseline ourselves under our compute budget.
The idea fails if induced units are not interpretable or not taggable reliably. It also fails if plans do not rescue weak teachers, or if free-text fact planning matches it at equal cost. Each of these outcomes is measured directly above.
References
- Improved Word Representation Learning with Sememes
- Enhancing Transformer with Sememe Knowledge
- TKDP: Threefold Knowledge-Enriched Deep Prompt Tuning for Few-Shot Named Entity Recognition
- Promptagator: Few-shot Dense Retrieval From 8 Examples
- UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers
- DISCO: Distilling Counterfactuals with Large Language Models
- PubMedQA: A Dataset for Biomedical Research Question Answering
- Dr. LLaMA: Improving Small Language Models on PubMedQA via Generative Data Augmentation
- Towards Medical Machine Reading Comprehension with Structural Knowledge and Plain Text
- TheoremQA: A Theorem-driven Question Answering Dataset
- Beyond Quantity: Trajectory Diversity Scaling for Code Agents
- Unleashing Reasoning Capability of LLMs via Scalable Question Synthesis from Scratch
- LMGQS: A Large-scale Dataset for Query-focused Summarization