People describe drug side effects on Twitter in their own words: "this pill has me feeling like a zombie," "passed out cold after the second dose." Drug-safety monitoring needs three things from such a post. It must find the exact words that describe the side effect, which is span extraction. It must then map those words to a standard medical vocabulary such as MedDRA (the Medical Dictionary for Regulatory Activities), which is normalization. Current systems for this task in the SMM4H 2024 shared task send hand-written prompts to GPT-4, and they perform poorly. One reached only 0.306 F1 on span extraction, and that ceiling dragged its normalization down to 0.354. The likely cause is not missing knowledge. The models are never forced to commit to a checkable statement about what the lay phrase means clinically. They drift off the tweet's actual wording, produce verbose output, and miss MedDRA terms they were never shown.
We propose making the explanation do real work. A large teacher model writes a short structured rationale with three parts: the verbatim span, a one-line clinical restatement ("sedation; excessive sleepiness"), and the link to the drug. Each part can be checked mechanically. The span must be an exact substring of the tweet. On training data, the restatement must retrieve the gold MedDRA term from the ontology. We use these checks as the objective for automatic prompt search, which replaces manual prompt writing. We keep only teacher rationales that pass, and we distill them into a compact model. At test time, normalization sees only the span and the restatement. The explanation is therefore the actual input to the coding decision, not a justification written after the fact. The bet is that the restatement is the missing bridge between lay vocabulary and the ontology, especially for terms absent from training.
Core Claim
On SMM4H 2024 Task 1, a compact student trained to produce a verified clinical restatement before normalizing should beat two systems on normalization. The first is the best reported retrieval-based system. The second is the same student architecture trained without restatements. The gain should be largest on the subset of test terms never seen in training. Span F1 should improve more modestly, mainly because enforcing exact substrings removes the boundary drift and paraphrasing that sank GPT-4 extractors. If the restatement adds nothing over a span-only student with the same retrieval step, the claim is false. This ablation is built into the design.
The idea differs from its closest neighbors in specific ways. MentalLLaMA fine-tunes 7–13B models to co-generate labels and free-text explanations for mental-health classification. Its explanations are judged for quality but never checked or used as input to a downstream decision, and the task involves no token-level extraction or ontology mapping. The cascading hybrid in Enhancing Social Media Health Prediction Certainty uses GPT-4 chain-of-thought on low-confidence tweets but discards the reasoning. DSPy compiles pipelines and distills them into a 770M T5, but it was evaluated on clean question answering and math, where correctness is a single answer check rather than grounding plus ontology retrieval.
Research Program
The prior work falls into three roles. Some of it motivates the gap. Deep-LLMADEminer is a RoBERTa filter, then manual-prompt GPT-4 extraction, then BioBERT normalization; it identifies span extraction as the bottleneck. RIGA feeds GPT-4 candidate spans into RoBERTa and DeBERTa and reports hallucination, verbosity, and spans that deviate from the tweet text. Some of it supplies methods. Error Taxonomy-Guided Prompt Optimization clusters failed outputs into error categories and appends targeted guidance to the prompt, at about a third of the search tokens of trial-and-error methods. Specializing Smaller Language Models towards Multi-Step Reasoning shows that teacher reasoning distilled into an 11B student can rival far larger models. Some of it constrains the design. That same distillation paper reports a steep loss on out-of-domain tasks after specialization. We accept that cost: the student is meant to do one surveillance job. Separately, the frequent finding that model-written explanations can be unfaithful is the reason we route the decision through the explanation rather than generating the explanation alongside it.
The work has four stages.
Stage 1: define the rationale and its checks. The rationale format is fixed: the verbatim span(s), a clinical restatement in MedDRA-like language, and a drug link. Three checks apply to training tweets. The grounding check requires an exact substring match. The coding check requires that the gold MedDRA Preferred Term appear among the top retrieved terms when the ontology is searched with the restatement. The negative check requires that the teacher output "no ADE" on the roughly 93% of tweets without one.
Stage 2: optimize the teacher prompt. We run error-taxonomy prompt optimization on the teacher. Its objective combines grounding rate, coding hit rate, and span F1 on training folds. We expect the error taxonomy to surface concrete failure types such as paraphrased spans, merged multi-symptom spans, and confusion with disease indications. Manual prompting then becomes a measured baseline rather than the method.
Stage 3: filter and distill. We collect teacher rationales over the 18k training tweets plus unlabeled tweets, keep those that pass all checks, and fine-tune a compact decoder. The student size is fixed at one setting, around 1–3B parameters, to keep the deployment argument honest.
Stage 4: normalize at inference. We embed the restatement concatenated with the span, retrieve candidate MedDRA terms, and let the student choose among them. This keeps the label space open to unseen terms.
Baselines
The naive reference point is the manual-prompt GPT-4 extractor of Deep-LLMADEminer. It reports 0.306 test F1 on spans and 0.354 on normalization.
The strongest reported system is TLab. It uses a fine-tuned BERT-large filter, Llama 3 extraction with ten retrieved similar tweets as examples, and GPT-4 choosing among ten retrieved MedDRA terms. It reports NER F1 0.392, normalization F1 0.359, and unseen-term normalization F1 0.363, against a task median NER of 0.376. It is the right competitor because it already uses retrieval for both extraction and ontology mapping, so any gain we show must come from the restatement rather than from retrieval itself. We will reproduce its pipeline with the same embedding model and retrieval depth, and with our student substituted where model size would otherwise differ. To count as beating it, we must exceed 0.359 on overall normalization and 0.363 on unseen normalization under the official scorer, with the improvement holding across three seeds.
The supervised encoder baseline is a DeBERTa-v3 or RoBERTa-large BIO tagger (a model that labels each token as beginning, inside, or outside a span) followed by span-embedding MedDRA retrieval. It gets the same hyperparameter search budget as our student. No reported result on Lacuna was found for a standalone fine-tuned tagger on this task. RIGA's tagger is described only as "comparable to the task's baseline."
The key ablation uses the same student, trained on the same filtered teacher outputs with the restatement deleted, so it emits span only and retrieves with the span text. A second ablation keeps the restatement but drops the verification filter.
Fairness rules apply throughout. All systems use the same data splits and the same MedDRA index. Every LLM call, including teacher calls, is counted.
The official test labels may be unavailable outside the shared task. In that case we report on the 959-tweet development set, which contains only 87 ADE mentions and is therefore noisy. We supplement it with five-fold cross-validation on the training data and bootstrap confidence intervals.
Why not just…?
Why not just fine-tune an encoder tagger and retrieve with the span? This is the real threat. Exact span boundaries are what supervised taggers learn well, and 1,711 training mentions is not tiny. We expect it to match our student on span F1. Our claim rests on normalization of unseen terms. In that setting, an embedding of "feeling like a zombie" is unlikely to land near "somnolence," while a restatement can. TLab's large gap over the task median on unseen terms (0.363 vs 0.209) suggests that what the retrieval query contains matters a great deal. If the tagger-plus-retrieval baseline matches us on unseen normalization, we will report the restatement as unnecessary.
Why not just optimize the prompt and keep calling GPT-4? This may match or exceed the student in accuracy. We will measure it as an upper reference using the same optimized prompt. The comparison then becomes a cost and privacy question, as in the local-model motivation of Natural Language Explanations for Suicide Risk Classification. We consider the student successful if it stays within a few F1 points of the optimized teacher while running locally.
Why not just pseudo-label more data with the teacher, as CTYUN-AI did? CTYUN won three classification tasks this way with Qwen-72B, using only labels. The restatement-deleted ablation is exactly this alternative applied at fixed student size, so the experiment settles it directly.
Why not paraphrase the span with an LLM at retrieval time, with no training? This query-expansion variant tests whether the restatement must be learned or merely generated. We run it with the optimized teacher prompt but without verification or distillation. If it matches the full system, distillation buys only cost savings, not accuracy.
Proposed Experiments
The primary results are span F1, normalization F1, and unseen-term normalization F1 on SMM4H 2024 Task 1 for all baselines, the two ablations, and the full system. Prompt optimization is evaluated separately. We report the grounding rate and coding hit rate of manual versus optimized teacher prompts, along with search tokens spent. The goal is to quantify how much of GPT-4's 0.306 was a prompt problem.
Faithfulness check. On held-out tweets, we replace a correct restatement with a plausible wrong one ("headache" for "sedation") and measure how often the predicted MedDRA term follows the edit. A high follow rate shows that the decision actually depends on the explanation.
Stability check. Following Stability-Aware Prompt Optimization, which found flip rates up to 0.309 on formal clinical notes, we rewrite ADE tweets with slang and misspelling variants. We then measure how often the predicted MedDRA code flips, comparing the student with the tagger baseline.
Secondary test. On SMM4H Task 5, we ask whether a student distilled with restatements can remove the GPT-4 fallback for uncertain tweets. The references are Twitter-RoBERTa at 0.938 validation F1 and the hybrid system at 0.956 validation and 0.927 test F1, from Enhancing Social Media Health Prediction Certainty.
Decisive outcome. The proposal stands or falls on one comparison: the full student against its restatement-deleted twin on unseen-term normalization, with confidence intervals that do not overlap.
References
- MentalLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Models
- Enhancing Social Media Health Prediction Certainty by Integrating Large Language Models with Transformer Classifiers
- DSPy: Compiling Declarative Language Model Calls into State-of-the-Art Pipelines
- LHS712_ADENotGood at #SMM4H 2024 Task 1: Deep-LLMADEminer: A deep learning and LLM pharmacovigilance pipeline for extraction and normalization of adverse drug event mentions on Twitter
- RIGA at SMM4H-2024 Task 1: Enhancing ADE discovery with GPT-4
- Error Taxonomy-Guided Prompt Optimization
- Specializing Smaller Language Models towards Multi-Step Reasoning
- TLab at #SMM4H 2024: Retrieval-Augmented Generation for ADE Extraction and Normalization
- Natural Language Explanations for Suicide Risk Classification Using Large Language Models
- CTYUN-AI@SMM4H-2024: Knowledge Extension Makes Expert Models
- Stability-Aware Prompt Optimization for Clinical Data Abstraction