Where Prompt Brittleness Lives: Realis Ambiguity in LLM Event Extraction

When a large language model (LLM) is asked to pull events out of text, rewording the request can change its answer. The prompt-sensitivity literature usually treats these flips as noise spread evenly across the data. We propose a sharper hypothesis. For extraction and factuality tasks, much of the flipping happens on one specific kind of input: mentions of events that the text does not assert actually happened. Examples are "might have the flu," "if you get the flu," "plans to attack," and "failed to escape." Linguists call this the event's realis status, meaning whether it is presented as real, hypothetical, planned, or denied. Most instructions never say whether such events count. Each paraphrase of the instruction therefore quietly implies a different policy, and the model's decisions on modally marked mentions swing with the wording.

If this is right, part of what the field calls brittleness is underspecification that sits on a small, linguistically identifiable subset of inputs, and it can be fixed there. We will measure how much of prompt-induced variance comes from non-asserted mentions. We will then test two fixes. The first makes the realis policy explicit in the instruction. The second takes the realis decision away from the prompt entirely, handing it to a small supervised classifier that runs after a deliberately inclusive LLM extraction pass, in the spirit of the "gatekeeper" filters of classical negation work. Comparing the two tells us whether the remaining instability is a specification problem or a competence problem.

Core Claim

Two claims can be tested separately. The first is diagnostic. On benchmarks that annotate realis, the per-mention rate at which prompt paraphrases flip the model's decision will be several times higher for non-asserted mentions than for asserted ones. Non-asserted mentions will carry a share of the brittleness variance far above their roughly 25% share of mentions. This should hold even for paraphrases that never touch occurrence language. The second claim is remedial. Separating extraction from realis judgment removes most of that concentrated variance, and a supervised realis classifier does so more reliably than explicit policy wording. The reason is that LLMs are independently weak at judging event modality.

To make the first claim precise, let $f_m$ be the fraction of prompt-variant pairs on which mention $m$'s inclusion decision differs. Let $\bar f_{N}$ and $\bar f_{A}$ be its means over non-asserted and asserted mentions. Let $p_N$ be the fraction of mentions that are non-asserted. The concentration ratio is

$$C = \frac{p_N \bar f_N}{p_N \bar f_N + (1-p_N)\bar f_A} \Big/ \, p_N,$$

which equals 1 when brittleness is spread evenly and grows as flips pile up on non-asserted mentions. The diagnostic claim predicts $C$ well above 1, say above 2, and $C$ falling toward 1 under both remedies.

Grounding in Prior Work

Three papers motivate the diagnostic. Evaluating Zero-Shot Event Structures finds that roughly 25% of ACE 2005 event mentions are non-asserted. It also finds that zero-shot LLMs asked for real-world occurrences skip them, and that legacy scoring understates zero-shot performance by up to 32%. That paper responds by filtering the gold data and never varies the prompt. We treat its observation as evidence that the realis policy is prompt-dependent and measure that dependence directly.

Event Semantic Classification in Context shows that ChatGPT and GPT-3 lag far behind a supervised XLM-RoBERTa model with dictionary glosses on judging event modality. That paper used fixed prompts. We use its finding to predict that explicit policy wording will not fully close the gap. Rethinking Hallucinations shows that models with similar accuracy give contradictory answers across rephrasings, with ambiguity above 50% on Med-HALT. It does not ask which inputs drive the contradictions, and its notion of factuality is answer correctness rather than whether the text asserts an event. We localize ambiguity by linguistic category instead.

The measurement methods come from Brittlebench, which splits variance into a data-difficulty term and a prompt-brittleness term. It reports brittleness up to 50% of variance for open-weight models. Its perturbations are unstructured, and we extend its decomposition by stratifying on realis. The remedy follows Factuality Detection on the Cheap, where a binary factuality classifier placed before negated-event detection raised precision from 62.24% to 78.48% on development data and from 58.26% to 68.72% on evaluation data. That work concerned pipeline precision, not prompt variance.

Two papers constrain the design. The Possible, the Plausible, and the Desirable shows that modal triggers span open word classes and appear in 48% of sentences, which rules out a closed wordlist as the gatekeeper. Mind the Biases found that hedged prompt framing raised a fine-tuned BERT's drug-interaction F1 from 10.2% to 44.6%. That result warns that hedging in the instruction itself is a strong lever. We therefore control whether paraphrases contain modal language, and report hedged and unhedged instruction families separately.

Research Program

We first build a paraphrase suite for event extraction instructions with two families. Realis-neutral paraphrases change wording, formatting, ordering, and politeness, following the perturbation types in The Butterfly Effect of Altering Prompts and Brittlebench, and never mention occurrence. Realis-loaded paraphrases vary only the occurrence verb: "events that occurred," "events mentioned," "events reported," "events that may happen." The neutral family is the real test, because any concentration it produces cannot be blamed on intentionally changing the task.

We then run open-weight models at two scales plus one commercial model. Each model sees every variant zero-shot and with one exemplar. We compute per-mention flip rates, the concentration ratio $C$, and Brittlebench-style variance stratified by realis class. Within the non-asserted stratum, we further break down by modal sense (epistemic, deontic, planned, conditional, negated) to see which kinds drive instability.

Finally, we build the decomposed pipeline. An LLM pass uses an inclusive instruction, "list every event mention, including hypothetical, planned, or denied ones," on the hypothesis that an inclusive policy leaves nothing for paraphrases to disagree about. A fine-tuned realis classifier then labels each extracted mention. The user's policy (asserted only, or all) is applied as a deterministic filter. The classifier is XLM-RoBERTa trained on ACE realis labels and on the ESC Modality and Affirmation dimensions, using the gloss input of the ESC paper. Its behavior cannot depend on instruction wording, so any remaining brittleness has to come from the inclusive extraction step. That makes the system easy to diagnose.

Baselines

The naive reference point is the LLM with a single standard extraction prompt, scored against the asserted-only gold set recommended by the ACE paper. Its accuracy and flip statistics across our paraphrase suite define the brittleness to explain. We found no reported absolute F1 on Lacuna for zero-shot LLMs on asserted-only ACE 2005, only the up-to-32% understatement figure, so we will measure this ourselves.

The strongest competitor is explicit policy specification. The instruction defines realis in plain words and states the inclusion rule ("include only events the text presents as having actually happened; exclude plans, possibilities, conditions, and denials"), paraphrased across the same neutral family. It is the cheapest fix a practitioner would try, and it goes straight at our underspecification hypothesis. It gets the same models, variants, and decoding settings as our pipeline, and we allow comparable effort tuning the definition wording on a development split. Our pipeline beats it only if two things hold on the test split. It must lower mean flip rate on non-asserted mentions with a confidence interval that excludes zero. It must also match or exceed asserted-only F1.

The second strong baseline is prompt ensembling in the spirit of SENSEL from On the Relation between Sensitivity and Accuracy in In-Context Learning. SENSEL used cross-prompt instability as an abstention signal and gained 4.1 points of area under the F1-coverage curve over max-softmax on GPT-J-6B. We run a majority vote over k paraphrases, with k chosen so its LLM calls match or exceed our one inclusive call plus a cheap classifier.

The third baseline puts realis-labeled exemplars in a one-shot prompt. This tests the anchoring effect reported by POSIX, where one exemplar cut Llama-2-7b's sensitivity score on MMLU from 1.12 to 0.513.

Three ablations isolate the key ingredient. The first replaces the supervised classifier with the LLM asked the realis question in a second call. This separates the benefit of splitting the decision off from the benefit of supervised competence. The second replaces the classifier with a modal-auxiliary wordlist, to test whether shallow cues suffice. The third keeps the classifier but drops the inclusive first pass in favor of the standard prompt. This tests whether the inclusive instruction is what stabilizes extraction.

For the classifier itself, the reference number is the 87.10 F1 of the binary factuality classifier in Factuality Detection on the Cheap on the Conan Doyle corpus. On ESC, Event Semantic Classification in Context reports an average F1 of 0.90 across six dimensions for its gloss-augmented model. We found no reported per-dimension Modality result on Lacuna.

Why not just…?

Why not just write a clearer instruction that defines what counts as an event? This is the most serious threat. If it drives $C$ to about 1 and matches our pipeline's F1, the honest conclusion is that realis brittleness is purely a specification problem, fixed by documentation, and the gatekeeper is unnecessary. We would report exactly that. Two pieces of evidence suggest it will fall short. The ESC paper finds LLMs weak at modality judgments even with dictionary definitions in the prompt. The POSIX paper finds instruction-tuned models more sensitive to the wording of task specifications than base models, so a policy definition is itself text that paraphrases can disturb. The explicit-policy baseline, run across the neutral paraphrase family, settles the question directly.

Why not just average over many prompts? Majority voting over paraphrases hides variance but does not decide the policy. On a non-asserted mention it returns whichever reading most paraphrases happen to imply, which need not match the user's intent. It also multiplies inference cost, a burden POSIX documents for sensitivity auditing. Our equal-compute comparison and the asserted-only F1 will show whether voting matches the pipeline on correctness, not just consistency.

Why not just add a few labeled examples? POSIX and ProSA report that one exemplar stabilizes models. Brittlebench instead finds that few-shot prompting raises accuracy but increases sensitivity to formatting. When Benchmarks are Targets shows models copying surface patterns of exemplars. Our one-shot baseline with realis-labeled exemplars, crossed with the neutral paraphrase family, helps resolve this disagreement for modal inputs specifically. We also vary whether the exemplar's realis status matches the test mention.

Why not just filter sentences containing "might" or "if"? GME shows modality carried by open-class words in nearly half of all sentences. UiO 2 shows that scope resolution needs syntactic structure, with discontinuous scopes causing 34% of its errors. The wordlist ablation quantifies what a shallow filter misses, especially on implicative verbs like "failed to."

Proposed Experiments

The primary testbed is ACE 2005, whose realis annotations give the asserted/non-asserted split. Scoring uses both the full gold set and the asserted-only gold set, together with the coreference-invariant matching recommended by the ACE paper, so that annotation artifacts do not pass for brittleness. The ESC dataset tests transfer across its six semantic dimensions and to Chinese. The Conan Doyle negation corpus checks whether denied events behave like hypothetical ones or form a separate unstable stratum. Metrics are per-mention flip rate, the concentration ratio $C$, stratified brittleness variance following Brittlebench, the ambiguity measure from Rethinking Hallucinations, and trigger-level F1 under each policy.

The first decisive check is whether $C$ is substantially above 1 under realis-neutral paraphrases, for every model tested. If $C$ is near 1, brittleness is not realis-localized, and the premise fails regardless of what the remedies do. The second check is whether the inclusive first pass alone has low flip rates on both strata. That would support the idea that removing the policy decision is what stabilizes extraction. The third check compares the three ways of making the realis decision: the explicit-policy prompt, the LLM-as-classifier ablation, and the supervised gatekeeper. If only the supervised gatekeeper drives $C$ toward 1 while holding asserted-only F1, the instability is a competence gap and not merely a wording gap.

A final transfer check applies the same stratified analysis to FEVER claims rewritten under controlled modal framings. This asks whether the concentration pattern extends from event extraction to claim verification, where AuthorityBench documents framing effects of a different kind.