Zero-shot event detection means finding the words in a text that signal an event, such as "fired," "seized," or "married," and labeling each with an event type, using only a written description of each type. The best current systems put a short description of each type into the model's input: its name, a definition, and a few example trigger words. That description is written in today's English. When the same system reads an 1820 novel or a 1650 pamphlet, a seed word like "engagement," "fire," or "quit" may still be common but may mean something different. A 19th-century "engagement" is as likely to be a battle or a social obligation as a betrothal. The prompt is still well formed, but it now points the model at the wrong sense.
This note proposes two things. The first is a controlled measurement of how much of the zero-shot failure on historical text comes from this kind of sense drift, as opposed to old spelling or old syntax. The second is a small, label-free fix: revise each event type's seed triggers using unlabeled text from the target era, so that drifted seeds are restricted to their intended sense and seed words native to that era are added. The fix is only worth publishing if it beats the obvious competitor, which is simply putting nearest-neighbor examples from the target era into the prompt. The experiments are designed to settle that question.
Core Claim
We claim that zero-shot event detectors lose accuracy on historical English mainly because their seed triggers have drifted in meaning. Spelling accounts for only part of the loss. If this is right, the lost accuracy should be concentrated on polysemous seed lemmas whose usage has measurably moved between the modern reference corpus and the target era. It should persist after spelling is normalized and syntax is modernized. And it should be recoverable by editing the seeds in the prompt rather than by retraining.
Two pieces are new. The first is a factorial test set that separates the three sources of drift (spelling, syntax, sense) for typed event detection. The second is a seed-revision procedure that is driven by drift scores. No existing work evaluates typed, prompt-based event detection across eras. The task description is useful only if it can be tested, so the measurement is the core of the project and the method is secondary.
Where This Sits
Three papers supply the systems we stress-test, and all three assume a timeless language.
The APEX prompt of The Art of Prompting: Event Detection based on Type Specific Prompts combines a type name, a definition, and prototypical seed triggers. It reaches 59.9% zero-shot F1 on MAVEN, and the authors explicitly credit the seeds and definitions with resolving polysemous triggers such as "get." Our hypothesis is that the same seeds become a liability across eras.
Improving Event Definition Following For Zero-Shot Event Detection fine-tunes LLaMA-2-7B to follow definitions. It reaches 28.2 average F1 across ACE05, M2E2, and MEE, against 15.6 for GPT-3.5-Turbo, and reports a drop of more than 50% when definitions are removed. That dependence on definitions is exactly what should break when a definition's key words change meaning.
Honey or Poison? frames over-reliance on trigger words as a confounding problem and corrects for it with counterfactual triggers generated by a masked language model. We take from it the warning that triggers are both signal and trap. Its correction relies on a modern masked language model, so we avoid using one to generate era replacements.
The drift literature motivates the hypothesis. ELMo and BERT in Semantic Change Detection for Russian shows that simply averaging a word's contextual embeddings into a prototype tracks human judgments of sense change (Spearman ρ = 0.557 on RuSemShift2). We reuse this averaging-prototype measure as our drift score.
Mitigating Temporal-Drift handles drift in named-entity recognition by retraining on "bursty" n-grams, meaning n-grams that suddenly rise in frequency. The authors note that this misses senses that shift quietly without any change in frequency. Our embedding-based score targets precisely that quiet case, and requires no labels.
Part-of-Speech Tagging for Historical English shows that spelling normalization helps but leaves most of the cross-era gap open. Its transfer accuracy from contemporary news to Early Modern English rises from 74.15% to 79.05% with era-aware features plus normalization. That result is a reason to control for spelling explicitly rather than assume it explains the drop.
Literary Event Detection supplies the historical text and an event layer. LitBank annotates only whether a word is an event (realis events), not which type. That is why we must add typed labels ourselves.
Research Program
Benchmark. We draw sentences from LitBank (19th and early 20th century novels) and from PPCEME and PPCMBE (Early Modern and Modern British English, 1500–1914). We oversample sentences containing lemmas that serve as seed triggers or frequent triggers in MAVEN types, and we also sample unrestricted sentences so that missed triggers can be counted. Annotators label triggers with MAVEN types. LitBank's existing realis spans serve as a detection-level check. Agreement is reported per era.
The key design element is that each historical sentence appears in three aligned versions. The first is the original. The second is spelling-normalized with VARD. The third is a human modernization that rewrites syntax and archaic function words but keeps the trigger lemma unchanged. Comparing versions one and two isolates the effect of spelling. Comparing two and three isolates syntax. Whatever error remains on version three is attributable to the trigger's sense.
Every trigger lemma is also given a drift score: the cosine distance between its averaged contextual embedding in MAVEN's Wikipedia text and in the era's unlabeled text. We additionally tag each type as trigger-dependent or context-dependent, following the distinction drawn in Jian Liu et al. 2022.
Method: drift-aware seed revision. The procedure has three steps for each event type and each target era, and it uses only unlabeled era text.
First, we score each APEX seed for drift. Second, any seed above a drift threshold is either removed or rewritten as a sense-restricted seed. To rewrite a seed, we cluster its era contexts, have the LLM write a one-line gloss of the cluster that matches the definition, and add one or two contrastive era sentences showing the wrong sense. Third, we add era-native seeds. These are lemmas from the era corpus whose contexts the base detector assigns to the type with high confidence when the candidate word itself is masked, a context-only view in the spirit of How Does Context Matter?. The new seeds therefore come from the era's own vocabulary and not from a modern model's guesses. The revised prompt stays short, in natural language, and readable, so it can be inspected and corrected by hand.
Baselines
The naive reference point is the unmodified APEX-style prompt and the unmodified definition prompt applied to each version of each sentence. This shows the raw cross-era drop and how the drop splits across spelling, syntax, and sense.
The strongest existing methods are two. One is APEX, which reports 59.9% zero-shot F1 on MAVEN in The Art of Prompting. The other is the definition-following LLaMA-2-7B of Cai et al., which reports 28.2 average F1 and 34.2 ACE05 trigger-identification F1. We found no reported result on Lacuna for typed zero-shot event detection on historical English, and none for MAVEN-typed detection on LitBank. The only historical event number in the file is 73.9 realis F1 for a supervised BiLSTM+BERT model on LitBank (Sims et al.), which serves as a detection-level ceiling.
Fairness rules apply to every comparison. All prompt variants use the same frozen backbone, the same decoding settings, and the same prompt-length budget measured in tokens. Any threshold is tuned on a small held-out era development split that is shared by all methods. The definition-following model receives the same revised seeds and definitions as APEX.
Two ablations isolate the key ingredient. The first removes drifted seeds without adding any. The second adds era-native seeds while keeping all the original seeds. A further control replaces the drift ranking with a random choice of which seeds to revise, holding the number of revisions fixed. This tests whether the drift score itself matters.
Beating the baselines means the following. The revised prompt must improve typed F1 on the original historical text over both unmodified systems. Most of that gain must fall in the top drift tertile. And it must exceed the retrieved-exemplar baseline described below by a margin larger than the seed-to-seed spread across five prompt paraphrases.
Why not just…?
Why not just normalize the spelling? Eisenstein et al. found that normalization still leaves Early Modern tagging near 79% against 97% on modern text. Even so, event typing could behave differently from part-of-speech tagging. Our version-two condition measures this directly. If F1 on normalized text approaches F1 on modernized text, sense drift is not the bottleneck and our claim fails.
Why not just use a bigger general LLM? Cai et al. report GPT-3.5-Turbo at 15.6 average F1 against 28.2 for a 7B model trained to follow definitions, so scale alone does not guarantee that definitions are followed. Still, a newer and larger model may know historical senses well. We therefore run seed revision on top of the largest open model we can afford. The measure of interest is whether the gain on high-drift triggers survives at that scale.
Why not just put nearest-neighbor era sentences in the prompt as few-shot examples? This is the real threat. Exemplars drawn from the target era might carry the right senses implicitly. We test it in two forms with the same token budget. The first is unlabeled retrieved sentences. The second is pseudo-labeled exemplars from the base detector. If either matches seed revision on the high-drift tertile, the honest conclusion is that grounding the prompt in era text is what helps, not drift-targeted seed editing. In that case the contribution narrows to the diagnostic benchmark and the attribution of errors to sense drift.
Why not just annotate a few era examples and fine-tune? Trend-based retraining helped named-entity recognition by 2.70–5.71 F1 (Chen et al.), but it requires labels at every time step, and MAVEN has 168 types. We report a learning curve with 10, 25, and 50 labeled era examples per type, using the same backbone fine-tuned lightly. This gives the label count at which supervision overtakes label-free revision.
Decisive Checks
The first check concerns where the loss comes from. Zero-shot F1 should fall from the modern MAVEN test set to modernized historical sentences, and that fall should rise with trigger drift score. A flat relationship would refute the sense-drift hypothesis, whatever the method results show.
The second check is a within-lemma comparison. For polysemous lemmas such as "engagement," "fire," and "quit," F1 should be lower in the era sense clusters that are far from the modern prototype, even though the surface word is identical.
The third check concerns the fix. Seed revision should beat both the unmodified prompts and retrieved era exemplars on the original historical text, with the gain concentrated in the high-drift tertile and in trigger-dependent types.
The fourth check concerns structure. Context-only masking should help less on Early Modern text than on 19th-century text, because modern encoders handle older syntax poorly. This would show the masked-candidate step is reliable only where syntax is close to modern, and that result bounds how far back in time the method can go.
References
- The Art of Prompting: Event Detection based on Type Specific Prompts
- Improving Event Definition Following For Zero-Shot Event Detection
- Honey or Poison? Solving the Trigger Curse in Few-shot Event Detection via Causal Intervention
- ELMo and BERT in Semantic Change Detection for Russian
- Mitigating Temporal-Drift: A Simple Approach to Keep NER Models Crisp
- Part-of-Speech Tagging for Historical English
- Literary Event Detection
- Saliency as Evidence: Event Detection with Trigger Saliency Attribution
- How Does Context Matter? On the Robustness of Event Detection with Context-Selective Mask Generalization