Familiarity-Gated Character Emissions for Few-Shot Event Detection on Informal Text

Event detection means finding the word or phrase in a sentence that signals an event, such as "protest", "strike" or "evacuated", and labeling it with an event type. In practice a new event type arrives with only a handful of labeled examples. The best few-shot detectors treat the task as sequence tagging. They compare each word against "prototypes" (average representations of the few labeled examples per type) and then use a conditional random field, a decoder that scores whole label sequences so that tags stay structurally consistent. These detectors are built and tested on clean Wikipedia- and news-style text. On social media, the trigger in a new post is often a spelling variant of the example the model was shown: "#protest", "protestin", "prtest". A subword tokenizer splits these into pieces that look nothing like the support example, so the prototype match fails before the decoder can help.

We propose adding a second, character-level matching channel to a prototypical tagger. The key design choice is that the model does not mix this channel in everywhere. A small gate decides, word by word, how much to trust characters versus contextual subwords. The gate's inputs are signals of how unfamiliar the word is to the subword encoder, such as how many pieces the tokenizer splits it into and how surprised the masked language model is by it. The claim is that, under a formal-to-informal shift, few-shot tagging errors concentrate on tokens the subword encoder fragments. A gated character channel should repair those tokens without degrading the clean tokens that plain concatenation of character features is known to hurt.

Core Claim

The closest existing system, PA-CRF, frames few-shot event detection as unified B/I/O tagging. It computes emissions as cosine similarity to support-set prototypes from BERT's final layer and infers episode-specific transition scores from prototype interactions. It reports 62.25 F1 on FewEvent 5-way 5-shot. It uses subwords only and is evaluated only in-domain. The closest work on character-word fusion with a decoder, BennettNLP's stacked Flair+BERT+GloVe+CRF, concatenates all granularities for a fixed binary label set on toxic comments, with 5,339 training posts and a static transition matrix.

We differ from both in three ways. First, we keep the channels separate and compute a prototype similarity for each. Second, the mixing weight is a per-token function of encoder unfamiliarity, rather than a fixed concatenation. Third, we evaluate in a regime neither paper touches: the detector is meta-trained on formal text, while the few support examples and the queries come from informal text.

Concretely, for word $t$ and candidate label $y$, the emission score is

$$ e_t(y) = g_t \cos(c_t, \mu^{c}_y) + (1 - g_t)\cos(h_t, \mu^{h}_y), $$

where $h_t$ is the word's contextual representation and $\mu^{h}_y$ its label prototype. Following Tuo et al., $h_t$ is a learned weighted sum of BERT layers rather than the last layer only. $c_t$ is a character-CNN encoding of the raw word string, including hashtag and elongation characters, and $\mu^{c}_y$ is its prototype from the same support set. $g_t \in [0,1]$ is the gate, a two-layer network over three inputs: the word's tokenizer fragment count, whether the whole word is in the subword vocabulary, and its masked-language-model pseudo-log-likelihood. PA-CRF's dynamic transition module and Viterbi decoding are reused unchanged on top of $e_t$.

Research Program

The work has three parts.

Constructing the evaluation. No few-shot event detection benchmark on informal text exists in the material. We therefore build one in three settings.

The first is a controlled setting. We apply realistic orthographic perturbations to FewEvent test episodes: character deletion, letter elongation, hashtag wrapping, phonetic abbreviation, and dropped punctuation. We apply them at graded rates to query sentences only, and in a second condition to both support and query. This isolates orthographic shift while holding event semantics fixed.

The second is a natural genre shift inside ACE 2005. We split the 33 event subtypes into disjoint meta-training and test type sets. Meta-training uses only newswire and broadcast news, and test episodes are drawn from weblogs, Usenet and conversational telephone speech. DAA already found these to be the dissimilar targets where adaptation matters most.

The third is genuinely noisy social media. We use the roughly 1,000 tweets with annotated event phrases from TwiCal for few-shot trigger identification (B/I/O without types), with support sets of 5 to 20 tweets.

Diagnosing the failure before fixing it. On all three settings we run unmodified PA-CRF and bucket every gold trigger by how many pieces the tokenizer splits it into. We also measure how much the inferred transition scores move between clean and perturbed versions of the same episode. Gap 2 in the source material conjectures that noisy emissions also corrupt the inferred transitions, and this measurement tests that conjecture. If errors do not concentrate in highly fragmented triggers, the gating idea has no target and we stop.

Building and dissecting the model. We then train the gated model episodically on FewEvent (or ACE formal genres) with the same backbone and episode sampler as PA-CRF. We inspect what the gate learns: whether it opens on hashtags and misspellings and stays closed on common formal words. We also measure whether cleaner prototypes reduce the transition drift found in the diagnosis.

We deliberately do not claim that characters fix PA-CRF's known failure on multi-word triggers when the support set holds only single-word triggers. That failure is a missing I- prototype, not an orthography problem. We report the multi-word bucket separately so that any effect, or the absence of one, is visible.

Baselines

The naive reference point is unmodified PA-CRF, meta-trained on formal data and applied directly to the shifted episodes. Its reported in-domain number is 62.25 F1 on FewEvent 5-way 5-shot. We reproduce it in-domain first, so its degradation under shift is measured against a verified starting point. No reported result was found on Lacuna for any few-shot event detector on informal or perturbed text, so all shifted numbers will be produced here.

The strongest existing tagging method is PA-CRF with Tuo-style weighted layer aggregation. Tuo et al. report 67.21 F1 on 10-way 10-shot FewEvent against 61.54 for the prior best, using a prototypical classifier without a decoder. Merging the two isolates what our character channel adds beyond better subword features.

AKE-BML reports 88.99 F1 on 5-way 5-shot FewEvent. However, it classifies pre-identified candidate triggers rather than tagging sequences, so its number is not comparable to the tagging F1 of PA-CRF. We will run it on our shifted episodes with gold candidate spans and report it separately as an upper reference for type classification. It will not be treated as the bar for end-to-end detection.

The most important ablation is ungated concatenation: the same character CNN, with its vector concatenated to $h_t$ before prototype computation. This is the design of BennettNLP and Plank and Nissim, transplanted into the prototypical tagger. Two further ablations isolate the gate's inputs: a fixed scalar mix with $g_t$ constant and tuned, and a gate that sees only the character vector and no familiarity signals.

All systems share BERT-base, the same episodes and five seeds, and an equal hyperparameter search budget of 20 configurations each on validation episodes from the shifted domain. The proposal counts as beating a baseline if it meets all three of the following:

  • It improves overall micro-F1 by at least 3 points over the strongest tagging baseline, with seed confidence intervals not overlapping.
  • It loses no more than 0.5 F1 on the unperturbed and newswire episodes.
  • It beats ungated concatenation specifically on low-fragmentation triggers.

Why not just…?

Why not just continue masked-language-model pre-training on unlabeled tweets? This is the real threat. Caragea et al. raised BERT accuracy on disaster tweets from 64.1% to 68.2% by continued pre-training on HurricaneExt. Pre-training of this kind might teach the subword encoder that "protestin" and "protest" are close, which would leave nothing for characters to do. We run PA-CRF on a domain-adapted BERT (adapted on unlabeled TwiCal-era tweets and ACE informal-genre text) as a baseline. We also run our gate on top of the same adapted encoder. If adaptation alone closes the gap on high-fragmentation triggers, we will report that and our contribution shrinks to a diagnostic. Our expectation is that adaptation helps frequent slang but cannot cover one-off misspellings and new hashtags, which only appear in the few support examples.

Why not just normalize the text first? Normalization commits to one standard spelling before the model sees context, and its errors are invisible downstream. TwiCal found news-trained pipelines reached only 0.44 NER F1 on tweets versus 0.67 when trained natively. The winning toxic-span system kept raw character offsets rather than normalizing them away. Since no normalizer is described in the material, we approximate one. On the controlled setting we use an oracle normalizer that inverts our own perturbations, which gives an upper bound for normalization; on the natural settings we use a dictionary-based normalizer. If even oracle normalization matches the gated model on perturbed FewEvent, the character channel is unnecessary there.

Why not just concatenate character embeddings, as prior systems do? Concatenation is simpler, and in fully supervised settings it works (Plank and Nissim reduced the out-of-vocabulary rate from 46.9% to 41.2% and reached 92.25% accuracy). With only five support examples, however, a concatenated prototype mixes character noise into every word. The material flags that character networks overfit to superficial character n-grams. The ungated ablation tests this directly, and the per-bucket analysis shows where any harm occurs.

Why not just prompt a large language model with the few examples? The material contains no reported result for prompted large language models on event detection, formal or informal. The TweetQA gap (69.6 vs 78.2 BLEU-1) concerns BERT, not prompting, so it cannot settle the question. We include a prompted model on the TwiCal and ACE informal episodes as an unbudgeted reference and report its span F1. We do not claim it is weaker.

Proposed Experiments

The first decisive check is the fragmentation-bucketed comparison on all three settings. The hypothesis predicts that the gated model's advantage over PA-CRF grows with fragment count and is near zero for whole-vocabulary words. It also predicts that ungated concatenation shows a measurable loss on those whole-vocabulary words. A flat gain across buckets would mean the improvement comes from extra capacity, not from the mechanism we claim.

The second check is a dose-response curve on perturbed FewEvent. We plot F1 against perturbation rate for PA-CRF, domain-adapted PA-CRF, oracle normalization, concatenation, and the gated model. If the idea is right, the gated model's curve should stay flat while the subword-only curves fall.

The third check measures transition drift: the change in PA-CRF's inferred transition scores between clean and perturbed copies of the same episode, with and without the character channel. This tests whether repairing emissions also stabilizes the structure the decoder relies on.

Finally, we report the gate's average value per token category: hashtags, misspellings, in-vocabulary words, and function words. We also report a support-perturbed versus query-only-perturbed comparison, which separates "noisy queries matched to a clean prototype" from "noisy prototypes."