Topic-Blind Ensemble Members for Cross-Corpus Narrative Detection

Detecting when someone is telling their own story ("my husband hit me last night") rather than reporting news, giving advice, or arguing about the same subject matters for public-health monitoring, harassment research, and the study of persuasion. The difficulty is that the topic words are identical across these kinds of text. What differs is grammar: first-person singular pronouns, past-tense action verbs, and concrete event sequences mark a story, while second-person pronouns, modal and future verbs, and nominalizations mark advice or reporting. Most systems ignore this. They fine-tune a pretrained transformer on one corpus and sometimes average several random seeds of it. Each paper reports its numbers on its own corpus, with its own preprocessing and its own way of choosing models, so nobody knows whether those gains survive a move to a new community or a new topic.

We propose a controlled cross-corpus test of one specific idea. An ensemble — several classifiers whose predictions are combined — helps only when its members make different mistakes. A member that has been made blind to topic, because content words are replaced by grammatical tags before it sees the text, should make mistakes that are unlike a standard transformer's, and this difference should matter most on unseen corpora, where topic cues mislead. In short, a cheap grammar-only model paired with a fine-tuned transformer may generalize to new communities better than five copies of the transformer, provided all systems are compared under one fixed protocol.

Core Claim

The claim has three parts. First, a fine-tuned transformer that leaves out one corpus at training time makes errors driven largely by topic: false positives on news and advocacy, and misses on stories told in unfamiliar vocabulary. Second, a "delexicalized" member, meaning a model trained on text where every content word is replaced by its part-of-speech tag while pronouns, auxiliaries, tense morphology, and modals are kept, has errors that correlate weakly with the transformer's on held-out corpora. Third, simply averaging the two beats a seed ensemble of equal or greater inference cost. The new element is the source of diversity. Earlier narrative ensembles drew diversity from random seeds. Our proposal draws it from a model built so that it cannot see topic. The test is run under a protocol designed to remove the selection leaks that the benchmarking literature has shown can manufacture apparent progress.

The mechanism follows from a standard identity for averaged regressors (the ambiguity decomposition):

$$ (\bar f - y)^2 = \frac{1}{M}\sum_{m=1}^{M}(f_m - y)^2 - \frac{1}{M}\sum_{m=1}^{M}(f_m - \bar f)^2 , $$

where $f_m$ is member $m$'s predicted probability, $\bar f$ is their mean over $M$ members, and $y$ is the label. The averaged ensemble beats its average member by exactly the members' spread. Seeds of one pretrained model share most of their inductive bias, so their spread should be small. We predict the spread between a lexical and a delexicalized member is much larger, specifically on out-of-domain items. This prediction can be measured directly and could be falsified.

Why This Gap Is Real

The narrative work in the file motivates the question but never runs the comparison. AIR-JPMC@SMM4H'22 ensembled five RoBERTa-large checkpoints for self-reported intimate partner violence and found that majority vote and F1-weighted averaging gave identical test results. That is what one would expect if seed members barely disagree. The authors also named part-of-speech distributions and narrative voice as the cues that separate self-reports from news, but they used those cues only in error analysis. Narrative Detection in Online Patient Communities found that its ensemble raised precision but not F1 over a character 3-gram linear classifier. It also documented the marker split we exploit: "my" and past tense for narratives versus "you", "will", and "may" for non-narratives, with "we" behaving unlike "my". Our feature design keeps pronoun person and number distinct for this reason. Reports of personal experiences and stories in argumentation found a shared "conceptual core" of narrative markers across RegRoom, Change My View, and Europolis, and showed that one BERT model trained on all three beats in-domain models. That result supports the premise that grammar transfers across corpora. It also supplies our strongest single-model competitor, and the paper never tried ensembling. Where Do People Tell Stories Online? (StorySeeker) found fine-tuned RoBERTa ahead of GPT-4 and confirmed the same pronoun and tense markers at the span level, but did not combine any systems.

The benchmarking literature constrains how we run the study. Revisiting Point Cloud Shape Classification showed that apparent architectural gains came from training recipes and from picking the best epoch on the test set. Under one protocol, a small baseline matched far larger models (SimpleView 93.0% on ModelNet40 against DGCNN's 91.9%). The S-Space Package showed that once preprocessing was standardized, no single model won everywhere. FusionBench built a common testbed for ensembling, merging, and mixture-of-experts methods, but only on GLUE-style and reasoning tasks, and it never asks whether diversity built on linguistic grounds helps. We borrow its discipline of fixed task pools and fixed fusion code, and we apply it to a problem with known structure. Finally, the ensembles in DialDoc21 (RoBERTa plus ELECTRA, reported as more than 4 points better on unseen domains) and BioLaySumm (truncation versus chunking views) show that deliberately engineered diversity pays off. Neither asks which kind of diversity matters, and that question is ours.

Research Program

Benchmark. We would first build the benchmark. It harmonizes the publicly released corpora in the file into a binary label: "contains a first-hand personal experience narrative." The corpora are StorySeeker (document level), the GIST forum posts, RegRoom/CMV/Europolis, StoryARG spans lifted to document level, the breast-cancer nonprofit posts of Narrative Detection and Feature Analysis in Online Health Communities, the narrativity annotations from Narrative Style and the Spread of Health Misinformation on Twitter, the COVID-19 test set of Stories and Personal Experiences in the COVID-19 Discourse, and the SMM4H 2022 intimate partner violence tweets.

Some definitions differ in real ways. "Story" is not the same as "testimony", and a self-reported violence event is narrower than either. We therefore keep each corpus's native label as a second evaluation target and report results on both.

The protocol fixes four things:

  • Preprocessing: one shared pipeline for every corpus.
  • Validation: leave-one-corpus-out, with model selection and early stopping done by inner cross-validation on the training corpora only.
  • Test access: each held-out corpus is scored exactly once.
  • Budgets: an identical hyperparameter-search budget for every method.

Delexicalized member. We would then build the delexicalized member in two forms. One is a linear model over counts of pronoun person and number, tense, aspect, modals, and part-of-speech n-grams. The other is a small transformer fine-tuned on the tag-substituted text, which can learn sequential patterns such as a first-person subject followed by a past-tense verb. Both are topic-blind by construction.

Analysis. We would measure pairwise error correlation and the spread term above, both in-domain and out-of-domain. We would then test whether plain probability averaging captures the gain. We would also try a small gate that weights members by the text's marker profile, but only as an ablation: if averaging matches the gate, the claim rests on averaging.

Baselines

Naive reference points. The do-nothing reference is the majority-class predictor together with a hand rule that fires when first-person-singular pronouns co-occur with past-tense verbs. Any learned system must clear the rule by a clear margin, or the task is mostly grammar plus noise.

Lexical baseline. This is the character 3-gram linear classifier from Narrative Detection in Online Patient Communities, which reports F1 0.815 on GIST. It is the best evidence that shallow models compete on short, noisy text.

Strongest single model. This is a pooled RoBERTa-large trained on all non-held-out corpora, the analogue of BERT-mixed in Reports of personal experiences and stories in argumentation. That paper reports macro F1 of 0.76 on Europolis, 0.85 on CMV, and 0.94 on RegRoom, though under in-sample corpus mixing rather than leave-one-corpus-out.

Strongest existing ensemble. This is a five-seed RoBERTa-large ensemble as in AIR-JPMC@SMM4H'22, which reports test F1 0.851 on SMM4H 2022 Task 8 against an organizer baseline of 0.756. We reproduce it on that split for calibration, then run it under our protocol.

LLM comparison. We include a frontier LLM prompted zero-shot with a fixed, preregistered prompt and fixed decoding settings. StorySeeker reports GPT-4 at about 0.79 document F1 against RoBERTa's 0.86. No reported leave-one-corpus-out result was found on Lacuna for any of these systems.

Fairness and the success bar. All learned systems get the same training data per fold and the same tuning trials. We compare ensembles at matched inference cost: one RoBERTa-large plus the delexicalized member costs well under two RoBERTa-large forward passes, so the seed-ensemble comparison is generous to the seed ensemble. Beating the baseline means a higher held-out macro F1 than the five-seed ensemble, averaged over held-out corpora with a paired bootstrap interval excluding zero, and a lower false-positive rate on topic-matched non-narratives.

Ablations. Two ablations isolate the key ingredient. The first is a lexicalized heterogeneous pair, RoBERTa plus ELECTRA in the style of DialDoc21, which tests whether any architectural diversity suffices. The second is RoBERTa plus a model trained on randomly masked words at the same masking rate, which tests whether topic-blindness specifically, rather than generic input corruption, drives the gain.

Why not just…?

Why not just train one pooled model on everything? This is the most serious threat. BERT-mixed already showed that pooling corpora helps, and with enough corpora pooling might teach the transformer to ignore topic on its own. Our leave-one-corpus-out comparison settles it directly. If the pooled model's held-out errors already correlate weakly with the delexicalized member's, the spread term will be small and the ensemble gain will vanish. We would then report that pooling makes explicit marker models unnecessary, which is a useful negative result.

Why not just add more seeds? The identical results of majority vote and weighted voting in AIR-JPMC@SMM4H'22 suggest that seed members agree almost everywhere, so extra seeds should add little. We test this by plotting held-out F1 against ensemble size for seeds, at up to ten members, against the two-member heterogeneous pair.

Why not just use a strong lexical classifier, or an LLM? The character n-gram model is cheap and strong on GIST. However, Narrative Detection and Feature Analysis in Online Health Communities reports SVM at 0.901 against BERT at 0.988, and the lexical model is itself topic-sensitive. We rerun both under one protocol to see whether that gap survives. The LLM avoids training altogether, but StorySeeker reports both a lower score and over-prediction under chain-of-thought prompting. We include the LLM as a member candidate as well as a competitor, since its errors may also be decorrelated from the fine-tuned models'.

Proposed Experiments

The decisive experiment is the leave-one-corpus-out table. For each held-out corpus, it reports macro F1 for the pooled RoBERTa, the five-seed ensemble, the delexicalized member alone, the RoBERTa-plus-delexicalized average, and the two ablation pairs. Alongside each pair, we report the measured error correlation and spread term. The hypothesis predicts three things: spread larger out-of-domain than in-domain, larger for the delexicalized pair than for the seed or lexicalized pairs, and ensemble gains that track that spread across corpora.

A second check targets topic confusion. We take negatives that share topic keywords with positives, such as violence news and advocacy tweets in SMM4H and advice replies in GIST and StorySeeker, and compare false-positive rates across systems.

A third check reruns the published in-domain comparisons (lexical against transformer, ensemble against single model) under our protocol. This shows how much of each reported gap survives once selection leaks are removed, as the point-cloud study did for its field.

Finally, on StoryARG and GIST, where annotator inconsistency was documented, we test one secondary question: whether disagreement between the topic-blind and lexical members flags the items that human annotators disputed. A positive answer would give a cheap, inspectable signal for auditing labels.