A classifier that sorts disaster tweets into categories such as "people trapped," "infrastructure damage," or "irrelevant" is always trained on past events and then deployed on the next one, which is often a different kind of disaster. Its hyperparameters (learning rate, regularization strength, class weighting, how hard it is pushed toward event-invariant features) are still chosen by checking accuracy on held-out posts from the same past events. This proposal argues that this choice is a hidden source of cross-event failure. The fix is a model-selection rule built from two parts. The first measures validation performance separately on posts whose vocabulary looks unlike anything in training. The second combines those measurements with an anchored version of TOPSIS, a decision method that ranks options by how close they sit to a best case and how far they sit from a worst case.
The two-sentence version is as follows. When tuning a crisis classifier, score each hyperparameter setting by how it does on the "vocabulary-gap" slice of validation data, meaning posts sharing few words with the training events, and on rare life-critical classes, not by overall validation accuracy. Then pick the setting that is closest to an ideal and farthest from the untuned default. We test whether this rule picks configurations that transfer better to unseen disaster types, including sudden events like explosions and attacks, than averaging or worst-case selection.
Core Claim
Cross-event degradation in crisis text classification is usually attributed to the model or the data. Methods attack it through representation learning: adversarial event-invariance in MT-DAAN, hidden-layer interpolation in Manifold Mixup for crisis tweets, class-balanced pseudo-labeling in DeCrisisMB, and physical weather grounding in CrisisTS. Each of these papers fixes its hyperparameters heuristically or through small preliminary runs.
Wiemerslage et al. show that architecture rankings flip once both sides are properly tuned. In their study, LSTMs match or beat Transformers on character transduction. Verma and Bergler find that a batch-size change alone moved macro-F1 by at least 3.2 points, nearly erasing the gap between two modeling approaches. These results motivate the claim that much of the reported cross-event gap may come from selection, and they suggest the right question is not "which method" but "which configuration of a method, chosen how."
Our claim is narrow and testable. Selecting hyperparameters by performance on low-vocabulary-overlap validation posts and on critical-class recall predicts held-out-type performance better than selecting by overall validation score. Combining those criteria with TOPSIS anchored to fixed reference points gives selections that are lower-regret and more stable than mean, min-max, or Tchebycheff aggregation. The lexical-gap stratification is the main bet. The anchored TOPSIS aggregator is a secondary contribution, and it must earn its place against simpler rules in the experiments.
Why TOPSIS, and Why It Must Be Modified
TOPSIS normalizes each criterion, weights it, and locates a positive ideal point (the best observed value on every criterion) and a negative ideal point (the worst observed). It then scores each option by its relative closeness to the positive ideal versus the negative one. It has been used for life-critical allocation, for example in combining community and societal risk for COVID-19 vaccine targeting in Chen et al.. Schöffer et al. reframed fair classification as distance to a "North Star" candidate using only a positive ideal. Neither paper applies ideal-point ranking to the space of model configurations. We supply that application.
Classic TOPSIS has a known defect that matters acutely in hyperparameter search: rank reversal. Because both ideal points are computed from the current set of options, adding more random-search trials can flip which configuration wins, even though no configuration's scores changed. We therefore anchor both points outside the candidate pool. The negative ideal $a^-$ is the score vector of the do-nothing reference, meaning the model trained with library-default hyperparameters. The positive ideal $a^+$ is a fixed target, either the in-event score ceiling or 1.0 per criterion. For a configuration with weighted, normalized criterion vector $v$, the score is
$$C(v) = \frac{\lVert v - a^- \rVert}{\lVert v - a^+ \rVert + \lVert v - a^- \rVert},$$
where both distances are Euclidean. Because the anchors do not depend on which other configurations were tried, the ranking of any two configurations is invariant to pool growth. This is a property we will verify empirically.
The anchored score asks something different from the worst-case rule. A configuration that is mediocre everywhere but never worse than the default can outrank one that is excellent on common classes and collapses on rare ones. That behavior is what we want when the rare classes are "search and rescue."
Research Program
The criteria come from a leave-one-event-type-out protocol. For each configuration we train on all source event types and validate on held-out posts from those sources. We then compute four scores. The first is macro-F1 on the low-overlap stratum, the second is recall on the life-critical classes, the third is macro-F1 on the high-overlap stratum, and the fourth is inference latency.
The low-overlap stratum is defined per post. It contains posts whose content-word overlap with the training-event vocabulary falls in the bottom tercile. An alternative definition uses the maximum BM25 similarity of the post to any training post, and we test both. The life-critical classes include injured or dead people, requests or urgent needs, and search and rescue where the label set has it.
The idea behind the low-overlap stratum comes from retrieval. Many correct passages share no content words with the question they answer, the core finding behind query-rewriting work such as RaFe and InteR. The low-overlap stratum is an in-distribution proxy for that out-of-event lexical gap. CMU-Informedia at TREC-IS names exactly this gap ("rising water" versus "aftershocks") as what limited their transfer.
The hyperparameter pool is a fixed random search of 200 configurations per model family. We use two families. One is a fine-tuned multilingual encoder, following Chowdhury et al., with mixup strength and mixup layer as searchable knobs. The other is the same encoder with a gradient-reversal domain head, following MT-DAAN, with the reversal weight searchable. Both families also search learning rate, epochs, dropout, and class-weight exponent. Every selection rule picks from the identical pool, so the comparison is purely about selection.
We also analyze when the rule should fail. Overtuning in Hyperparameter Optimization warns that aggressive selection overfits finite validation sets. The low-overlap stratum is small by construction, so we report bootstrap confidence intervals on every criterion and test a shrinkage variant that blends the stratum score toward the overall score according to stratum size. Schöffer et al. found that ideal-point ranking can sharply cut ordinary accuracy, from 78.5% to 56% on German Credit. That result constrains our design: we will report the high-overlap and overall scores of the selected configuration so that any such sacrifice is visible.
Baselines
The do-nothing reference is the library-default configuration with no search. It is also our negative anchor, so every rule must at least beat it.
The standard practice is selecting the configuration with the best overall validation macro-F1 on source events. This is what existing crisis papers approximate. It is the most important baseline because it is cheap and nearly universal.
The averaging competitor is the mean of leave-one-source-event-out validation scores. It uses event structure without any lexical stratification.
The strongest principled competitor is worst-case selection, which takes the configuration maximizing the minimum score across source events or across our four criteria. This follows the finding in MORBiT that worst-case objectives generalize better to unseen tasks than average-case ones. MORBiT itself is a bilevel gradient method. Here we apply its min-max objective as a selection rule over the shared pool, which isolates the objective from the optimizer.
We also include Tchebycheff scalarization and classic unanchored TOPSIS, both described in the multi-objective HPO overview. All rules receive the same 200 trained models, the same validation data, and the same criterion weights. Weights are equal by default, with a sensitivity sweep.
Two ablations isolate the key ingredients. The first runs anchored TOPSIS on overall-only criteria, with no lexical stratum, to isolate stratification. The second runs the mean over the stratified criteria, to isolate the aggregator.
Beating the field means lower median regret than both overall-F1 selection and worst-case selection across held-out types, with a bootstrap interval excluding zero. Regret is the gap between the held-out-type macro-F1 of the selected configuration and that of the oracle best configuration in the pool.
For absolute reference, published systems report the following numbers, though not under our exact splits. CrisisTS reports a French utility-classification F-score of 59.20 for text only and 75.03 with weather fusion (Meunier et al.). Chowdhury et al. report F1 of roughly 79 to 87 on unseen events in CrisisNLP and CrisisLex. MT-DAAN reports 73.29% Factoid accuracy on TREC-IS. No reported result was found on Lacuna for selection regret on any crisis benchmark.
Why not just…?
Why not just average validation scores across held-out source events? This is the most plausible threat, because with equal weights and few criteria, TOPSIS and the mean often agree. The second ablation, the mean over stratified criteria, addresses it directly. If that ablation matches anchored TOPSIS, we will report that the gain comes from the lexical stratum and recommend the mean as the simpler tool. The aggregator claim would be dropped rather than defended.
Why not just use worst-case (min-max) selection, as MORBiT suggests? MORBiT shows worst-case optimization helps unseen tasks, but only on Letter and FashionMNIST, with tasks of similar difficulty. With seven crisis types of very different difficulty, the minimum is dominated by whichever source event is intrinsically hardest, so the min-max rule may chase noise. The experiment settles this by comparing regret and also selection stability across seeds. We predict min-max will show higher variance in which configuration it picks.
Why not just close the vocabulary gap with query or text rewriting instead of better selection? Rewriting in the style of RaFe or InteR is complementary, not competing. It changes the input, while we change which model is kept. The two also carry costs. RaFe's rewriter inherits its teacher reranker's blind spots, and InteR's multi-round LLM loop is too slow for sub-second triage. We will run one secondary experiment that adds an LLM paraphrase of each test post to the pipeline. The question is whether our selection gain persists on top of it or is absorbed by it.
Why not just use physical grounding as CrisisTS does? CrisisTS itself reports that weather signals help only slow-onset events and give no signal for attacks, explosions, or collapses. Those sudden types are where we concentrate our hardest held-out evaluations.
Proposed Experiments
The primary benchmark is CrisisTS, with its 29 events across seven types (fire, flood, storm, hurricane, collapse, explosion, attack). We use the text-only French and English subsets and hold out one type at a time, giving seven folds. We also use HumAID, training on the ThreeCrises subset (floods, wildfires, earthquakes) and testing on the Hurricane subset and vice versa, matching the cross-disaster setting of DeCrisisMB. Finally, we use the CrisisNLP and CrisisLex unseen-event splits of Chowdhury et al. TREC-IS serves as a replication benchmark.
The first decisive check is predictive validity. Across the 200 configurations in each fold, we compute the Kendall correlation between each criterion and true held-out-type macro-F1. The core claim fails if the low-overlap stratum score correlates no better than overall validation F1.
The second check is regret, with three seeds per configuration and bootstrap intervals over held-out types and events. Sudden-onset types are reported separately.
The third check is the rank-reversal test. We grow the pool from 25 to 200 configurations in random order and record how often the selected configuration changes for classic versus anchored TOPSIS. Anchoring should eliminate pool-driven flips by construction, which the test verifies empirically. The remaining instability should come only from new, better configurations.
The fourth check guards against hidden trade-offs. For the selected configuration we report critical-class recall and high-overlap macro-F1, so that any gain on unseen types bought with in-event losses, the trade-off Schöffer et al. observed, is visible rather than hidden.
References
- Unsupervised and Interpretable Domain Adaptation to Rapidly Filter Tweets for Emergency Services
- Cross-Lingual Disaster-related Multi-label Tweet Classification with Manifold Mixup
- DeCrisisMB: Debiased Semi-Supervised Learning for Crisis Tweet Classification via Memory Bank
- CrisisTS: Coupling Social Media Textual Data and Meteorological Time Series for Urgency Classification
- Quantifying the Hyperparameter Sensitivity of Neural Networks for Character-level Sequence-to-Sequence Tasks
- CLaC at SemEval-2023 Task 2: Comparing Span-Prediction and Sequence-Labeling Approaches for NER
- Strategic COVID-19 vaccine distribution can simultaneously elevate social utility and equity
- A Ranking Approach to Fair Classification
- RaFe: Ranking Feedback Improves Query Rewriting for RAG
- Synergistic Interplay between Search and Large Language Models for Information Retrieval
- CMU-Informedia at TREC 2019 Incident Streams Track
- Overtuning in Hyperparameter Optimization
- Min-Max Multi-objective Bilevel Optimization with Applications in Robust Machine Learning
- Multi-Objective Hyperparameter Optimization in Machine Learning - An Overview