Tenor-Controlled Contrastive Bootstrapping for Informal-Domain Lexicons

Lexicon bootstrapping grows a word list from a few seed words. It alternates two steps: find the textual patterns around known words, then find new words that fill those patterns. The method is cheap and needs almost no labels, but it has a well-known failure called semantic drift: within a few rounds, generic patterns pull in generic words and the lexicon wanders away from its category. On social media the problem is worse. The usual fix, down-weighting words that are frequent everywhere, cannot tell two kinds of frequent word apart. Some are frequent because they define the topic, like a slur in a hate-speech corpus. Others are frequent simply because they are how people type online, like "lol", emojis, and "ngl". We propose to score every candidate word and every pattern inside the bootstrapping loop against two background corpora rather than one. The first is edited formal text. The second is generic informal text written in the same casual register as the target but on no particular topic. Only candidates that are over-represented relative to both are accepted.

The second contrast carries the new part. It borrows the distinction in Cost-effective Selection of Pretraining Data between field (subject matter) and tenor (communicative style), and moves it from choosing whole pretraining corpora down to accepting individual words. The expected payoff is a target-domain lexicon that stays on-topic for more iterations on noisy text. It can then drive the additive half of domain adaptation that current hate-speech transfer methods lack, since those methods only penalize source-domain words and never acquire target-domain ones.

Core Claim

Semantic drift on informal text is largely register drift. Patterns that fire on casual-style tokens recruit more casual-style tokens, and a formal-only background corpus rewards exactly those tokens, because they are rare in Wikipedia.

Let $p_T(w)$ be a word's relative frequency in the unlabeled target-topic corpus, $p_F(w)$ its frequency in a formal background, and $p_I(w)$ its frequency in a topic-neutral informal background. We score a candidate as

$$ s(w) = \min\left(\log \frac{p_T(w)}{p_F(w)},\ \log \frac{p_T(w)}{p_I(w)}\right), $$

with add-$k$ smoothing on each ratio. The minimum means a word must beat both backgrounds. A pattern's score is the mean $s$ of the instances it extracts, so generic patterns are demoted before they recruit anything. The loop stops when the median $s$ of the newest accepted batch falls into the distribution of $s$ over randomly drawn words from the informal background. That gives an unsupervised stopping rule, which Graph-based Analysis of Semantic Drift and the Basilisk line leave to hand-set thresholds.

The claim is falsifiable at three points. First, the tenor term must matter over the formal-only score. Second, scoring inside the loop must matter over filtering the lexicon once at the end. Third, the lexicon must help downstream transfer beyond what uniform continued pretraining buys at the same compute.

Relation to Prior Work

Three bootstrapping systems motivate the proposal by showing both its value and its fragility on informal text. New Word Detection for Sentiment Analysis bootstraps Weibo neologisms using part-of-speech frames and internal statistics (a likelihood-ratio test, pattern entropy, and a multiword-expression distance). MASIVE grows more than 1,600 English affect terms from Reddit with the fixed frame "I feel [seed] and [MASK]". Cross-Domain Co-Extraction of Sentiment and Topic Lexicons transfers lexicons across review domains using dependency paths. None of the three compares candidates against any external background. We keep their pattern machinery and replace the acceptance function.

Two lines of work supply the contrastive idea but use it statically. Building Specialized Bilingual Lexicons Using Large Scale Background Knowledge scores terms by their specificity to a domain relative to general Wikipedia, in one pass over formal text. Retrieval of Research-level Mathematical Information Needs boosts extracted terminology in retrieval. We use the same kind of ratio, but repeatedly inside a loop, with a second, register-matched background that neither paper needs, because their text is formal.

Graph-based Analysis of Semantic Drift is the closest competitor on drift. It shows that Espresso-style alternating scoring converges to the most-connected, most generic nodes of the pattern–word graph, and it damps them with regularized graph kernels computed within a single corpus. That fix constrains our design. If within-corpus damping already suppresses register tokens, the second background is unnecessary. We argue it cannot, because a within-corpus kernel penalizes high connectivity regardless of cause. It will therefore also demote high-frequency topic anchors, which are exactly the words a hate-speech lexicon needs. The experiments test this directly.

Downstream, Domain Classification-based Source-specific Term Penalization identifies source-domain shortcut words with a domain classifier and feature attribution, then penalizes attention to them. It reaches an average cross-domain Macro-F1 of 60.5 across Waseem, HatEval and Vidgen transfer pairs. That method is subtractive, so it cannot teach a model target-domain slang it has never seen. fBERT is additive but brute-force: it spends 5 V100-days of uniform masked-language-model retraining on 1.4M offensive tweets. We use the induced lexicon additively and cheaply, by biasing which tokens get masked during a short adaptation run.

Research Program

The system has three components. The first is a bootstrapping engine with pluggable patterns. We will run both MASIVE-style masked frames, which a masked language model fills, and surface n-gram context windows. The n-gram windows serve because dependency parses degrade on telegraphic text. This lets us show the acceptance function is not tied to one pattern type.

The second component is the background corpora. The formal background is English Wikipedia. The informal background is the SOLID collection SOLID, with offensive-scored tweets removed. SOLID was gathered by querying 20 common stopwords, which makes it close to a topic-neutral sample of Twitter register. A sensitivity study will swap in forum text as the informal background, since Cost-effective Selection of Pretraining Data found forum text to be a robust intermediate tenor. It will also subsample each background in size, because that paper's broader lesson is that background choice drives results.

The third component is downstream use. The induced target lexicon sets per-token masking probabilities during a fixed-budget continued-pretraining run. Lexicon tokens are masked at a higher rate and everything else at the standard rate, with the overall masking rate held at 15%. The adapted encoder is then fine-tuned for cross-domain hate-speech detection, optionally combined with the source-term penalty of Bose et al. to test whether the additive and subtractive halves are complementary.

The analysis will track, per iteration, lexicon precision and the fraction of accepted items that are register tokens. Register tokens are words whose frequency ratio against the informal background is near one. This tracking shows whether drift on informal text really is register drift, which is the proposal's central empirical hypothesis. It is also informative if the hypothesis is false.

Baselines

The naive reference point is plain bootstrapping with the original systems' internal validation. For English Reddit affect this means MASIVE's frame with no filtering. For Weibo it means the three internal statistics of New Word Detection. For downstream transfer it means vanilla BERT fine-tuned on the source dataset. All bootstrapping variants use the same seeds, the same pattern sets, the same number of candidates accepted per iteration, and the same number of iterations, or each variant's own stopping rule where stopping is being evaluated. Beating the naive loop means higher precision at matched lexicon size.

The strongest existing drift control is the regularized Laplacian kernel from Graph-based Analysis of Semantic Drift. That paper reports 67.1% recall on Senseval-3, against 66.5% for Filtered Espresso and 54.5% for the most-frequent-sense baseline. Senseval is a formal word-sense task, so no reported result was found on Lacuna for any drift-control method on informal-lexicon precision. We will build the pattern–word graph from the same target corpus and tune the kernel's damping parameter on the same development split we use for our own acceptance threshold. Beating it means higher precision at 500 and 1,000 accepted items without lower recall of seed-category gold terms. A second strong baseline is a formal-only contrastive score in the style of Building Specialized Bilingual Lexicons, applied inside the loop. It isolates the tenor term.

For downstream transfer, the strongest competitor is source-term penalization Domain Classification-based Source-specific Term Penalization, which reports 60.5 average Macro-F1 across Waseem/HatEval/Vidgen transfer pairs (60.6 for its combined variant). We will rerun it on our splits rather than copy the number. We also rerun uniform-masking continued pretraining in the fBERT style on the same unlabeled target text, with the same GPU-hour budget as our lexicon-guided run. fBERT's own in-domain HatEval Macro-F1 is 0.596 against 0.483 for BERT, but that result is not a cross-domain number. Beating the transfer baselines means a gain in average cross-domain Macro-F1 over both, with seed-to-seed variance reported across five runs. Adding our method on top of Bose et al. should also beat Bose et al. alone.

The ablations that isolate the key ingredients use identical candidate pools. The first drops the $p_I$ term. The second applies $s(w)$ only once, as a post-hoc filter on the naive loop's final output. The third replaces lexicon-guided masking with random masking at the same rate.

Why not just…?

Why not just bootstrap normally and filter the final lexicon with the contrastive score? This is the most serious threat. If drift only adds junk that a final filter removes, the post-hoc variant will match us. Our argument is that drifted patterns consume the per-iteration acceptance budget and recruit further generic patterns. On-topic words that would have been reached through on-topic patterns are then never discovered, and no filter can recover them. The post-hoc ablation settles this directly by comparing recall of gold terms at matched precision. If the two tie, we will report that the contribution is the two-background score and not its placement in the loop.

Why not just use the regularized Laplacian, which already fixes drift? It fixes drift toward highly connected nodes within one corpus. In a target-topic corpus the most connected nodes include both register filler and the topic's core vocabulary, and the kernel demotes both. We expect it to lose recall on high-frequency anchors while keeping register tokens that happen to be moderately connected. The per-iteration register-fraction analysis will show whether this is what happens.

Why not just ask a large language model to list or vet domain terms? On MASIVE's informal affect task, zero-shot Llama-3 8B reached only 1.29% Top-1 accuracy against 17.91% for a fine-tuned mT5 (MASIVE). That result concerns identifying affect in context, not vetting lexicons, so it is only suggestive. An instruction-tuned model judging candidate relevance remains a plausible competitor. We will include it as a candidate-acceptance function in the same loop, which makes it a strict comparison of acceptance rules. We will also report its cost.

Why not just continue pretraining on target text with uniform masking? Uniform masking works, as fBERT shows, but it spends most of its updates on tokens the model already handles. SOLID reports that models stay near 57% F1 on implicit offense even though they exceed 90% on explicit offense. The matched-compute comparison decides this. If lexicon-guided masking does not beat uniform masking at an equal GPU budget, the downstream claim fails and the contribution reduces to lexicon induction.

Proposed Experiments

Intrinsic evaluation uses two informal settings with existing gold material. On English Reddit we reseed from a handful of MASIVE's basic affect words. We then measure precision and recall at fixed lexicon sizes against MASIVE's 1,627 English affect states, plus manual judgments of accepted items outside that list. On Weibo we reuse the 5,000 annotated candidates, of which 323 are true neologisms, from New Word Detection for Sentiment Analysis. With a Chinese informal background replacing SOLID, we measure precision and recall of neologism detection and the downstream sentiment accuracy on its 4,500-post benchmark, where that paper reports a 2–3% gain over HowNet alone. No reported result was found on Lacuna for intrinsic neologism precision in that paper, so our rerun of its method establishes the baseline. A regional-dialect probe on MASIVE's Spanish splits tests whether a dialect-matched informal background reduces the more-than-10% dialect drop the authors observed.

Downstream evaluation covers all six directed transfer pairs among Waseem, HatEval and Vidgen, reporting Macro-F1, with target-side unlabeled text used for bootstrapping and masked adaptation only.

Three results are decisive. The tenor ablation must show that removing $p_I$ raises the register-token fraction and lowers precision. If it does not, the core claim is wrong. The post-hoc ablation determines whether in-loop scoring matters. The matched-compute masking comparison determines whether the lexicon is worth building for adaptation at all. We will also check the stopping rule by plotting precision per iteration against the iteration the rule selects. The rule succeeds if it stops within one or two iterations of the precision peak on both languages without any tuning on those languages.