Cluster-Coded Masking: Hiding Entity Names Without Hiding What Kind of Thing They Are

When a text classifier sees the same few names over and over, it learns the names instead of the sentences around them. A common fix is entity masking: replace the target name with a blank placeholder such as [MASK], so the model has to read the context. That fix has a cost. The model can no longer tell whether the blank stood for a drug, a gene or a city. Precision falls, and the model loses type cues that would let it reject absurd readings. This proposal tests a middle option. Instead of a blank, we replace the hidden word with a code for its unsupervised word cluster: a group of words that appear in similar contexts, learned automatically from unlabeled text in the target domain. The code says "a word that behaves like these other words in this community's writing" without saying which word it was. Because the clusters can be cut at any depth of a hierarchy, the code gives one dial that runs from full blinding (depth zero, the same as [MASK]) to near-full identity (deepest level, close to the word itself).

The setting where this matters most is domain mismatch between formal training text and informal target text, specifically illicit-drug jargon on Reddit and dark-web forums. Slang changes faster than anyone can annotate it, and no off-the-shelf entity typer knows that "molly" or "cokaine" is a drug. Clusters induced on the forum's own text can supply that type-like signal without labels, without a seed-term typer, and without exposing the surface form the model would otherwise memorize. To check that the idea is not specific to one task, we add a second test in biomedical relation extraction, where entity masking is known to roughly double recall while cutting precision sharply.

Core Claim

Replacing masked mentions with prefixes of target-domain Brown cluster bitstrings will beat both blank masking and full lexicalization on detection of unseen jargon under domain shift. (Brown clustering is a hierarchical, unsupervised word clustering that gives every word a binary path in a tree; shared prefixes mean similar distribution.) Plotted against prefix depth, held-out F1 should be inverted-U shaped: shallow codes carry too little type information, and deep codes collapse into near-singleton clusters that let the model memorize again. The gain should come specifically from unseen terms whose cluster prefix matches that of a training seed term. That prediction is the mechanism test that separates this idea from a generic "more features help" story.

Where This Sits in Prior Work

Three lines of work motivate the proposal. The first shows that unsupervised clusters carry most of what a supervised model needs to generalize across vocabulary. Word Representations: A Simple and General Method for Semi-Supervised Learning appended Brown clusters and neural embeddings to conditional random field taggers and raised CoNLL-2003 NER from 84.39 to 88.52 F1. Robust Multilingual Named Entity Recognition with Shallow Semi-Supervised Features stacked Brown, Clark and word2vec clusters so that 25% of the training data matched a cluster-free model trained on all of it. Simple Semi-supervised Dependency Parsing used Brown bitstring prefixes of several lengths as parser features. Clinical Information Extraction Using Word Representations showed that PubMed-induced Brown clusters beat word unigrams for evidence-based-medicine sentence classification (75.6% vs. 73.3% accuracy). All four append clusters next to visible words. None uses a cluster in place of a hidden word, and none combines clusters with masking. That substitution is our change, and it matters because an appended cluster leaves the memorizable word in view.

The second line supplies the masking regime and its known failure. Covering Cracks in Content Moderation: Delexicalized Distant Supervision for Illicit Drug Jargon Detection, called JEDIS below, masks 46 seed drug terms with [MASK] during distant supervision. It pairs a context-only BERT module with a word-attribute module that reads the target word's contextual embedding, and it transfers from Reddit to Silk Road. JEDIS is our direct host system. We keep its data and architecture and change only what fills the mask. EMBRE: Entity-aware Masking for Biomedical Relation Extraction constrains the design. Its entity-span masking raised BioRED recall from 0.1151 to 0.2173 but cut precision from about 0.40 to 0.23. Entity or Relation Embeddings? An Analysis of Encoding Strategies for Relation Extraction names the two failure modes: entity bias when entities are visible and context blindness when they are masked. It fixes them by concatenating entity and mask representations, $[H,T]+\text{Mask}$. That fix reintroduces the full entity embedding. We offer a coarser, unsupervised substitute and ask whether it keeps the precision benefit without the bias.

The third line explains why the clusters must come from the target domain. Zero-Resource Cross-Domain Named Entity Recognition found that fine-tuning word embeddings on source data overfits to source vocabulary, so frozen embeddings transfer better. Cluster codes are frozen by construction. CrossNER: Evaluating Cross-Domain Named Entity Recognition and fBERT: A Neural Transformer for Identifying Offensive Content show that continued pre-training on target text helps: span-level masking in CrossNER, and unnormalized social posts in fBERT. They are the main competitor to our cheaper route of target-domain adaptation through clusters alone. Semantic type masking, as in the @GENE$ substitution of Entangled Relations: Leveraging NLI and Meta-analysis to Enhance Biomedical Relation Extraction, is the supervised analogue of our idea and requires an entity typer.

Research Program

We induce Brown clusters, about 1,000 and 3,000 classes following the Malmasi setup, separately on three corpora: the unannotated Reddit drug corpus, the unannotated Silk Road forum corpus, and a general-domain corpus as a control. Every word then gets a bitstring path. A masked mention $w$ is replaced by a single new vocabulary token $c_k(w)$, the first $k$ bits of its path. Its input embedding is initialized to the mean of the encoder's embeddings for the training words in that prefix group. Multi-word or sub-word-shattered mentions take the code of their head word. When the code is used in place of a token, the code token is fed rather than the word's sub-word pieces. This also sidesteps the tokenizer fragmentation problem that EntityBERT documents. The depth $k$ takes values from 0 (identical to [MASK]) up to the full path.

For each $k$ we measure leakage: the fraction of seed terms whose prefix group contains only seed terms. High leakage means the code is effectively a word identifier, and we expect performance on unseen terms to fall as it rises. Discrete code tokens also avoid the instability Turian et al. reported when dense, real-valued embeddings are mixed with sparse features, since nothing needs rescaling.

We run the same substitution in two hosts. In JEDIS it fills the context module's mask during training and at test time; the word-attribute module stays unchanged. In BioRED relation extraction it replaces entity spans in a PubMedBERT relation classifier, with clusters induced on PubMed abstracts. A stacked variant, in the style of Agerri and Rigau, concatenates Brown, Clark and word2vec k-means codes as two or three tokens. It is tested only if the single-code version shows the predicted curve.

Baselines

The do-nothing reference is the JEDIS architecture with the seed term left visible, a fully lexicalized classifier. It shows how much the masking itself was buying. Next is JEDIS as published, with [MASK] substitution. This is the strongest existing method on the exact question and the system we must beat. Its reported F1 is 0.64 on the Reddit benchmark and 0.63 on the Silk Road benchmark (Covering Cracks in Content Moderation). We retrain it ourselves with the same seed list, corpora, number of training steps and hyperparameter search budget as our variant, and report both our reproduction and the published figure. Beating it means a gain over our reproduced JEDIS on both benchmarks that holds across five seeds under a paired bootstrap, with precision no lower than JEDIS's.

The key ablation is "append, don't replace": JEDIS with the seed word visible and its cluster code added. This is the Turian and Agerri recipe moved into a neural model. If it matches cluster-coded masking, then the substitution is irrelevant and the gain is simply cluster features. Two further ablations isolate where the clusters come from. One uses general-domain clusters in place of target-domain clusters. The other uses random codes with the same group sizes, which tests whether any partition of the vocabulary helps as regularization. A typed-placeholder baseline replaces the mask with a single [DRUG] token. In this task that token is constant across all training masks, so it tests whether gains come from a learned slot rather than class information.

For BioRED, the references are the PubMedBERT baseline and EMBRE as reported in EMBRE: overall F1 0.1789 vs. 0.2254, relation-extraction F1 0.2640 vs. 0.3245, and precision roughly 0.40 vs. 0.23. We also run gold-type `@GENE$`-style masking and the $[H,T]+\text{Mask}$ encoding from Entity or Relation Embeddings?, which reports 78.5% F1 on TACRED but no BioRED number on Lacuna. METAENTAIL-RE reports 0.891 F1 on BioRED. That is a different task formulation, natural-language inference over candidate hypotheses with a hypothesis filter, so we cite it for context and do not treat it as a like-for-like target. Here, beating the baselines means recovering precision toward about 0.40 while keeping most of EMBRE's recall.

Why not just…?

Why not just use semantic type placeholders like [DRUG] or @GENE$? In jargon detection the type is the label, so a type placeholder is either constant and uninformative or requires the classifier we are trying to build. Cluster codes are the only type-like signal available without supervision there. In BioRED, gold types exist and type masking is a real threat. If it matches cluster codes, the claim narrows honestly to settings without a reliable typer, which covers informal and emerging-slang domains. The BioRED comparison settles this directly.

Why not just append cluster features to the visible word, the classic semi-supervised recipe? That recipe (Word Representations, Robust Multilingual NER) leaves the seed word visible. A neural classifier will exploit the word itself, which is the memorization that JEDIS's masking exists to stop. We expect appending to help on seen terms and not on the seed-excluded benchmarks. The append-don't-replace ablation is designed to show this. If it does not, our mechanism is wrong.

Why not just continue pre-training the encoder on Silk Road text, fBERT-style? This could plausibly work as well, because domain-adaptive pre-training is the standard remedy for informal shift (fBERT reports HatEval macro-F1 0.596 vs. 0.483 for vanilla BERT). We therefore run it at matched compute, alone and combined with cluster codes. Cluster induction costs a small fraction of a pre-training run. A useful outcome is either that the two gains add up, or that clusters reach most of the adapted encoder's gain at that small cost. If adapted pre-training alone dominates and the combination adds nothing, we report that as the main finding.

Why not just prompt a large language model? JEDIS reports that GPT-4o-mini reaches high recall but low precision on these benchmarks, flagging benign slang. We include it as a reference point on the same 1,000-sentence test sets and do not expect it to be competitive on F1.

Proposed Experiments and Decisive Checks

The primary experiment trains on Reddit with the seed terms masked and tests on both JEDIS benchmarks. Each benchmark has 1,000 sentences, and both exclude the 46 seed terms. We sweep prefix depth $k$ for each cluster source and report precision, recall, F1 and the number of unique jargon terms detected; JEDIS reports detecting 4.23× more unique terms than word-based systems. The first decisive check is the shape of the depth curve. An inverted U whose peak lies above the $k=0$ (blank-mask) point supports the core claim. A monotone curve in either direction refutes it: rising all the way means plain lexicalization wins, and falling all the way means blinding wins.

The second decisive check is mechanistic. We split the gold jargon terms in each benchmark by whether their cluster prefix at the chosen depth is shared with any seed term. Cluster-coded masking should improve recall mainly on shared-prefix terms and leave non-shared terms near JEDIS's level. Uniform gains would point to generic regularization, which the random-code ablation also probes. The third check is the transfer direction. Clusters induced on Silk Road, the test domain, should beat clusters induced on Reddit when evaluating on Silk Road, and both should beat general-domain clusters. This ordering is the specific prediction tying the method to domain mismatch rather than to clustering in general.

In BioRED we report precision, recall and F1 for the relation-extraction and overall tasks across depths. The question is whether an intermediate depth sits on a better precision-recall frontier than both the PubMedBERT and EMBRE operating points. We also log the polysemy failure cases, such as "speed", that a static cluster cannot resolve. These check whether JEDIS's contextual word-attribute module absorbs them, or whether a static class code actively hurts on ambiguous terms. That limitation would remain open and would need context-sensitive clusters as follow-up.