Distributional Siblings as Hard Negatives Against Lexical Overlap Bias

Sentence encoders often judge two sentences to mean the same thing simply because they share most of their words. They will rate "the defendant was found guilty" as a near-paraphrase of "the defendant was found innocent". This proposal argues that the swaps most likely to fool an encoder are the ones that word statistics say are interchangeable. Distributional word clustering groups words by the contexts they appear in. Brown clustering, for example, builds a binary tree over the vocabulary from neighboring-word statistics, so words with similar usage share long bit-string prefixes. Such clustering places guilty and innocent, rise and fall, Microsoft and Nvidia in the same or adjacent clusters. The property usually counted as clustering's flaw, merging antonyms and co-hyponyms (sibling terms under one category), can be turned into a tool: a cheap, graded, language-independent generator of exactly the minimal pairs that encoders get wrong.

The plan has two steps. First, we measure whether an encoder's insensitivity to a one-word substitution grows with how deep in the cluster tree the two words share a prefix. Second, we fine-tune the encoder contrastively on substitutions drawn by prefix depth. An existing natural-language-inference (NLI) cross-encoder labels each substituted sentence as a paraphrase (a positive) or a meaning change (a hard negative). The result is a fast bi-encoder that keeps its speed but stops treating "same distributional slot" as "same meaning". It is evaluated on held-out overlap diagnostics it was never trained on.

Core Claim

The claim has two parts. First, the antonym and negation blindness reported for sentence encoders is predicted by shared distributional cluster depth, not merely by surface word overlap. Second, hard negatives sampled from deep cluster siblings repair that blindness more effectively than random, thesaurus-based, or embedding-neighbor substitutions at equal data size. The new part is using an explicit, tuned cluster hierarchy as both the diagnostic axis and the sampling distribution for counterfactual training pairs. None of the work in the file does either.

Relation to Prior Work

The motivating evidence comes from four sources. Vahtola et al.'s SemAntoNeg benchmark shows general-purpose encoders such as all-MiniLM-L6-v2 scoring 1.9% against a 33.3% random baseline by choosing high-overlap antonym distractors, while NLI-tuned models reach 93.6%. ALIGN-SIM finds an "antonym blind spot" in nearly all 13 encoders it tests, including models that score well on downstream tasks. Peinelt et al. define "non-obvious" pairs by Jensen-Shannon divergence over the pair's surface word counts, meaning high-overlap non-paraphrases and low-overlap paraphrases, and show that system rankings flip on those pairs. We extend their surface measure to a cluster-level one: overlap computed after mapping words to cluster prefixes. If the claim is right, this measure should identify a larger and harder non-obvious set.

The clustering side supplies the method. Pereira et al. introduced soft hierarchical distributional clusters. Tune Your Brown Clustering, Please shows that the cluster count matters a great deal, with Twitter part-of-speech tagging moving from 76.9% to 80.0% when tuned. It also shows that hierarchical prefix features hold up better than flat cluster IDs, which is why we use prefix depth as a graded variable rather than a single cluster ID. Feng et al.'s Cluster-CBOW acts as a constraint. Replacing rare words by their cluster vector hurt rare-word similarity, so treating same-cluster words as equivalent is harmful. We use cluster membership to propose contrasts, never to merge representations.

The closest competing mechanism is Neighboring Distribution Divergence from Peng et al.. It masks positions along the shared word sequence and compares masked-language-model predictions, catching synonym-to-antonym swaps that cosine similarity misses. It runs heavy masked-model inference for every pair at test time, and it was never evaluated on PAWS, HANS, SemAntoNeg, or Hard STS. We move the cost to training time and keep single-vector retrieval at inference. BERT-flow reshapes the embedding geometry so that similarity tracks edit distance less (correlation drops from −50.49 to −28.01), but its authors note it cannot add distinctions the encoder never learned, and antonymy is such a distinction. Disentanglement methods, namely BGT (46.6 on Hard− STS versus Sentence-BERT's 3.6) and ParaBART (75.9 average STS Pearson), require parallel corpora or parsers. Our method requires unlabeled text and an off-the-shelf NLI model.

Research Program

We first induce Brown clusters on a large unlabeled English corpus, choosing the cluster count by random search weighted toward larger counts, following Derczynski et al. Each word therefore receives a bit-string, and any two words have a shared-prefix depth $d$.

The diagnostic phase takes natural sentences from sources disjoint from every evaluation set; in particular we exclude Opusparcus, the source of SemAntoNeg. For each sentence we make single-word substitutions at controlled depths $d$, including a "different top-level branch" control. For each encoder we plot cosine similarity between original and substituted sentence against $d$. The NLI cross-encoder's meaning-preservation judgment serves as a reference label, and a sample of several hundred items is human-checked to estimate how often that label is wrong. The prediction is that encoder similarity rises with $d$ much faster than the meaning-preservation rate does. The gap between these two curves is the "distributional blind spot".

The training phase builds triples from the same substitutions. An anchor sentence is paired with a substituted variant labeled by bidirectional NLI entailment. Variants entailed in both directions become positives, and contradicted or non-entailed variants become hard negatives. The encoder is fine-tuned with a standard contrastive loss that also includes in-batch negatives. Sampling over depth is a tunable distribution, and the depth-sweep ablation tests whether deep siblings carry the benefit. Because thesaurus antonyms cover mostly adjectives, we report gains separately for adjectives, verbs, named entities, and numbers. Cluster siblings cover all of these, and this coverage is the concrete advantage we expect over thesaurus-based perturbation.

Baselines

The naive reference point is the unmodified encoder. We use all-MiniLM-L6-v2, which scores 1.9% on SemAntoNeg according to Vahtola et al., plus an NLI-trained Sentence-BERT. Sentence-BERT scores 3.6 on Hard− STS per Wieting et al.. Both are evaluated with no further training.

The strongest existing competitors depend on the benchmark. On SemAntoNeg, the best NLI-tuned models reach 93.6% (Vahtola et al.). This is also the ceiling for our teacher, so the relevant question is whether a fast bi-encoder distilled through our pairs approaches it. On Hard− STS, BGT's 46.6 is the reported number to beat. On standard STS, ParaBART's 75.9 average Pearson and BERT-flow's +8.16 average Spearman gain are the references for confirming that we lose no general quality.

Neighboring Distribution Divergence is run on our evaluation sets with its published procedure. There is no reported result found on Lacuna for NDD on SemAntoNeg, PAWS, HANS, or Hard STS, so we produce those numbers ourselves and also report its wall-clock cost per pair.

The ablations that isolate the key ingredient keep everything fixed except the substitution source. Each draws the same number of triples, uses the same NLI labeling, the same training steps, and the same tuning budget. The sources compared are random same-part-of-speech words, WordNet antonyms and synonyms, the encoder's own embedding nearest neighbors, and masked-language-model fill-in proposals.

Beating the baselines means the following. Cluster-sibling training must exceed every alternative substitution source on SemAntoNeg and Hard− by a margin larger than the seed-to-seed spread. It must surpass BGT's 46.6 on Hard−. It must lose no more than one point of average STS correlation relative to the starting encoder.

Why not just…?

Why not just add PAWS or HANS examples to training? Doing so raised BERT from 33% to 85% on those sets (PAWS; McCoy et al.). However, it is template-specific, and the same pattern of near-perfect in-template scores with collapse elsewhere appears on SAM, where challenge-trained models reach 98 DICE (Schlegel et al.). Our method faces the same risk. To test it honestly, we train only on substitution pairs and evaluate on PAWS (word order) and HANS (argument structure), which our pairs never contain. Gains on substitution diagnostics without transfer to these sets would show our method is a cheaper, broader instance of augmentation rather than a fix for the general heuristic, and we would report it that way.

Why not just use WordNet antonym substitution, as ALIGN-SIM and SemAntoNeg do? This is a direct ablation. We expect WordNet to match cluster siblings on adjectives but to miss entities, numbers, domain terms, and languages without a thesaurus. If it matches cluster siblings overall, the coverage argument fails, and the contribution reduces to the diagnostic finding.

Why not just pick substitutes from the encoder's own embedding neighbors or from a masked language model? This is the most serious threat, because those neighbors also encode distributional similarity and may already contain the antonyms. If they perform as well, the claim narrows to "distributional neighbors from any source", and the remaining argument for Brown clusters is their graded depth control and lower cost. We report this outcome plainly rather than defend the clustering itself.

Why not just run NDD or an NLI cross-encoder at test time? Both work on highly overlapping pairs, but they require a heavy model call per pair, which rules out indexing a large corpus. Our evaluation reports accuracy together with per-pair cost, so readers can see the tradeoff directly.

Proposed Experiments and Decisive Checks

We evaluate on SemAntoNeg accuracy, Hard+ and Hard− STS, the ALIGN-SIM antonym-replacement and semantic-distinction criteria, PAWS-Wiki and PAWS-QQP accuracy, HANS non-entailed accuracy split by heuristic, standard STS 2012–2016 and STS-B, and Peinelt et al.'s non-obvious F1 on MSRP and QQP. On MSRP and QQP we compute both the surface-overlap and the cluster-level-overlap definitions of "non-obvious".

The first decisive check is the diagnostic curve. If encoder insensitivity does not rise with shared prefix depth more steeply than the meaning-preservation rate, the central premise is false and the training phase is not justified.

The second check is the source ablation at matched data. If cluster siblings beat random, WordNet, and embedding-neighbor substitutions on SemAntoNeg and Hard−, and a sweep over depth shows the gain comes from deep siblings, then the cluster structure is doing the work.

The third check is held-out transfer to PAWS and HANS, which tests whether the method avoids the template memorization described above. We also expect the cluster-level non-obvious split to rank systems differently from Peinelt's surface split. If it does not, the cluster-level overlap measure adds nothing to evaluation.