Contested Regions: Using Encoder Disagreement to Decide Where Fusion and Alignment Are Safe

Most language systems start from a frozen pre-trained encoder, a model that turns text into vectors, and build everything else on top: classifiers, augmenters, out-of-distribution detectors. When that encoder places two different concepts close together, nothing downstream can separate them again. A common response is to fuse several encoders, on the hope that their mistakes are different. A second common response is to align their vector distributions so the fused space is coherent. Both responses quietly assume the sources should agree. This proposal argues that the points where pre-trained encoders disagree about which examples are neighbors carry the most useful signal in the whole pipeline. Those points mark where one encoder has a blind spot. Pulling the encoders into agreement at those points spreads the blind spot to the other encoders. Generating synthetic training points there amplifies it.

The concrete idea is to compute, without labels, a "contestedness" score for every example. The score measures how much the encoders' nearest-neighbor sets around that example fail to overlap. The method then uses this one map in three ways. It aligns encoder distributions only on uncontested examples. It synthesizes augmentation points only in uncontested regions. In contested regions it lets the few available labels decide which encoder to trust. The test bed is few-shot out-of-distribution intent detection. Labels there are too scarce to learn a fusion gate, and the best existing method is known to hit a ceiling set by its single encoder.

Core Claim

Fusing several pre-trained encoders helps few-shot classification mainly through a small set of contested examples. Uniform distribution alignment, which pulls every example's representation toward a shared distribution, actively hurts on exactly those examples. Restricting alignment and latent augmentation to uncontested regions, and adjudicating contested ones with the scarce labels, should beat both single-encoder augmentation and uniform optimal-transport fusion. A separate, checkable claim comes with it: contestedness should predict where a single-encoder classifier makes errors, which gives a label-free diagnostic of embedding quality failures.

Where This Sits

The motivating evidence comes from work on dependence on the base encoder. A Closer Look at Few-Shot Out-of-Distribution Intent Detection trains a denoising autoencoder, a network that learns to reconstruct clean vectors from corrupted ones, in BERT's vector space. It uses this autoencoder to generate in-distribution points and mixes points from different classes to create synthetic outliers. It reports a fidelity limit: generating more than about 100 synthetic points per example degrades the classifier, because the generator reproduces BERT's own confusions. Specializing Word Embeddings (for Parsing) by Information Bottleneck shows that compressing an encoder can filter out irrelevant information but cannot add structure the encoder never had. Both results point to the same fix: bring in a second source of information about the input, which here means a second encoder.

The fusion literature supplies the components. Learning Word Meta-Embeddings combines several word-vector sets by concatenation, by SVD (a linear compression), and by learned linear maps that predict one space from another. Learning Word Meta-Embeddings by Autoencoding does the same with nonlinear autoencoders. Learned in Translation: Contextualized Word Vectors concatenates static GloVe vectors with vectors from a translation encoder. CSE-SFP gets two complementary views from a single large language model in one forward pass. All four treat every input the same way, and none asks where the sources disagree.

The alignment literature provides the closest competitors. OTKGE fuses text, image and graph features of knowledge-graph entities into a Wasserstein barycenter, an average distribution under optimal transport, which is the cheapest way to move probability mass from one distribution onto another. It explicitly concedes that errors in the pre-trained extractors survive this fusion. Unsupervised Multilingual Alignment using Wasserstein Barycenter aligns many language spaces to a barycenter "virtual pivot." Supporting Multimodal Intermediate Fusion with Informatic Constraint and Distribution Coherence (called IID below) adds a Wasserstein loss that makes fused modality distributions coherent, motivated by a generalization bound that its own authors derive only for linear classifier heads. Gromov-Wasserstein Alignment of Word Embedding Spaces matches spaces by comparing within-space similarity structure. That method supplies our idea of comparing neighborhoods, but it also illustrates the risk we target: it maps one space's distortions onto the other.

The contrast is the same in every case. These methods align everything. We align only where the sources already agree, and we treat disagreement as information to keep.

Research Program

The setup uses $M$ frozen encoders. In the main configuration $M=3$: BERT (the encoder of Zhan et al.), averaged GloVe vectors, and a Mistral-7B sentence encoder trained as in CSE-SFP. The three come from deliberately different families, so their errors are plausibly less correlated. Each encoder embeds a pool $U$ of unlabeled utterances. For an example $x$, let $N_k^m(x)$ be its $k$ nearest neighbors in $U$ under encoder $m$. The contestedness score is

$$c(x) = 1 - \frac{2}{M(M-1)} \sum_{m<m'} \frac{|N_k^m(x) \cap N_k^{m'}(x)|}{k},$$

which is one minus the average pairwise neighbor overlap. It is 0 when all encoders agree on the neighborhood and 1 when they share no neighbors. A Gromov-Wasserstein local-distortion score will be tested as an alternative to the overlap count.

The three encoder spaces are mapped into a shared space with learned projections $P_m$. The projections are trained with a Wasserstein alignment loss that is weighted by $(1-c(x))$, so contested examples contribute almost nothing to it. The fused representation then has two parts. The first is the aligned shared part. The second is the per-encoder residuals, which are kept but not aligned, so that information unique to one encoder survives.

Inside contested regions, the method decides which encoder to trust by checking label purity: for each encoder, how often the labeled examples among $x$'s neighbors share a class. It upweights that encoder's residual accordingly. This is how scarce labels are used where geometry alone cannot settle the question. Agreement among encoders cannot tell "uniquely right" from "uniquely wrong," and labels can.

Augmentation follows the Zhan et al. recipe in the fused space. The denoising autoencoder synthesizes in-distribution points, and cross-class mixtures provide outliers. Sampling is gated by $1-c$, so synthetic points come mostly from uncontested regions. If the fidelity limit really comes from the generator reproducing encoder confusions, gating should push the point where adding synthetic data starts to hurt to much larger sample counts.

One limitation is stated up front. If all encoders share a bias, for example because all were trained on similar web text, then $c(x)$ will be low on examples that are in fact wrongly placed, and the method gains nothing there. The controlled-degradation experiment below measures how much of the signal survives that case.

Baselines

The naive reference point is one frozen BERT encoder with a softmax classifier that rejects inputs whose top probability falls below a confidence threshold. The threshold is tuned on the same validation split.

The strongest existing method is the denoising-autoencoder augmentation of A Closer Look at Few-Shot Out-of-Distribution Intent Detection. That paper reports 62.19% macro F1 on CLINC150 in the 5% few-shot setting, more than 9 points above its ADB baseline. We will re-run it with the authors' protocol to obtain matched numbers on Banking77 and StackOverflow, for which the research file gives no reported figures. Beating it means a macro F1 gain on CLINC150 at 5% that exceeds the across-seed standard deviation, together with no loss on the other two datasets.

To separate "more encoders" from "our mechanism," the same augmentation is also run on plain concatenation of all three encoders. This is the strongest naive fusion baseline. Two alignment-based competitors follow: uniform Wasserstein-barycenter fusion in the style of OTKGE, and an IID-style distribution-coherence loss applied to the three encoders. Both use the same projections, the same augmentation, and the same tuning budget. The baseline set also includes a learned attention gate over encoders, trained on the labeled examples, and the best single encoder on its own.

Three ablations isolate the key ingredient. The first replaces contestedness with random weights of the same overall distribution. The second removes gating of alignment only. The third removes gating of augmentation only. All methods get an identical hyperparameter search budget and five data seeds, and they share the same unlabeled pool $U$.

Why not just…?

Why not just concatenate the encoders and run the existing augmentation? This is the most likely real threat, because concatenation preserves every source's structure, as Yin and Schütze argue. However, the augmentation generator would then still interpolate inside regions where one source conflates classes. Our prediction is that concatenation helps on uncontested examples but inherits the fidelity limit on contested ones. The concatenation-plus-augmentation baseline, broken down by contestedness bucket and swept over synthetic sample counts, tests this directly. If concatenation matches us in every bucket, the contestedness mechanism is unnecessary and we will report that.

Why not just learn an attention gate that weights encoders per example? A gate learns which encoder to trust from labels. At 5% of CLINC150, that leaves a handful of examples per class, so the gate is fit on almost nothing and tends to collapse onto the encoder that is best on average. Contestedness is computed on the unlabeled pool, and labels are used only to adjudicate within the contested set. The learned-gate baseline at 5% versus 10% data tests whether the gap closes as labels grow. We expect it to narrow, and that narrowing would locate the method's value in the low-label regime.

Why not just use the biggest encoder? A 7B-parameter encoder may simply have fewer blind spots, and CSE-SFP shows that strong sentence representations can be extracted from one. If the Mistral encoder alone matches the fused system, fusion is moot. The best-single-encoder baseline covers this. So does the bucketed analysis: if Mistral's errors concentrate in examples that are contested between Mistral and the other encoders, the other encoders are carrying information that Mistral lacks.

Why not just align everything, as OTKGE and IID do? Their gains are reported on paired multimodal data, and in the IID paper the coherence argument is proven only for linear heads. Within one modality, alignment pulls a correctly placed encoder toward a wrongly placed one. The comparison between uniform and gated alignment, restricted to high-contestedness examples, is the decisive test of this point.

Proposed Experiments

The main experiment covers CLINC150, Banking77 and StackOverflow at the 5% and 10% labeled settings. It reports macro F1 over known classes plus the out-of-scope class, out-of-distribution F1, and AUROC (area under the curve that trades off false alarms against missed detections) for out-of-scope rejection. Results are given overall and per contestedness quintile.

The first decisive check tests whether contestedness is a real diagnostic of embedding quality. We measure whether $c(x)$, computed without labels, predicts the errors of the single-BERT classifier, scored by error-prediction AUROC, and compare it against the classifier's own confidence.

The second check is controlled degradation. We corrupt one encoder by masking its domain-specific subwords on Banking77, creating a known vocabulary deficiency. We then verify three things: contestedness rises on the affected utterances, label adjudication shifts trust away from the corrupted encoder, and uniform alignment loses more accuracy than gated alignment. We repeat this with two encoders sharing the same corruption to measure how far correlated errors erode the signal.

The third check sweeps synthetic samples per example from 10 to 500, with and without contestedness gating. If the fidelity limit moves to higher counts under gating, that supports the claim that the limit is caused by encoder confusion rather than by generator capacity.