Diversity Only Where Words Change Meaning: Targeted Ensemble Disagreement for Cross-Domain Sentiment

A sentiment classifier trained on book reviews and deployed on electronics reviews fails in a specific way. Most sentiment words keep their meaning across domains. A small set flips: "unpredictable" praises a thriller and condemns a steering system, and "sharp" praises a knife. An ensemble, meaning several models whose predictions are averaged, should in principle help. When members make different mistakes, the mistakes cancel. The ensemble literature, however, has found that forcing modern, high-capacity models to disagree usually hurts, because disagreeing with a model that is already right means being wrong.

This proposal resolves that conflict by making ensemble members disagree only on target-domain sentences that contain words whose sentiment appears to have flipped, and agree everywhere else. The flipped words are detected without any target labels. They are the words whose association with sentiment in the source labels contradicts their association, in unlabeled target text, with universal cue words such as "excellent" and "terrible." The claim is that diversity applied this narrowly improves cross-domain accuracy and flags likely errors. Uniform diversity, by contrast, gives no benefit or does harm, and plain random-seed ensembles leave the flipped cases uncorrected.

Core Claim

Two findings motivate the work. Buchanan et al. analyzed about 600 ensembles and found that diversity-promoting training helped large networks in only 10% of cases and hurt them in 61%. They called this the "correctness pathology" and argued for error-specific diversity instead of global disagreement. In the other direction, Local Independence Training (LIT) makes members' input gradients orthogonal, which pushes their decision boundaries apart. Under distribution shift it clearly beats random restarts: AUC rose from 0.87 to 0.96 on Ionosphere. Its reasoning is that models trained normally all extrapolate the same way into regions the training data never covered.

Cross-domain sentiment contains exactly such a region: target sentences whose key word carries the opposite meaning from the one learned in the source domain. Wilson et al. showed that contextual polarity shifts cluster around identifiable linguistic structures. On MPQA, prior polarity alone gave only 48% accuracy, compared with 65.7% when contextual features were added. We therefore claim that the conditions under which diversity helps (shift, unexplored regions) and the conditions under which it hurts (confident, correct inputs) can be separated sentence by sentence using unlabeled target text. An ensemble regularized to follow that separation should beat both a uniformly diversified ensemble and an unregularized one.

Research Program

The first component is a label-free detector for words whose polarity has shifted. We borrow the pivot idea from Yu and Jiang, who showed that universal pivot words ("excellent," "terrible") keep their sentiment across domains. They masked those pivots so that a network had to infer sentiment from the surrounding context.

We use pivots in the opposite way, as a weak polarity signal inside unlabeled target text. For each word $w$ we compute two quantities. The first, $a_S(w)$, is its log-odds association with positive labels in the labeled source data. The second, $a_T(w)$, is its log-odds association in the target domain with pivot-derived polarity, which is the polarity implied by the pivots that co-occur with it. A word is unstable when the two associations disagree in sign and both are estimated confidently. The sentence-level instability score $s(x) \in [0,1]$ is the maximum instability over the words in sentence $x$.

These pivot signals are used only to flag words. They are never used to label sentences for training. This matters because UDEED documents how pseudo-label errors propagate through semi-supervised ensembles.

The second component is the training objective. We train $M$ members, each a LoRA adapter plus classification head on a shared BERT-base encoder, so that the compute cost stays close to that of one fine-tuned model. The objective is

$$ \mathcal{L} = \sum_{m} \mathrm{CE}_S(f_m) + \lambda_d \sum_{x \in T} s(x) \sum_{m \neq \ell} \cos^2\!\big(\nabla_x f_m(x), \nabla_x f_\ell(x)\big) + \lambda_a \sum_{x \in T} \big(1 - s(x)\big) \sum_m \mathrm{KL}\big(\bar p(x) \,\|\, p_m(x)\big). $$

The terms are defined as follows. $\mathrm{CE}_S$ is the cross-entropy loss on labeled source data. $T$ is the unlabeled target text. $\nabla_x f_m(x)$ is member $m$'s gradient with respect to the token embeddings of $x$. $p_m(x)$ is member $m$'s predicted class distribution, and $\bar p(x)$ is the ensemble mean. The weights $\lambda_d$ and $\lambda_a$ set the strength of the two target-side terms.

In words, the loss makes members rely on different evidence on flagged target sentences and makes them agree on unflagged ones. Consider a flagged sentence such as "the steering is unpredictable but it looks great." Here some members are pushed to base their decision on cues other than "unpredictable." Averaging then gives less weight to the single source-learned reading of that word. This is LIT's independent-extrapolation argument, applied only where Buchanan et al.'s analysis says it is safe.

The third component is analysis. Member disagreement on flagged sentences should mark the ensemble's errors. We will therefore report selective accuracy, meaning accuracy on the sentences the ensemble does not abstain on, alongside plain accuracy. We will also check the instability detector after the fact against target test labels. Those labels are used only for evaluation, by recomputing each word's true target association.

Baselines

The naive reference point is a single BERT-base model fine-tuned on source labels only. On the 12 Amazon transfer tasks, Du et al. report 88.25% average accuracy for this setup.

The strongest existing method is BERT-DAAT, from the same paper. It combines adversarial domain alignment with masked-language-model training on target text. Its reported 12-task average is 90.12%, the number to beat. It also uses unlabeled target text, which makes it the fairest competitor. We will rerun it with our backbone and seeds and treat the reported figure as a sanity check.

The second baseline is a random-seed ensemble with the same $M$ and the same adapter setup. Buchanan et al. argue that this is what diversity methods must beat for large models.

The third is uniform diversity: the same objective with $s(x)$ fixed at 1, which is LIT applied to all target text. Together with the second baseline, this ablation isolates the key ingredient, namely the targeting.

We also include two further ensembles. One uses random targeting: $s(x)$ is shuffled across sentences while its overall frequency is kept. The other is a CoCMD-style two-view ensemble, following Peng et al., who co-train a domain-invariant encoder and a target-specific encoder with pseudo-labels.

For fairness, every method uses the same BERT-base backbone, the same unlabeled target pool, the same number of trainable parameters where it applies, and an equal hyperparameter-search budget. Model selection uses source validation data only, never target labels.

Beating the baselines means two things. The method must match or exceed the rerun BERT-DAAT on the 12-task average. It must also beat both the random-seed and the uniform-diversity ensembles on the flagged-sentence subset, with gains that hold across seeds.

Why not just…?

Why not just fine-tune one large pretrained model on the source domain? The file shows a consistent gap from domain-agnostic fine-tuning to domain-aware methods: 88.25% versus 90.12% on Amazon for Du et al., and 87.2% versus 91.9% on Financial PhraseBank for FLANG. Nothing in pretraining tells the model which learned associations flip in a new domain. The naive baseline measures this gap directly, broken down by flagged and unflagged sentences.

Why not just use a random-seed ensemble? This is a serious threat, since Buchanan et al. find that seeds are hard to beat in the large-model regime. Our prediction is specific. On unflagged sentences, seed ensembles and our method should perform the same. On flagged sentences, seed members share the source reading of the flipped word, as Ross et al. describe for data gaps, so our method should win there. If there is no gap on flagged sentences, the core claim fails.

Why not just mask or drop the flagged words during training? This is the simplest competitor to the targeting idea. It could capture most of the gain, because it also stops the model from over-relying on flipped words. It does so by discarding them, however, whereas Peng et al. argue that domain-specific words carry real sentiment signal. We will run flagged-word dropout on single models and on seed ensembles. If it matches our method, we will conclude that the detector, not the diversity, does the work, and we will report the result as such.

Why not just self-train on target pseudo-labels, for example ones derived from pivots? UDEED reports that pseudo-labeling methods such as ASSEMBLE propagate early errors. Under polarity inversion, confident pseudo-labels are likely to lock in the source reading of exactly the words that flipped. We will include a pivot-pseudo-label self-training baseline to test this directly.

Why not just use part-of-speech-split ensembles? Xia and Zong and SS-FE split the vocabulary by part of speech and train one classifier per group. They reach 74.48% on average and 80.38% on DVD→Books. Both need labeled target data to set the ensemble weights, and a split by part of speech cannot separate a flipped adjective from a stable one. We cite them for context rather than rerun them.

Proposed Experiments

The primary benchmark is the 12-task Amazon suite (Books, DVDs, Electronics, Kitchen). The second is the five-domain sentence-level suite of camera, laptop, restaurant and two movie domains, where the pivot-hiding network of Yu and Jiang reports 78.7% average accuracy. For a transfer into financial text on Financial PhraseBank, where neutrality bias is severe, we found no reported result on Lacuna for cross-domain transfer, so we will establish our own baselines there. MPQA will be used only to check whether the flagged words overlap with Wilson et al.'s polarity-shifting constructions.

Three checks decide the question. First, on each transfer pair, the gain over the random-seed ensemble should grow with the fraction of flagged target sentences and should be concentrated on those sentences. A flat gain would mean the mechanism is something other than what we claim. Second, uniform diversity should do no better than seeds on unflagged sentences and should often do worse, reproducing the correctness pathology in text. Random targeting should recover little of the gain. Third, member disagreement on flagged sentences should predict ensemble errors better than seed-ensemble disagreement does, measured as AUROC for error detection. The flagged-word dropout comparison determines whether the contribution is the ensemble objective or only the detector.