Sparse Sense-Level Polarity Corrections for Cross-Domain Sentiment

A sentiment classifier trained on one kind of text, such as movie reviews, gets worse when moved to another, such as kitchen-appliance reviews. Much of that loss comes from a small number of words whose meaning, and so whose sentiment, changes with context. An "unpredictable" plot is praise; "unpredictable" steering is a complaint. A "sharp" knife is good; a "sharp" pain is not. Current domain-adaptation methods try to make the whole feature space of the new domain look like the old one. That treats these flipping words as ordinary, and forcing them to align with their old polarity pushes the decision boundary the wrong way. We propose the opposite assumption. The source model is mostly right in the new domain and needs a few corrections, and those corrections attach to individual word senses (distinct meanings of a word), not to whole words or whole documents.

Concretely, we keep a source-trained classifier fixed and add an additive correction table with one row per induced word sense. We train it on weak target-domain labels using a group-sparse, outlier-resilient objective: an unsquared per-example loss, so noisy labels have bounded influence, plus a penalty that drives most correction rows to exactly zero. This is the joint "$\ell_{2,1}$" recipe from robust feature selection, applied to sentiment. The rows that survive form an explicit, inspectable list of senses whose polarity shifted in the new domain. The claim to test is that this sparse, sense-level correction recovers most of the cross-domain error on text containing flipped senses. It should also beat both whole-model fine-tuning on the same weak labels and word-level corrections that flip every use of "sharp" at once.

Core Claim and What Is New

Three lines of prior work each supply one piece and leave out the others. Lexicon adaptation by Adapting a Polarity Lexicon using Integer Linear Programming for Domain-Specific Sentiment Classification flips discrete word polarities under linguistic constraints, but it works at the word level, needs expression-level labels, and has no mechanism for tolerating label noise. Sense-level methods, namely ConnotationWordNet with graph propagation over a joint word-and-synset network and Neuro-Symbolic Sentiment Analysis with Dynamic Word Sense Disambiguation with synonym substitution trained by reinforcement rewards, show that separating senses matters. The first yields static sense polarities that do not adapt across domains. The second is tied to WordNet and trains with plain cross-entropy. Robust domain adaptation, namely margin widening in SAIM² and density-based sample avoidance in Active Sentiment Domain Adaptation, handles "outliers" by discarding samples. SAIM² in particular keeps only confident samples far from the boundary, which is exactly where flipped senses tend not to be.

What we do that none of these do is treat polarity-flipped senses as a row-sparse residual on top of a fixed source model. We estimate that residual with an objective that bounds the pull of both noisy labels and rare contexts. The method borrows the joint $\ell_{2,1}$ formulation from Efficient and Robust Feature Selection via Joint ℓ2, 1-Norms Minimization, which uses an unsquared loss together with a row-sparse regularizer and reports 5–12% gains over two-stage filter methods. We move it from gene selection to selecting which senses changed sentiment. The motivation for rejecting forced alignment comes from Transport or Discard, which shows in reinforcement learning that paying a cost to discard mismatched source data beats both forced matching and fixed-threshold filtering. We adopt the "discard" idea at the level of senses rather than samples, and we do not claim its transfer results carry over to text.

The Model

We start with a source-trained encoder and classifier $g(x)$, which outputs class scores. For every sufficiently frequent word type that appears in both domains, we induce sense units by clustering its contextual embeddings across source and target text, with at most four clusters per word. The number of clusters is chosen by silhouette score, and clusters are optionally mapped to WordNet synsets for inspection only. Inducing senses from the data rather than looking them up avoids the WordNet coverage and part-of-speech error problems noted for the neuro-symbolic pipeline. Each token $t$ in input $x_i$ is assigned a sense unit $s(t)$. The corrected prediction is

$$p(x_i;R)=\mathrm{softmax}\Big(g(x_i)+\tfrac{1}{|x_i|}\textstyle\sum_{t\in x_i} R_{s(t)}\Big),$$

where $R$ is the correction table, with one row $R_s$ per sense unit and one column per class, initialized to zero. We fit $R$ on target examples $T$ with weak labels $\tilde y_i$, given as one-hot vectors:

$$\min_R\ \sum_{i\in T}\big\|\tilde y_i-p(x_i;R)\big\|_2\;+\;\lambda\sum_s\|R_s\|_2 .$$

The first term is unsquared, so a mislabeled example contributes a bounded gradient instead of one that grows with its error. The second term is a group penalty: it switches whole sense rows off, so only senses with consistent contrary evidence receive a correction. We solve it with proximal gradient steps, whose group soft-thresholding step is cheap. This sidesteps the repeated matrix inversions of the reweighting solver used by Efficient and Robust Feature Selection via Joint ℓ2, 1-Norms Minimization. We choose $\lambda$ by a stability criterion: the overlap of selected rows across bootstrap resamples of $T$. This avoids tuning on target labels.

The weak labels come from one of two regimes. In the zero-label regime, each review's label is the source model's document-level prediction, kept only for confident documents. It is then propagated to clauses and adjusted by conjunction rules: clauses joined by "and" share polarity, clauses joined by "but" oppose. These are the co-occurrence cues that Active Sentiment Domain Adaptation mines from unlabeled text. This gives a signal that is not circular: a sense whose clause-level contribution systematically contradicts its document and its conjoined partner gets flipped. In the few-label regime, we add 100 gold target labels.

Research Program

The work has three parts. The first is building the sense inventory and the correction layer on a single shared backbone (RoBERTa-base), fine-tuned on source labels. The second is the analysis the method makes possible: for each domain pair, the list of surviving corrected senses ranked by row norm. We check it against a new human-annotated set of sense flips. For each of the 12 Amazon pairs, annotators judge the top 50 corrected senses and 50 random sense units shown in context. This yields precision of flip detection and an estimate of how common flips are. The same annotation lets us build a "flip subset" of each target test set: reviews containing at least one human-confirmed flipped sense. That subset is where the hypothesis predicts the gains. The third part is a controlled comparison against the alternatives below, all on the same backbone and the same weak labels.

Baselines

The naive reference is the source-only model: the RoBERTa-base classifier fine-tuned on source labels and applied unchanged. Every gain is measured against it on both the full test set and the flip subset. SAIM² reports a source-only 79.5% against its 87.3% on DVD→Kitchen with its own encoder, which indicates the size of the gap we are working in.

The strongest practical competitor is continued pre-training: masked-language-model training on unlabeled target text, followed by source fine-tuning. That is the FinBERT-style recipe examined in When FLUE Meets FLANG, where plain domain pre-training reached 87.2% on Financial PhraseBank against FLANG-ELECTRA's 91.9%. It requires no sense machinery and adapts every token's representation. We give it the same unlabeled target text and a compute budget at least equal to ours, which is easy because our method adds only a small table on top. We also stack our correction layer on the continued-pre-training model, since the two are complementary rather than exclusive.

The strongest published domain-adaptation result on the main benchmark is KinGDOM, which combines a ConceptNet graph encoder with adversarial alignment and reports 84.0% average accuracy over the Amazon pairs (82.3% on Kitchen→DVD), ahead of TAT at 82.4%. SAIM² reports 87.3% on DVD→Kitchen. These numbers come from older encoders, so beating them with a modern backbone proves little. We therefore reimplement adversarial alignment in the style of DANN and SAIM²'s margin widening on our backbone with an equal hyperparameter search budget of 20 configurations per method. We report both the reimplemented and the published numbers. Beating KinGDOM means beating its reimplementation on the same backbone on average and on the flip subset, not just exceeding 84.0%.

The most important competitor is full fine-tuning on the same weak labels with cross-entropy. It uses exactly our supervision signal but unconstrained capacity. If it matches us, the sense structure and sparsity add nothing.

Four ablations isolate the key ingredients. The first uses word-level rows instead of sense-level rows. The second replaces the group penalty with a squared (ridge) penalty, which spreads corrections over many senses. The third uses an elementwise $\ell_1$ penalty, which is sparse but not by whole row. The fourth uses a squared residual in place of the unsquared one. For lexicon-level comparison we also run ILP lexicon adaptation from Adapting a Polarity Lexicon using Integer Linear Programming for Domain-Specific Sentiment Classification, whose adapted lexicon moved CRF accuracy from 70.4% to 71.2% on MPQA. We use its flipped words as a word-level flip list scored against our annotations.

Why not just…?

Why not just continue pre-training on target text and skip senses entirely? This is the real threat. Contextual encoders already separate senses to some degree. The evidence that this is not enough is suggestive but not decisive. Neuro-Symbolic Sentiment Analysis with Dynamic Word Sense Disambiguation reports a 1.23% average gain over RoBERTa-large and FinBERT from explicit sense handling. Polarity inversion operators in PLM finds that BERT does not encode polarity reversal invertibly or compositionally. Neither shows that masked-language-model adaptation fails on domain-flipped senses specifically. Our flip-subset comparison settles this. If continued pre-training alone closes most of the flip-subset gap, the contribution shrinks to interpretability: the explicit flip list. We will report it that way.

Why not just fine-tune the whole model on the weak target labels? Unconstrained fine-tuning on roughly 80%-accurate pseudo-labels can memorize their errors, and it cannot say which senses changed. The robust feature selection results show joint $\ell_{2,1}$ training beating two-stage filtering by 5–12% under heavy noise. That is a different domain, so we test the claim directly. We vary the pseudo-label noise rate by injecting known flips into the few-label regime. We predict that the gap between our method and full fine-tuning widens as noise rises. If it does not, the robustness argument fails.

Why not just filter low-confidence target examples and train normally? Filtering by confidence or density, as in ASDA or SAIM²'s far-from-boundary sampling, removes exactly the examples where a flipped sense conflicts with the rest of the review. The fixed-threshold results in Transport or Discard show that thresholds tuned without knowing the true overlap cause negative transfer. We run a confidence-filtered cross-entropy baseline at three thresholds. The diagnostic is the share of flip-subset examples each threshold discards.

Proposed Experiments

The main benchmark is the Amazon Books, DVD, Electronics and Kitchen benchmark, with all 12 transfer pairs, using the 2,000 labeled reviews per domain and the unlabeled pools. The secondary benchmark is the five-domain sentence-level set (MV1, MV2, CR, LT, RT) from Learning Sentence Embeddings with Auxiliary Tasks for Cross-Domain Sentiment Classification, which reports 78.7% average for its joint model. We add a harder transfer from Amazon reviews into Financial PhraseBank in the few-label regime, where the non-emotive, fact-heavy vocabulary tests whether corrections can move neutral-looking senses. We found no reported result on Lacuna for review-to-Financial PhraseBank transfer. FLANG-ELECTRA's 91.9% with full in-domain supervision serves only as a ceiling. Metrics are accuracy and macro-F1 on the full test sets and on the flip subsets, precision@50 of detected flipped senses against the human annotations, and the number of nonzero correction rows.

The decisive checks are four. First, on the flip subset, sense-level group-sparse corrections should beat both full weak-label fine-tuning and word-level corrections by a margin larger than the seed variance over five seeds. The whole-test-set gain may be small, since flips are rare, and we will not overstate it. Second, precision@50 of the flip list should clearly exceed that of the ILP-adapted word list and of the ridge-penalty ablation's top rows. Third, as injected label noise rises, the unsquared residual should degrade more slowly than the squared one. Fourth, stacking the correction layer on the continued-pre-training model should still help on the flip subset. If the first and fourth fail together, sense-level correction is not needed once the encoder has seen target text, and the proposal reduces to a method for auditing which senses shift polarity across domains.