Suppose we train a sentiment classifier on book reviews and then use its score as the reward for teaching a language model to write positive reviews of kitchen appliances or cars. Two things predictably go wrong. First, the language model learns to pad its text with words that every domain treats as positive, such as "excellent" and "wonderful." This reliably raises the reward without saying anything favorable about the product. Second, some words reverse meaning across domains. An "unpredictable" plot is praise, while "unpredictable" steering is a defect. A classifier trained on books will push the generator toward exactly the wrong phrasing for cars. Ordinary reward hacking and context-dependent sentiment polarity are therefore the same failure: the reward model leans on a few domain-general words and misreads the domain-specific ones.
Domain-adaptation research has a tool built for this structure. Structural correspondence learning (SCL) picks out "pivot" words that behave the same way in both domains. It then hides those words and trains the model to predict them from the surrounding domain-specific text, so that phrases like "to die for" come to sit next to "excellent" in the learned representation. We propose to use this hiding trick on the reward model itself. We train it to judge sentiment only from text whose pivots are masked, and we also score generated text with pivots masked. Under this design, pivot stuffing earns no reward. The model is also forced to learn what positive and negative look like in the target domain's own vocabulary, using only unlabeled target text. Words whose meaning flips between domains are found and dropped from the pivot set before they can distort the reward.
Core Claim
Our central claim is that a sentiment reward model adapted to a new domain with pivot masking will give a smaller gap between the reward it reports and the true target-domain sentiment of what the policy writes, compared with an unadapted reward model or a standard domain-adapted one. Here "the policy" means the language model being trained, and true sentiment is judged by a held-out oracle classifier and by humans. We further claim that this gap comes from two failures that can be measured separately: the policy concentrating pivot words, and errors on polarity-inverted words. Each failure should respond to a different part of the method.
As far as the literature compiled here shows, SCL has never been used to build or adapt a reward function. The research file finds no paper joining correspondence learning with reward design. The closest alignment work, Towards Aligning Language Models with Textual Feedback, conditions generation on scalar-bucket or textual critiques. It assumes that labels such as "toxic" or "positive" mean the same thing everywhere, and it does not study domain shift.
Prior Work and How This Builds On It
Three papers supply the mechanism. Domain Adaptation with Structural Correspondence Learning introduced pivots, auxiliary tasks that predict whether a pivot is present, and a low-rank projection learned from those tasks. Biographies, Bollywood, Boom-boxes and Blenders made pivot choice depend on mutual information with source labels and reported a 46% relative error reduction over an unadapted classifier on the Amazon four-domain benchmark. Learning Sentence Embeddings with Auxiliary Tasks for Cross-Domain Sentiment Classification brought the idea into neural networks. It replaced pivots with an unknown-word token and trained a network jointly to classify source sentiment and to predict the hidden pivot's polarity from context. All three evaluate classifiers on fixed test sets.
We reuse the hiding objective, but we change where it matters. When a classifier is optimized against by a generator, the pivots become the cheapest route to high reward. Hiding them at scoring time removes that route. A fixed test set never exposes this vulnerability, so prior work had no reason to study it.
Weighted SCL reweights pivots by how reliable they are across domains. AX-MABSA documents polarity inversion of words like "sharp" and "unpredictable." Both motivate our inversion filter, which drops a candidate pivot when its source-label correlation disagrees in sign with its correlation to the adapted model's predictions on target text. When FLUE Meets FLANG shows that domain-targeted masking during pretraining helps with factual, adjective-free sentiment. It reports 91.9% accuracy on Financial PhraseBank against 87.2% for FinBERT. We treat masked-language-model adaptation of this kind as a competing baseline rather than as our method.
On the reward side, CoRe and MineDojo deploy learned or foundation-model evaluators without first measuring how far the target domain lies from the evaluator's training data. Our secondary analysis tests whether the proxy A-distance predicts how much the reward will be exploited. The A-distance is the error of a classifier trained to tell source text from target text, and Blitzer et al. found it tracks adaptation loss.
One finding argues against our design. ALT found that rich, unconstrained critiques performed worse than coarse buckets. That is a warning that richer reward signals can be noisier. Our reward stays a single scalar, so we take on none of that critique noise. We do change which input features that scalar can depend on.
Research Program
The reward model is a pretrained encoder fine-tuned with two losses. The first is source-domain sentiment classification on reviews in which every selected pivot is replaced by a mask token. The second is the auxiliary task from Yu and Jiang: on unlabeled source and target reviews, predict the polarity class of the hidden pivot from its masked context. Pivots are selected by mutual information on source labels, subject to a frequency floor in both domains. We then run the inversion filter and retrain. At policy-training time, every generation has its pivots masked before scoring.
For policy optimization we use two methods so that conclusions do not depend on one algorithm. The first is PPO with a KL penalty. The second is the reward-bucket conditional fine-tuning of ALT-RM, which that paper found more stable than PPO. The task is controlled review generation: given a product prompt from the target domain, write a positive or a negative review.
Ground truth comes from an oracle sentiment classifier with a different architecture. It is trained on the target domain's labeled reviews, which no adapted reward model ever sees, and it is checked with human ratings on a stratified sample. We also build a small evaluation set of polarity-inverted contexts for each domain pair, mined from target reviews containing words whose source and target label correlations have opposite signs. This set tests the reward model directly.
Baselines
The naive reference point is a reward model fine-tuned on source labels and applied to target generations unchanged. It tells us how much exploitation occurs when nothing is done. The adaptation baselines are SCL-MI from Biographies, Bollywood, Boom-boxes and Blenders, run as a projection-feature reward, and the joint auxiliary model of Learning Sentence Embeddings with Auxiliary Tasks used as a reward with pivots visible at scoring time.
The joint auxiliary model is the strongest existing competitor. It already learns pivot-equivalent target phrases, so it isolates what masking at scoring time and the inversion filter add on top. On the five-domain suite it reports 78.7% average accuracy across 18 transfer pairs, against 77.0% for a plain CNN and 71.2% for classical SCL. For SCL-MI on the Amazon benchmark, the file contains only relative error reductions (30% over SCL, 46% over unadapted), not absolute accuracies. We found no reported result on Lacuna for any reward model used to steer cross-domain generation, so the RL comparisons have no published numbers to beat and must be run here.
Three further competitors are run on equal footing. The first adapts the reward model by continued masked-language-model pretraining on unlabeled target text in the FLANG style. The second adds a booster of 50 labeled target reviews, the amount Blitzer et al. used. The third uses an instruction-tuned LLM prompted as a target-domain sentiment judge.
Two ablations isolate our ingredients: one masks only during training and one removes the inversion filter. Every reward model uses the same encoder size, the same unlabeled corpora, and the same hyperparameter search budget. Every policy run uses the same base model, KL coefficient sweep, and number of optimization steps. We judge results at matched KL divergence from the base model, since more drift can always buy more proxy reward.
Beating the strongest baseline means two things. At matched KL, the method must reach higher oracle-judged sentiment success and a smaller proxy-minus-oracle gap. The improvement must hold on a majority of the 12 Amazon pairs, with confidence intervals from three policy seeds that do not overlap.
Why not just…?
Why not just penalize pivot frequency in the generated text? This is the most direct fix for stuffing, so it is a real threat on that failure. It does nothing for polarity inversion, because a books-trained reward still pays for "unpredictable" in car reviews. We include it as a baseline, sweeping the penalty strength. If it matches our method on both oracle success and the inversion set, the inversion component of our claim fails. We expect it to match on pivot density but not on the inversion set.
Why not just prompt a large LLM as the judge? FLANG's results show general pretrained models defaulting to neutral on domain jargon. ALT shows that unconstrained LLM judgments can be inconsistent. Neither result, however, tests a strong instruction-tuned judge on review sentiment, so this remains open and could win outright. We run the judge on the same generations and the same inversion set, and we report its query cost per training run. If it wins, the useful conclusion becomes that pivot masking is the cheap substitute when a judge cannot be queried at every step. In that case the inexpensive judge still needs to be made robust, and we will say so.
Why not just label 50 target reviews? Blitzer et al. found such a booster corrects misaligned projections. We test it alone and combined with our method. If 50 labels alone close the gap, our method's value shrinks to settings where no target labels exist. The combination tells us whether the two help in different ways.
Why not just do target-domain masked-language pretraining? This adapts the representation, but it leaves pivots visible to the scorer and therefore still exploitable. The comparison at matched KL, with pivot density reported, shows whether adaptation without scoring-time masking is enough.
Proposed Experiments
The decisive experiments run on all 12 Amazon four-domain pairs, with the five-domain suite as a replication.
The first check establishes that the phenomenon exists. With the unadapted reward, we measure whether pivot density and the proxy-minus-oracle gap rise during training. If neither rises, the hacking motivation fails, and we report the result as a negative finding.
The second check is the main comparison. Plotting oracle success against KL divergence, our method should dominate the joint auxiliary baseline and the frequency-penalty baseline.
The third check targets inversion. Accuracy on the inversion set, and the oracle-judged sentiment of policy outputs on inversion-prone prompts, should separate the full method from its no-filter ablation and from the frequency penalty.
The fourth check is diagnostic. Across the 12 pairs, we compute the rank correlation between the source-to-target A-distance and the size of the unadapted reward's exploitation gap. A strong correlation would give practitioners a label-free warning before deploying a transferred reward. A weak one would show that classifier-era divergence measures do not carry over to optimized rewards.
Fluency is reported as perplexity under a fixed reference model, and human ratings check that masked scoring does not produce stilted, pivot-free text that the oracle happens to like.
References
- Towards Aligning Language Models with Textual Feedback
- Domain Adaptation with Structural Correspondence Learning
- Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification
- Learning Sentence Embeddings with Auxiliary Tasks for Cross-Domain Sentiment Classification
- Improving SCL Model for Sentiment-Transfer Learning
- AX-MABSA: A Framework for Extremely Weakly Supervised Multi-label Aspect Based Sentiment Analysis
- When FLUE Meets FLANG: Benchmarks and Large Pretrained Language Model for Financial Domain
- CoRe: Combined Rewards with Vision-Language Model Feedback for Preference-Aligned Reinforcement Learning
- MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge