A Corpus-Statistics Critic: Semantic Orientation as a Free Baseline for Language-Model RL

Fine-tuning a language model with reinforcement learning (RL) means letting the model write its own text, scoring that text, and nudging the model toward higher-scoring outputs. Much of the cost goes into reducing noise in that nudge. The most common algorithm, PPO, does this by training a second network almost as large as the policy, called a critic, which predicts how good a partial output is likely to turn out. Critic-free methods such as group-relative policy optimization (GRPO) drop that network. Instead they generate several completions per prompt and compare each one against the group average, which shifts the cost into extra text generation. This proposal tests a third option for subjective, polarity-shaped rewards such as "make this review continuation positive." The critic would be a lookup table: semantic orientation scores counted from an unlabeled corpus, of the kind Turney used for unsupervised review classification, combined with a handful of linearly fitted weights.

The core idea fits in two sentences. Any function of the text generated so far can be subtracted from the final reward without biasing the policy gradient, so the critic never has to be right; it only has to correlate with the eventual reward. A word-polarity score computed from co-occurrence counts is almost free to evaluate incrementally, and it plausibly correlates with a sentiment reward. If it does, it can replace the critic network and several extra rollouts per prompt. This sidesteps the obvious failure of using corpus polarity as the reward, where the policy would simply learn to spam words like "excellent." A baseline that depends only on the prefix cannot be exploited this way. A poor baseline only fails to reduce noise.

Core Claim

Semantic orientation is defined over a positive seed set $P$ and a negative seed set $N$. For a token or phrase $x$, it is $\text{SO}(x)=\sum_{p\in P}\text{PMI}(x,p)-\sum_{n\in N}\text{PMI}(x,n)$, where PMI is pointwise mutual information, the log ratio of how often two items co-occur to how often they would co-occur by chance. This is the paradigm-set form of Measuring praise and criticism. Let $s_t$ be the prompt plus the first $t$ generated tokens, and let $\phi(s_t)$ be a small feature vector: the running mean and sum of SO over the extracted phrases in $s_t$, the prompt's SO, the count of negation tokens in a short window, and the normalized position $t/T$. The proposed critic is

$$b(s_t)=w^\top\phi(s_t),$$

where $w$ has under twenty entries and is refit by ridge regression on recent returns every few updates. The policy gradient for a sequence with terminal reward $R$ is then

$$\nabla_\theta J \approx \sum_t \nabla_\theta \log \pi_\theta(a_t\mid s_t)\,\big(R-b(s_t)\big),$$

where $\pi_\theta$ is the policy and $a_t$ the token chosen at step $t$. The estimate stays unbiased for any $b$, because $b(s_t)$ does not depend on $a_t$.

The claim is quantitative and falsifiable. On sentiment-steered continuation, this critic should match PPO's learning curve per prompt while removing the value network from GPU memory. It should also match GRPO's curve while using fewer completions per prompt. The size of the benefit should track how well corpus SO correlates with the reward model in that domain.

That last prediction is where the two literatures meet. Turney found SO-PMI predicts review ratings with Pearson correlation 0.5174 and 74.39% accuracy overall. Accuracy fell to 65.83% on movies because descriptive words like horrific carry negative polarity even in praise (Thumbs Up or Thumbs Down?). Variance reduction from a baseline scales with the squared correlation between baseline and return, which is the standard control-variate identity. Turney's per-domain correlations therefore predict how much each domain stands to gain.

Positioning

Three papers frame the RL cost problem this work targets. Is Reinforcement Learning (Not) for NLP built the RL4LMs library and GRUE benchmark, including IMDB sentiment continuation. It reduced gradient noise by masking low-probability tokens (NLPO) but kept a full critic. Act Only When It Pays showed that up to 80% of GRPO rollouts carry no learning signal and skipped them using per-prompt pass/fail history. The authors note this does not transfer to subjective rewards. Getting More Juice Out of the SFT Data documented that even reward-model-free alignment pays heavily for in-loop generation. These motivate the target. None uses corpus statistics, and none touches the critic as the object to make cheap. Our method is complementary to GRESO: it reduces the number of completions needed per prompt that is kept, rather than deciding which prompts to keep.

Teacher Forcing Recovers Reward Functions is the closest in spirit. It removes the reward model by reading dense token rewards out of a pretrained model's logits. We leave the reward untouched and replace the critic instead, which keeps the optimization target identical to the baseline methods and makes comparisons clean.

Three lines of work show that corpus polarity can be computed cheaply and used as features, and they supply our method. SVMs with Diverse Information Sources fed SO-PMI as features into a learned linear classifier, the precedent for a fitted $w$. SKEP mined sentiment words from unlabeled Amazon text by PMI from 46 seeds, and we reuse that seed expansion. Morpheme-based Derivation of Bipolar Semantic Orientation showed SO can be estimated from modest corpora. Two papers constrain the design: On the Impact of Seed Words and Modifying SO-PMI for Japanese Weblog Opinion Mining show seed choice and frequency imbalance bias SO. This is why we fit an intercept and per-feature weights rather than thresholding raw SO at zero. Unlike PMI-Masking, which precomputes an 800,000-span dictionary for encoder pretraining, our table is vocabulary-sized plus adjective/adverb bigrams. It is built once from the task's unlabeled corpus.

Research Program

The work has three parts. The first builds domain SO tables from the unlabeled IMDB reviews and from the unlabeled Amazon-2 corpus. It uses Turney's adjective/adverb bigram patterns, SKEP-style seed expansion, and a window-based co-occurrence count in place of web hit counts. The second implements the critic inside RL4LMs: an incremental feature update per generated token on CPU, plus a ridge refit of $w$ on a rolling buffer of (prefix features, return) pairs. The third is measurement. We record gradient-estimator variance directly, by repeating the gradient computation on held-out prompt batches. We record the correlation between $b(s_t)$ and $R$ as a function of $t$, peak GPU memory, rollouts consumed, and wall-clock time.

We also test the predicted dose-response. Correlation between corpus SO and the reward model should be lower on movie prompts than on product prompts, following Turney. Variance reduction and learning-speed gains should shrink accordingly. Within IMDB, gains should be weakest on horror and thriller prompts, which can be identified by genre words in the prompt.

Baselines

The naive reference point is REINFORCE with a constant baseline, the running mean of rewards. It costs nothing extra and shows how much any state-dependent baseline buys. It is run with the same policy (GPT-2 sized, as in the GRUE IMDB setup), the same DistilBERT-style sentiment reward model, the same KL penalty to the reference model, the same prompts, and the same learning-rate sweep.

The strongest existing method with a critic is PPO with a separate value network, and NLPO with its token masking, both as implemented in RL4LMs (Is Reinforcement Learning (Not) for NLP). We run them from the library defaults with an equal hyperparameter-search budget, counting trials, for every method. For reference numbers, no reported result was found on Lacuna for PPO or NLPO sentiment reward on IMDB. The file only states qualitatively that NLPO is more stable and scores higher than PPO across GRUE, so we reproduce both ourselves. Beating PPO means matching its final reward and perplexity within seed noise, measured over 5 seeds, while using strictly less peak memory. A further win would be reaching its final reward in fewer rollouts.

The strongest critic-free competitor is a group-mean baseline (GRPO/leave-one-out) with $k\in\{2,4,8\}$ completions per prompt, compared at matched total generated tokens. Beating it means the SO critic with $k=1$ or $k=2$ reaches the reward that the group baseline reaches with a larger $k$, at equal token budget. We also test SO combined with the group mean, using the SO critic on the residual, because the two may be complementary.

The ablation that isolates the key ingredient keeps the identical critic but swaps in a shuffled SO table: the same score distribution randomly reassigned across words. It has the same features and the same fitting procedure, with no polarity content. If the shuffled critic matches the real one, the gains come from position and length features rather than semantic orientation. We additionally ablate prompt-only SO against running prefix SO, and web-like general corpus against in-domain corpus.

Why not just…?

Why not just use semantic orientation as the reward and drop the neural reward model? This is the obvious intersection, and we reject it on purpose. A lexical reward is easy to game: the policy can raise SO by emitting seed-adjacent adjectives, the same degenerate-repetition hacking that Is Reinforcement Learning (Not) for NLP documents for surface metrics. It would also inherit Turney's 65.83% movie accuracy as a ceiling on what is being optimized. The saving would also be small, because the sentiment reward model is a small classifier, not the dominant cost. We include SO-as-reward as a single diagnostic run to demonstrate the hacking, not as a contender.

Why not just use group-relative baselines and skip the critic entirely? This is the real competitor. It needs no corpus, no features, and no fitting. Its cost is extra completions per prompt, and generation is the dominant bottleneck according to Act Only When It Pays and Getting More Juice Out of the SFT Data. The matched-token comparison above settles it. If the group baseline at $k=4$ beats the SO critic at $k=1$ per generated token, the proposal fails on IMDB. Its fallback contribution would then be only the combined SO-plus-group variant.

Why not just put a small value head on the policy's own hidden states? A linear head on shared activations costs almost no memory and sees context, including the movie-genre context that defeats SO. This is a genuine threat, and we run it with the same fitting budget. Our argument for SO is that the critic stays decoupled from the moving policy. It cannot drift or destabilize the shared trunk, and it is interpretable. Those advantages only matter if the head-based critic shows higher gradient variance early in training or instability across seeds. If the policy-head critic dominates on both variance and final reward, we will report that corpus statistics add nothing beyond what the policy already encodes.

Why not just score prefixes with the reward classifier itself? That gives a near-perfect state baseline, but it requires a classifier forward pass per prefix length, multiplying reward-model calls by the sequence length. We include it as an oracle-like upper bound at sparse checkpoints, every 8 tokens. It measures how much variance reduction a cheap critic leaves on the table.

Proposed Experiments and Decisive Checks

The primary setting is GRUE's IMDB positive-sentiment continuation via RL4LMs. The secondary setting is an analogous continuation task built from Amazon-2 review prefixes, scored by a sentiment classifier fine-tuned on Amazon-2 labels. The secondary task provides the product-domain contrast Turney's results predict. Metrics are reward-model score, perplexity under the reference model, KL to the reference, distinct-n diversity, peak GPU memory, rollouts and generated tokens consumed, wall-clock time, and directly measured gradient variance. All results are reported over 5 seeds with intervals.

Three checks decide the outcome. First, the real SO critic must reduce measured gradient variance relative to both the constant baseline and the shuffled-table ablation. If it does not beat the shuffled table, the idea is dead. Second, at matched generated tokens, the SO critic with $k=1$ or $2$ must reach the final reward of the group baseline with a larger $k$, and match PPO with lower peak memory. Third, the gain must order across domains as SO-reward correlation predicts: larger on Amazon than IMDB, smallest on horror-genre IMDB prompts. This would show the mechanism is the corpus correlation and not an artifact. If the first two checks pass but the third fails, the critic works for reasons we do not understand, and we would report that as such.