Reinforcement Learning Generalization Across Model Scales

239 concepts224 papers

Reinforcement learning (RL) has become the definitive paradigm for transforming capable language models into strategic, step-by-step reasoners. By moving beyond static supervised fine-tuning (SFT) toward dynamic trial-and-error exploration, algorithms like Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO) allow models to discover optimal reasoning paths autonomously.

However, a critical assumption underlies much of this optimization research: an RL framework designed, tuned, and validated on a small-to-medium scale model (such as 1B to 8B parameters) will generalize predictably when scaled to frontier-class models (70B+ parameters).

As recent literature demonstrates, this assumption frequently breaks down. The interaction between policy optimization and parameter scale is highly non-linear. As model capacity expands, the internal representation space, baseline entropy distributions, and susceptibility to reward hacking change fundamentally, making parameter-scale generalization a central challenge in modern AI alignment.

Why Scale Alters the Exploration Landscape

To understand why RL frameworks fail to transfer predictably across scales, we must look at how parameter count changes a model's baseline behavior and optimization landscape.

1. Entropy and Policy Constraints

In smaller models, the policy's action space is implicitly constrained by the model's limited capacity. When optimizing these architectures, researchers can often omit complex regularization mechanisms. For instance, Xiangxiang Chu et al. introduced GPG, a minimalist RL baseline that strips away KL divergence constraints and critic models, proving highly effective at the 7B parameter scale. Similarly, Haoran He et al. demonstrated in Random Policy Valuation is Enough for LLM Reasoning with Verifiable Rewards that evaluating a fixed, uniform random policy can stabilize training for smaller models.

Small Model (e.g., 7B):
[Constrained Action Space] ---> [Simplified RL / No KL Penalty] ---> Stable Optimization

Large Model (e.g., 70B+):
[Massive Action Space]      ---> [Simplified RL / No KL Penalty] ---> Reward Hacking / Instability

When moving to frontier scales, this unconstrained optimization becomes hazardous. Because a 70B+ model possesses a vastly larger and more expressive action space, it can easily discover degenerate text patterns or "gibberish" tokens that maximize the reward function without performing genuine reasoning. The simplified optimization strategies that succeed in small regimes often suffer from catastrophic instability or mode collapse at scale.

2. The Overthinking Phenomenon and Length Dynamics

Frontier models exhibit an inherent capacity for extended generation, which frequently manifests as verbose, redundant reasoning traces. Jingyang Yi et al. address this in ShorterBetter, noting that while length-constrained rewards successfully curb "overthinking" in medium models, ultra-large models exhibit more entrenched verbosity that alters the trade-off between trace length and accuracy.

When models scale, the boundary between necessary deliberation and redundant token generation shifts. Consequently, efficiency mechanisms like the selective reasoning proposed by Jiaqi WANG et al. in Think or Not? or the adaptive depth framework by Wenyi Xiao et al. in Fast-Slow Thinking GRPO face entirely different trade-offs when applied to models with hundreds of billions of parameters. A constraint that forces a 7B model to be concise might inadvertently truncate the critical computational pathways of a 70B model.

Granularity and Credit Assignment at Scale

Another core dimension of parameter-scale sensitivity is how rewards are distributed—whether globally (outcome-based) or locally (step-based).

In smaller models, sparse outcome rewards (simply checking if the final answer is right or wrong) are often insufficient to guide complex reasoning, prompting researchers to develop dense, step-grained feedback loops. For example, Wu Fei developed Self-Guided Process Reward Optimization to derive step-wise advantages intrinsically, while Ganqu Cui et al. explored Process Reinforcement through Implicit Rewards to bypass explicit process reward models.

Step-Grained Reward (Small Model):
Step 1 (Reward) -> Step 2 (Reward) -> Step 3 (Reward) ---> Correct Path Found

Step-Grained Reward (Massive Model):
Latent State Explosion -> Dense Reward Overhead       ---> Optimization Bottleneck

Why does this dense credit assignment encounter friction at scale? In a smaller model, a step-wise reward provides a critical guide rail through a narrow reasoning corridor. In a frontier model, however, the internal latent states are vastly more complex. As JingHao Zhang et al. observe in RLFR, deriving dense rewards from internal velocity fields or latent states introduces extreme computational overhead and potential optimization bottlenecks when scaled to massive architectures. The dense feedback that stabilizes a small model can over-constrain a large model, suppressing the emergent, non-linear reasoning paths that larger capacities naturally enable.

The Non-Linear Transfer of Skills

The unpredictability of scale is also highly visible when training on synthetic environments or multi-stage pipelines. Anmol Kabra et al. demonstrated in Learning from Synthetic Data Improves Multi-hop Reasoning that while training on rule-generated data teaches models knowledge composition, the transferability of these skills is highly inconsistent across scales, with larger models showing less predictable improvements.

Furthermore, the optimization friction between different training phases changes as parameter counts cross certain thresholds. In standard post-training, models undergo SFT followed by RL. The authors of Don't Peek at the Answer and SRFT point out that this sequential transition induces optimization conflicts and severe reward hacking (such as answer token repetition). While single-stage blending or gradient masking mitigates this in smaller models, the extent to which these optimization conflicts evolve in frontier models remains an active frontier of uncertainty.

Ultimately, achieving true parameter-scale generalizability requires moving away from empirical tuning on small models. Instead, the field is shifting toward designing RL algorithms that explicitly account for how exploration boundaries, entropy distributions, and representation capacities expand alongside the parameter count.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_model_scale_generalization_model_scale_s_eaabc655,
  title = {model scale generalization (model scale sensitivity)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/model-scale-generalization-model-scale-sensitivity/rdr_eaabc655e0cf4561acb2e433db3a7dd3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_model_scale_generalization_model_scale_s_eaabc655_v1,
  title = {model scale generalization (model scale sensitivity)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/model-scale-generalization-model-scale-sensitivity/rdr_eaabc655e0cf4561acb2e433db3a7dd3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}

See something wrong? Report a mistake