Bottom-up Policy Optimization: Your Language Model Policy Secretly Contains Internal Policies

Introduction

Traditional reinforcement learning (RL) for Large Language Models (LLMs), such as RLHF or DeepSeek's GRPO, typically treats the model as a "black box." Optimization focuses solely on the final output token, viewing the dozens of intermediate transformer layers as a unified, opaque policy. However, LLMs process information through a residual stream where each layer contributes incrementally to the final computation.

This paper introduces Bottom-up Policy Optimization (BuPO), a novel training paradigm that decomposes an LLM into a hierarchy of internal policies. By optimizing these internal layers during the early stages of training, BuPO forces the model to build a stronger reasoning foundation, leading to significant performance gains on complex mathematical and logical tasks.

Internal Policy Decomposition and the EIC Pattern

The core of BuPO lies in treating the hidden states of any given layer as a standalone policy. This is achieved by projecting internal representations through the model's existing unembedding matrix—the component that maps high-dimensional vectors back into the vocabulary space. This decomposition allows researchers to measure the "entropy" (uncertainty) and policy distribution of every single layer.

Expand

Source, p. 1

Through this lens, the authors identified a universal reasoning pattern termed Exploration–Integration–Convergence (EIC). In high-performing models like the Qwen series, early layers engage in "Exploration" (high entropy), middle layers perform "Knowledge Integration," and top layers finally "Converge" on a specific answer.

In contrast, many other models exhibit a "lazy" reasoning pattern. For example, Llama-3.2-3B-Instruct maintains high uncertainty across almost all layers, only converging abruptly in the final few steps of the transformer block. This suggests that the model is "cramming" its decision-making into the very end of the architecture rather than reasoning progressively.

Expand

Source, p. 3

Bottom-up Policy Optimization (BuPO)

To rectify "lazy" reasoning and encourage a more robust internal logic, BuPO employs a two-phase training strategy:

1. Phase 1: Internal Optimization: For the initial steps of training, the gradient is applied not to the final output, but to a specific "early" internal layer (e.g., Layer 6). This forces the model to capture high-level reasoning features much earlier in its architecture, a process the authors call "feature preemption."
2. Phase 2: Global Optimization: Once the internal foundation is established, the training switches to standard GRPO (Group Relative Policy Optimization) to fine-tune the final output for accuracy and fluency.

This approach adheres to a "Goldilocks Principle": while early internal optimization builds a better foundation, over-aligning internal layers can cause language fluency (perplexity) to collapse. BuPO identifies the "sweet spot" where internal logic is strengthened without compromising the model's overall generative capabilities.

Experimental Results

BuPO was evaluated using the DeepMath-103k dataset and tested against rigorous benchmarks including MATH500, AMC23, and the highly challenging AIME24 and AIME25 examinations. The results demonstrate that BuPO consistently outperforms standard RL baselines (PPO, GRPO, RLOO) across different model families:

• Qwen3-4B: Achieved a +4.69 point gain on AIME24 and a +3.43 point average gain across all benchmarks compared to standard GRPO.
• Llama-OctoThinker-8B: Showed an average improvement of +3.68 points.
• Efficiency and Robustness: BuPO does not require additional training data; it simply optimizes the existing architecture more effectively. Pass@K analysis further indicates that BuPO models are more stable and likely to find correct answers with fewer samples.

Conclusion and Future Work

Bottom-up Policy Optimization represents a shift from treating LLMs as monolithic black boxes to understanding and guiding their internal reasoning flow. By encouraging early-layer feature refinement, BuPO creates models with more robust logical foundations. Future research will focus on automating the selection of target layers through a curriculum-based approach and exploring the applicability of BuPO to non-mathematical domains such as creative writing and multi-modal tasks.