TRE: Encouraging Exploration in the Trust Region

Chao Huang, Yujing Lu, Quangang Li, Shenghe Wang, Yan Wang, Yueyang Zhang, Long Xia, Jiashu Zhao, Zhiyuan Sun, Daiting Shi, Tingwen LiuarXiv 2026 · arXiv · OpenReview

TL;DR

Large Language Models (LLMs) struggle with standard exploration techniques in reinforcement learning (RL) due to the risk of generating nonsensical outputs. This paper introduces Trust Region Entropy (TRE), a novel method that confines exploration to the model's "trust region" of plausible tokens, significantly improving performance on reasoning and alignment tasks.

What Problem Does This Paper Address?

Reinforcement learning (RL) is a powerful paradigm for aligning Large Language Models (LLMs) with human intent and enhancing their reasoning capabilities. A key challenge in RL is the exploration-exploitation trade-off: how to encourage an agent to explore new possibilities without sacrificing performance. A common technique for this is entropy regularization, which penalizes deterministic policies and promotes diverse action sampling.

However, this standard approach falters when applied to LLMs. The authors argue that this failure stems from two core issues:

  1. Sparse Valid Manifolds: LLMs operate with massive vocabularies (hundreds of thousands of tokens). The set of tokens that maintain syntactic and logical coherence forms an extremely sparse "manifold" within this vast space.
  2. Cumulative Tail Risk: Standard entropy regularization indiscriminately pushes probability mass towards all tokens, including the vast majority that are syntactically or semantically invalid. In short-horizon tasks, this noise might be negligible. But in LLMs, which often generate long sequences (e.g., for complex reasoning), this small noise accumulates exponentially. A single invalid token can derail an entire chain of thought, making coherent reasoning statistically impossible.

Essentially, standard entropy regularization, intended to encourage exploration, ends up injecting high-variance noise that disrupts LLM reasoning.

Methods

The paper proposes Trust Region Entropy (TRE), a novel regularization framework designed to address the limitations of standard entropy regularization in LLMs. The core idea is to encourage exploration only within the model's trust region.

Here's how it works:

  • Defining the Trust Region (ATR): Instead of considering the entire vocabulary, TRE identifies a "trust region" of tokens that the pre-trained model considers most plausible. This region is dynamically determined at each generation step.
  • Local Entropy Maximization: TRE then computes the entropy only over this identified trust region. This means it encourages diversity among the most likely next tokens, rather than diluting probability mass across the entire vocabulary.
  • Two Variants:
    • TRE-K: A size-based strategy where the trust region consists of the top K tokens with the highest logit values.
  • TRE-P: A nucleus-based strategy where the trust region is the smallest set of tokens whose cumulative probability mass reaches a threshold P. This allows the region size to vary dynamically based on the model's confidence.
  • Loss Function: The regularization loss is scaled to match the full vocabulary entropy scale and is added to the standard RL objective.

Datasets

The paper evaluates TRE on three distinct tasks:

  • MATH: A dataset of mathematical reasoning problems, compiled from various competitions (e.g., AMC, AIME). It covers seven subject areas and consists of 7,500 training and 5,000 testing examples. The maximum generation length is 8,192 tokens.
  • Countdown: A task involving generating mathematical equations from a given set of numbers to reach a target value. The dataset is sourced from "Countdown Tasks" and uses 7,500 training and 5,000 testing instances. The maximum generation length is 512 tokens.
  • HH (Helpful and Harmless): A task focused on general preference alignment, using the UltraFeedback dataset. It involves generating human-aligned, holistic responses. 7,500 training and 5,000 testing samples are used, with a maximum generation length of 1,024 tokens.

Key Insights and Ideas

  • Tail Risk is the Culprit: The paper convincingly demonstrates that the failure of standard entropy regularization in LLMs is not due to a lack of need for exploration, but rather the detrimental effect of "tail risk" – the cumulative probability mass leaked into invalid tokens over long generation sequences.
  • Trust Region is Key: The intuition that exploration should be confined to a "trust region" of plausible tokens is a powerful insight. This localizes exploration, preventing the injection of high-variance noise.
  • Dynamic vs. Fixed Regions: The TRE-P variant, which uses a dynamic nucleus-based trust region, generally outperforms the fixed-size TRE-K, suggesting that adapting the exploration scope based on model confidence is more effective.
  • Overconfidence in Vanilla RL: Standard RL training, especially on tasks with strict binary rewards (like MATH), leads to premature overconfidence in the LLM, where the policy quickly saturates and stops exploring. TRE effectively mitigates this.

Key Results

TRE consistently outperforms vanilla PPO and other existing regularization techniques (like Forking-Tokens and KL-Cov) across all three tasks and on both 1.5B and 7B parameter models.

  • MATH Task: TRE-P achieves the highest Pass@1 score (74.27% on 1.5B model), outperforming vanilla PPO (57.04%) by a significant margin.
  • Countdown Task: TRE-P again leads with the highest Pass@1 score (75.3% on 7B model), a substantial improvement over vanilla PPO (64.12%).
  • HH Task: TRE-P achieves the highest reward (3.0331 on 7B model), significantly outperforming vanilla PPO (2.8839). The paper notes TRE-P provides a reward gain approximately 4.1 times larger than KL-Cov on this task.
  • Policy Confidence: Figures show that TRE methods maintain lower and more stable peak probabilities (confidence levels) during training compared to vanilla PPO, which quickly saturates.

Limitations and Future Work

  • Ultra-Long Horizons: The paper acknowledges that TRE's efficacy over extremely long generation horizons (e.g., 128k+ tokens) and its impact on long-range coherence require further verification.
  • Massive Models: While TRE scales well up to 7B parameters, its behavior and hyperparameter stability on even larger models need to be empirically studied.
  • Hyperparameter Sensitivity: The paper conducts a hyperparameter analysis, finding that TRE-P with P=0.99 and TRE-K with K=2 are generally optimal. However, the sensitivity of these choices to different model architectures or tasks could be further explored.

Summary updated

Abstract

Entropy regularization is a standard technique in reinforcement learning (RL) to enhance exploration, yet it yields negligible effects or even degrades performance in Large Language Models (LLMs). We attribute this failure to the cumulative tail risk inherent to LLMs with massive vocabularies and long generation horizons. In such environments, standard global entropy maximization indiscriminately dilutes probability mass into the vast tail of invalid tokens rather than focusing on plausible candidates, thereby disrupting coherent reasoning. To address this, we propose Trust Region Entropy (TRE), a method that encourages exploration strictly within the model's trust region. Extensive experiments across mathematical reasoning (MATH), combinatorial search (Countdown), and preference alignment (HH) tasks demonstrate that TRE consistently outperforms vanilla PPO, standard entropy regularization, and other exploration baselines. Our code is available at https://github.com/WhyChaos/TRE-Encouraging-Exploration-in-the-Trust-Region.
BibTeX
@misc{huang2026tre,
  author = {Chao Huang and Yujing Lu and Quangang Li and Shenghe Wang and Yan Wang and Yueyang Zhang and Long Xia and Jiashu Zhao and Zhiyuan Sun and Daiting Shi and Tingwen Liu},
  title = {{TRE: Encouraging Exploration in the Trust Region}},
  year = {2026},
  eprint = {2602.03635},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.03635},
}

Loading related content...

Figure