Dynamic Balancing of Intrinsic and Extrinsic Rewards

54 concepts51 papers1 proposal

In environments where external feedback is rare or absent, reinforcement learning (RL) agents cannot rely solely on environmental rewards ($r_e$) to learn. To escape local optima and discover sparse goals, agents are often equipped with an internal compass: an intrinsic reward ($r_i$) driven by concepts like curiosity, novelty, or prediction error.

The standard way to combine these signals is a simple linear combination:

$$r_{total} = r_e + \beta r_i$$

Here, $\beta$ is a scaling hyperparameter that dictates how much the agent values exploration versus exploitation. While this formulation is mathematically straightforward, its practical application reveals a deep fragility. The success of curiosity-driven exploration hinges entirely on the precise calibration of $\beta$, a challenge that has sparked a shift from static tuning toward dynamic, adaptive reward balancing.


The Goldilocks Dilemma of Intrinsic Motivation

Why is a fixed $\beta$ so difficult to tune? If we examine the optimization landscape of an intrinsically motivated agent, we find two catastrophic failure modes when $\beta$ is held constant.

Failure Mode 1: Insufficient Curiosity ($\beta$ is too low)

When the intrinsic reward coefficient is too small, the agent's behavior defaults to standard exploitation. In sparse-reward tasks, the external reward signal $r_e$ is zero across almost all trajectories. Without a strong enough $r_i$ to propagate exploration interest across state-action trajectories (as described by Fox et al. in their work on DORA), the agent fails to escape its initial state distribution. As noted by Schäfer and the authors of Hyper, an inadequate exploration incentive prevents the agent from uncovering the sparse external rewards necessary for task completion.

Failure Mode 2: Chronic Distraction ($\beta$ is too high)

Conversely, if $\beta$ is too large, the intrinsic drive overpowers the actual objective of the environment. The agent becomes a pure novelty-seeker, wandering aimlessly or becoming captivated by unlearnable environmental noise—a classic manifestation of the "Noisy TV" problem. Houthooft et al. demonstrate in VIME that an improperly scaled exploration drive causes agents to ignore external goals entirely in favor of local novelty. Similarly, Flet-Berliac et al. observe that excessive intrinsic weights cause agents to prioritize discovering new behaviors over exploiting known, high-reward paths.


Why the Optimal Balance Shifts Over Time

The core flaw of a static scaling factor is the assumption that the optimal trade-off between exploration and exploitation remains constant. In reality, an agent’s relationship with its environment is deeply temporal.

Learning Phase:   [ Early Discovery ]  --->  [ Mid-Training Transition ]  --->  [ Task Mastery ]
Optimal Drive:    High Curiosity             Balanced Calibration               Goal-Oriented Focus
Agent Action:     Explore Novel Paths        Refine Trajectories                Exploit Known Rewards

During the early stages of training, the agent knows very little about the environment. It needs a high curiosity drive ($\text{high } \beta$) to map out the state space. However, once a reliable path to the external goal is discovered, the utility of curiosity drops. If $\beta$ remains high during late-stage training, the agent will continuously deviate from the optimal policy to investigate minor, irrelevant variations in the environment.

As Chen et al. argue, the optimal balance is non-static: it requires high curiosity early on and a strict goal-oriented focus later. Manual tuning of fixed schedules or decay rates is not only computationally expensive but also fails to generalize across different task complexities or reward scales, as highlighted by Zheng et al. and Liu et al..


Algorithmic Solutions for Dynamic Adaptation

To resolve this sensitivity, recent research focuses on mechanisms that automatically calibrate or bypass the linear weighting bottleneck.

1. Constrained Optimization

Instead of treating intrinsic and extrinsic rewards as a single blended objective, some frameworks treat task performance as a constraint or a primary objective that bounds exploration.

Chen et al. developed Extrinsic-Intrinsic Policy Optimization (EIPO), which uses constrained optimization to automatically throttle curiosity. When the agent is making steady progress toward the external goal, EIPO minimizes the influence of $r_i$, ensuring the agent stays focused. If progress stalls, the exploration constraint relaxes, allowing curiosity to drive the agent toward new regions of the state space. A similar philosophy is shared by Shenfeld in the context of TGRL, where constrained optimization prevents an agent from becoming trapped by sub-optimal guidance signals.

2. Decoupled Architectures

Another elegant solution is to abandon the scalar blending of rewards entirely by separating the roles of exploration and exploitation into distinct architectural components.

                    +-----------------------+
                    |  Environment State s  |
                    +-----------+-----------+
                                |
               +----------------+----------------+
               |                                 |
               v                                 v
   +-----------------------+         +-----------------------+
   |   Exploration Policy  |         |  Exploitation Policy  |
   |   Optimizes: r_i      |         |   Optimizes: r_e      |
   +-----------+-----------+         +-----------+-----------+
               |                                 |
               +----------------+----------------+
                                |
                                v
                    +-----------------------+
                    |  Action Selection     |
                    +-----------------------+

Schäfer introduces Decoupled Reinforcement Learning (DeRL), a framework that maintains two independent policies: one dedicated exclusively to maximizing intrinsic novelty, and another focused solely on task exploitation. By separating these objectives, the exploitation policy is shielded from the erratic scales of intrinsic rewards. The authors of Hyper leverage a similar decoupling strategy, demonstrating that isolating exploration into a distinct phase makes the entire learning system provably efficient and remarkably robust to the exact magnitude of curiosity coefficients.

3. Reactive and Performance-Driven Feedback Loops

Other approaches dynamically adjust reward weights based on real-time performance metrics or environmental shifts. For instance, Pang et al. suggest that adjusting the balance between curiosity and supervised signals based on the agent's confidence prevents sub-optimal learning trajectories.

This responsiveness is particularly vital in changing environments. Steinparz et al. demonstrate that by monitoring sudden spikes in dynamics prediction errors, an agent can detect environmental non-stationarity and reactively scale its curiosity weight back up. This allows the agent to autonomously trigger re-exploration loops exactly when its learned policy becomes obsolete.

Key Foundational Work

VIME: Variational Information Maximizing Exploration (2016) — contributed 1 concept elements to this direction.
DORA The Explorer: Directed Outreaching Reinforcement Action-Selection (2018) — contributed 1 concept elements to this direction.
Exploration by Distributional Reinforcement Learning (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_intrinsic_reward_weighting_80bf7c7a,
  title = {intrinsic reward weighting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/intrinsic-reward-weighting/rdr_80bf7c7a1f734abc956f6b5f2898f5db},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_intrinsic_reward_weighting_80bf7c7a_v1,
  title = {intrinsic reward weighting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/intrinsic-reward-weighting/rdr_80bf7c7a1f734abc956f6b5f2898f5db/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake