Correcting Trajectory Length Bias in Reinforcement Learning

27 concepts14 papers1 proposal

In reinforcement learning, we often treat the duration of an episode as a background detail—a container for the rewards we care about. However, a growing body of research suggests that the length of a trajectory is not a neutral observer. Because an agent’s policy often determines how long an interaction lasts (e.g., a robot stopping once it reaches a goal), the time horizon itself becomes a source of systematic bias and numerical instability.

The Vanishing Weight Problem

The most visible failure of trajectory length occurs in Off-Policy Evaluation (OPE). When we use historical data to estimate the value of a new policy, we typically rely on Importance Sampling (IS). As Thomas et al. demonstrate, IS weights for long trajectories often vanish. Because these weights are calculated as a product of probability ratios across every timestep, the variance explodes and the magnitude collapses as the episode length increases.

This creates a "fairness" problem. Shen et al. point out that standard estimators effectively ignore data from longer episodes, even if those extra steps contain critical information. This bias unfairly favors policies that terminate quickly, leading to skewed rankings where a policy that causes early failure might be rated higher than one that sustains a long, productive interaction.

The Moving Target of Episode Duration

Even when we are not evaluating old data, the fact that a policy influences its own runtime creates a moving target for optimization. In many tasks, the average episode length increases as the agent improves. Bojun Huang argues that failing to account for this dynamic change introduces a significant bias in the policy gradient.

If the gradient estimator treats the number of steps as an independent constant, the "effective" learning rate fluctuates uncontrollably. Borrell & Richter show that in environments with state-dependent termination, the gradient must be explicitly scaled by the expected runtime of the policy. Without this correction, the optimization process can stall or diverge because the agent's evolving behavior changes the very scale of the updates it receives.

Bridging Finite and Infinite Horizons

A fundamental tension exists between the infinite-horizon theory of reinforcement learning and the finite-horizon sampling used in practice. Metelli et al. highlight that truncating trajectories at a fixed length ignores the "infinite tail" of discounted rewards, introducing a systematic bias that practitioners often try to hide by using high discount factors.

To resolve this, researchers have proposed moving toward infinite-horizon formulations that don't rely on arbitrary truncation:

  • Bootstrapping for Long-Term Vision: Cai et al. propose using infinite-horizon value gradients. By backpropagating through learned transition models over an indefinite timeframe, agents can account for long-term consequences without the myopia inherent in short-horizon rollouts.
  • Steady-State Equilibrium: In continuing tasks, Ganesh et al. suggest discarding initial trajectory steps based on the environment's "mixing time." This ensures that updates are based on the long-term stationary distribution of the system rather than transient starting conditions.
  • Imagination as a Buffer: In model-based settings, Ni et al. find that extending the horizon of "imaginary" rollouts to hundreds of steps actually reduces value overestimation. By substituting biased value function estimates with accumulated rewards from long-term simulations, the agent becomes less sensitive to the errors that typically compound in short-term planning.

Stability through Temporal Normalization

When decision intervals or option durations vary, the math of standard RL begins to break down. Wan et al. found that in average-reward RL with temporal abstractions (options), scaling updates by the actual sampled duration of an action leads to extreme instability. Instead, they propose scaling by the estimated expected duration. This acts as a normalization mechanism, preventing noisy, realized time intervals from destabilizing the learning process.

Similarly, in environments with highly variable time steps, Vaartjes & Francois-Lavet suggest that treating every decision step as equal—rather than using time-variable discounting—provides greater training stability. By decoupling the discount factor from elapsed physical time, the agent is protected from the temporal fluctuations that would otherwise cause the gradient to explode.

Ultimately, these works converge on a single intuition: time is not just a coordinate in RL. Whether through expected duration scaling, mixing-time adjustments, or long-horizon imagination, we must explicitly account for how the temporal structure of a trajectory interacts with our estimators to achieve stable, unbiased learning.

Key Foundational Work

Importance Sampling for Fair Policy Selection (2018) — contributed 3 concept elements to this direction.
Imagined Value Gradients: Model-Based Policy Optimization with Tranferable Latent Dynamics Models (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_trajectory_length_bias_59cf812e,
  title = {trajectory length bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/trajectory-length-bias/rdr_59cf812e19774bf1a5ca7771fc1f8831},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_trajectory_length_bias_59cf812e_v1,
  title = {trajectory length bias},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/trajectory-length-bias/rdr_59cf812e19774bf1a5ca7771fc1f8831/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake