In standard Reinforcement Learning (RL), algorithms typically rely on a finite time horizon $H$ or a discount factor $\gamma < 1$ to ensure that value functions remain bounded and learning terminates. However, many real-world sequential decision-making tasks are inherently goal-oriented: an agent must navigate an environment until it reaches a specific terminal state, minimizing the total cost accumulated along the way. This is formalized as the Stochastic Shortest Path (SSP) problem.
Historically, proving that an RL agent could efficiently learn an optimal policy in an SSP required a restrictive theoretical crutch: the assumption that every action taken in a non-goal state incurs a strictly positive minimum cost ($c(s,a) \ge c_{min} > 0$). As noted by Luo et al., this positive cost constraint acts as a mathematical engine that forces the agent toward the goal. If an agent wanders indefinitely, its accumulated cost explodes to infinity, allowing the learning algorithm to easily penalize and discard inefficient policies.
Recently, a major thrust of research has focused on eliminating this dependency, allowing algorithms to operate in realistic environments where transitions can cost zero—or even be negative.
The Pathology of Zero-Cost Transitions
Why does allowing $c_{min} = 0$ break traditional RL algorithms? The core issue is a loss of identifiability and control. Without a guaranteed penalty for every step taken, the environment's feedback loop suffers from several theoretical and practical failures:
- Mathematical Unlearnability: Tirinzoni et al. settled a long-standing open question by proving that if actions can incur zero cost, the sample complexity required to identify an optimal policy can become completely unbounded. Without $c_{min} > 0$, an agent cannot effectively bound the time required to reach the goal, making it impossible to distinguish between a policy that is safely idling in a zero-cost loop and one that will fail to reach the goal entirely.
- Algorithmic Divergence and Undecidability: In planning and heuristic search, Hansen et al. demonstrated that zero-cost actions create cycles that prevent standard algorithms from computing reliable sub-optimality bounds. When extended to partially observable settings (POMDPs), Chatterjee et al. proved that finding a minimum-cost policy with absolute certainty becomes mathematically undecidable if zero or negative costs are permitted.
- Sub-optimal Regret Scaling: When zero-cost transitions are present, the efficiency of online learning degrades significantly. Tarbouriech et al. and Cohen et al. showed that traditional frameworks see their regret bounds worsen from the ideal square-root rate ($\sqrt{K}$) relative to the number of episodes to a much slower power law (such as $K^{2/3}$).
If an agent can accumulate infinite profit or wander endlessly without penalty—a risk highlighted in declarative planning frameworks by Eiter et al.—standard value iteration and exploration mechanisms collapse.
Breaking the Dependency: How It Works
To design algorithms that achieve optimal performance without relying on $1/c_{min}$, researchers had to replace the global cost penalty with finer, structure-aware mathematical tools.
Traditional SSP Bounds: Regret ∝ 1 / c_min ───► Explodes when c_min → 0
Modern Cost-Agnostic: Regret ∝ Variance ───► Stable even when c_min = 0
1. Variance-Sensitive Bernstein Bounds
The primary breakthrough for removing $c_{min}$ involves changing how algorithms handle uncertainty. Instead of using coarse, worst-case Hoeffding confidence intervals to bound an agent's cost-to-go, modern algorithms use variance-sensitive Bernstein confidence sets.
Rosenberg et al. leveraged this intuition to achieve a near-optimal regret bound of $\tilde{O}(B_* \sqrt{S|A|K})$, where $B_*$ is the maximum expected cost of the optimal policy. By tightly tracking the variance of the transitions and costs rather than just their absolute magnitudes, the algorithm can detect when it is stuck in a loop based on statistical fluctuations, completely bypassing the need for a strictly positive minimum cost.
2. Implicit Finite-Horizon Approximations
Another elegant mechanism is to dynamically simulate a time horizon without explicitly forcing a hard cutoff. Chen et al. introduced implicit finite-horizon approximation, a technique that mathematically maps an infinite-horizon SSP into a sequence of finite-horizon problems. This allows model-free algorithms to maintain minimax optimality and handle zero-cost actions gracefully, because the implicit horizon naturally bounds the maximum steps the algorithm considers during an optimization update.
3. High-Order Moment Estimators
When scaling to massive state spaces that require function approximation, tracking simple variance is often insufficient. To solve this, Di et al. developed a recursive high-order moment variance estimator for linear mixture SSPs. By recursively bounding higher-order statistical moments of the value function, their algorithm eliminates polynomial regret growth relative to the minimum cost, matching theoretical lower bounds even when zero-cost state-action pairs are widespread.
Generalizing Across Settings
The push toward cost-agnostic learning has forced a re-evaluation of several foundational RL paradigms:
- Function Approximation: In large-scale environments, Vial et al. demonstrated that traditional linear function approximations suffer severe degradation as $c_{min} \to 0$. They proposed Optimistic Approximate Fixed Points (OAFPs) to stabilize stationary policies under these fragile cost structures.
- Offline Learning: When learning purely from historical datasets rather than active exploration, Yin et al. established the first instance-dependent bounds for offline SSPs, highlighting that a unified framework capable of handling zero-cost states is vital for stable offline value iteration.
- Complex Dynamics: The challenge extends beyond single-agent paths. For instance, Perez et al. showed that model-free learning in Branching MDPs (where an agent can split into concurrent entities) traditionally requires strictly positive costs to prevent non-terminating populations from breaking convergence proofs. Meanwhile, Teichteil-Königsbuch proposed dual-criterion optimization to balance safety and cost when goals are not guaranteed to be reachable.
Ultimately, this research direction shifts the field away from rigid transition-based metrics. As argued by Dai et al., replacing blind transition diameters with reward-aware metrics like the Maximum Expected Hitting Cost (MEHC) allows theoretical bounds to reflect the true informativeness of an environment's feedback signal. By focusing on variance and reward density rather than arbitrary minimum step costs, modern goal-oriented RL achieves both mathematical rigor and practical flexibility.
Key Foundational Work
Stochastic Safest and Shortest Path Problems (2012) — contributed 1 concept elements to this direction.
Answer Set Planning Under Action Costs (2011) — contributed 1 concept elements to this direction.