In standard Reinforcement Learning (RL), we often assume the "best" strategy is stationary—meaning that if an agent finds itself in a specific state, it should always take the same action, regardless of whether it is the first step of the mission or the last. This assumption works well for infinite-horizon problems where the goal is to find a stable, long-term equilibrium.
However, real-world tasks often have deadlines. Whether managing a hospital's inventory, trading stocks before the market closes, or controlling a robot with a limited battery, the "time remaining" is a critical factor. In these finite-horizon settings, the optimal strategy is inherently non-stationary. As Sara Schön and Sara Klein argue in Beyond Stationarity: Convergence Analysis of Stochastic Softmax Policy Gradient Methods, an agent must be able to select different actions for the same state depending on how much time is left to complete the task.
The Necessity of Temporal Context
The core intuition behind non-stationary policies is that the utility of a state changes as the horizon approaches. In a finite-horizon Markov Decision Process (MDP), a state that is valuable at the beginning of an episode might be worthless at the very end if there isn't enough time left to reap its rewards.
To capture this, researchers like Vivek VP in Finite Horizon Q-learning: Stability, Convergence and Simulations propose using time-indexed value estimates. Instead of a single Q-table, the agent maintains a sequence of tables—one for each discrete time step. This allows the agent to realize that an aggressive action might be optimal when time is running out, even if that same action would be too risky at the start of the episode.
This temporal flexibility is even more vital in environments with inhomogeneous transitions, where the rules of the world itself change over time. As Elynn Y. Chen, Michael Jordan, and Sai Li note in Transfer Q-learning for finite-horizon Markov decision processes, modeling these time-varying dynamics is essential for applications like healthcare or personalized analytics, where the underlying system behavior evolves throughout the duration of the observation.
Learning Policies Backwards
If the optimal policy depends on the time remaining, how do we train it efficiently? A powerful approach is to leverage the principles of dynamic programming.
In Policy Search by Dynamic Programming (PSDP), J. Andrew Bagnell, Sham M. Kakade, and Andrew Y. Ng demonstrate that we can build a non-stationary policy by working backwards from the end of the horizon. By solving for the best action at the final time step $T$, then the second-to-last step $T-1$, and so on, we can construct a sequence of policies that are provably effective even in partially observable environments (POMDPs).
Bruno Scherrer expands on this in On the Performance Bounds of some Policy Search Dynamic Programming Algorithms with Non-Stationary Direct Policy Iteration (NSDPI). Instead of updating a single stationary rule, NSDPI "prepends" a new greedy policy to the existing sequence at each iteration. This sequencing mechanism allows the algorithm to achieve tighter theoretical performance bounds than standard stationary methods by explicitly accounting for the temporal structure of the problem.
Balancing Safety and Urgency
Non-stationarity is not just about maximizing rewards; it is often a requirement for safety. In constrained RL, an agent must stay within certain safety bounds while pursuing a goal.
Akhil Agnihotri and colleagues introduced e-COP in e-COP : Episodic Constrained Optimization of Policies to address this. They argue that stationary policies often fail in constrained finite-horizon tasks because they cannot effectively balance long-term safety requirements with the immediate pressure of a deadline. By using time-dependent representations, the agent can learn to be more cautious when it has plenty of time and more decisive as the horizon closes, ensuring that safety constraints are met throughout the entire episode.
Similarly, Soumyajit Guin emphasizes in A policy gradient approach for Finite Horizon Constrained Markov Decision Processes that ignoring the urgency of a deadline leads to suboptimal performance in time-critical applications.
Implementation via Time-Dependent Architectures
To put these ideas into practice, researchers have moved toward time-dependent actor architectures. In Dynamic Energy Dispatch in Isolated Microgrids Based on Deep Reinforcement Learning, Lei Lei and Wei Xiang describe systems where the agent uses distinct neural network parameters for each discrete time step within an operational window (e.g., a 24-hour cycle).
While this approach can be memory-intensive—as noted by Yuda Song and Drew Bagnell in Hybrid Reinforcement Learning from Offline Observation Alone—it allows the agent to learn specialized decision rules tailored to specific phases of a task. By treating time as a fundamental dimension of the policy, these agents can navigate the complex trade-offs inherent in any problem where the clock is ticking.