Abstract
Standard reinforcement learning (RL) agents often struggle in environments with periodic dynamics or rewards, as they treat time-dependent states as ambiguous or fail to generalize across a cycle. This paper introduces Phase-Parametric Policies, a novel architecture where neural network weights are smooth functions of a cyclic phase parameter. By using spline interpolation between learned control weights, the model enforces a periodic structure directly into the policy. Evaluated on tasks ranging from grid worlds to bipedal locomotion, the proposed Phase-Functioned Multi-Layer Perceptron (PF-MLP) significantly outperforms standard baselines in sample efficiency and final performance.
Introduction
In many real-world RL tasks, the optimal action for a specific state is not static but depends on the current "phase" of a cycle. For instance, in robotic locomotion, the correct movement for a leg depends on whether it is in the swing or stance phase of a gait. Similarly, traffic patterns or environmental factors like wind may follow periodic cycles. Standard deep RL agents typically handle this by either ignoring the phase—leading to state ambiguity—or by appending a phase variable as a scalar input. However, high-dimensional networks often struggle to effectively "warp" their entire behavior based on a single input feature.
Phase-Parametric Architecture
The core innovation presented is the Phase-Functioned Multi-Layer Perceptron (PF-MLP). Unlike traditional networks with fixed weights, the PF-MLP treats weights as functions of a phase parameter $p \in [0, 2\pi)$.
Weight Manifolds and Splines
The network learns a set of "control weights" that act as basis vectors for a cyclic manifold. To ensure smooth transitions as the phase changes, the actual weights used in a forward pass are calculated using Catmull-Rom splines to interpolate between these control weights. A modulo operator ensures that the weights at phase $0$ and $2\pi$ are identical, enforcing a perfect cyclic constraint.
This logic is integrated into several standard RL frameworks, including Phase-DQN for discrete actions, Phase-DRQN for partially observable tasks, and Phase-DDPG for continuous robotic control.
Experimental Evaluation
The authors tested the architecture across four environments with varying degrees of complexity, including a grid world with fluctuating rewards and a drone navigation task with cyclic wind. Most notably, the approach was applied to Monoped (Hopper) and Biped (Walker) MuJoCo-based robots.
In these locomotion tasks, the agent does not receive an external clock signal. Instead, it derives a "phase" from its own vertical oscillation (center-of-mass height). This allows the network to synchronize its internal weight configurations with its physical gait rhythm.
Key Insights and Results
The phase-parametric approach offers several advantages:
- Sample Efficiency: Sharing weights via spline interpolation allows experience gained in one phase to inform adjacent phases, accelerating learning compared to training independent networks.
- Superior Performance: In the Walker task, Phase-DDPG achieved an average return of 5124.44, significantly higher than the 3740.77 achieved by the best baseline using phase as a standard input.
- Robustness: The models showed lower performance variance across episodes and faster convergence in tasks with reward ambiguity, such as the Freeway grid world.
Conclusion
Phase-parametric policies provide a powerful prior for RL in cyclic environments. By embedding the concept of phase directly into the network's parameters, agents can learn more stable and efficient policies for periodic tasks. While current implementations require some manual phase engineering, future work aims to automate "Phase Discovery," allowing agents to extract cyclic patterns directly from raw observations.
Summary updated