This paper tackles a common but often overlooked reality in robotics and complex systems: not every part of a system operates at the same speed. While standard Reinforcement Learning (RL) assumes a "heartbeat" where every action is decided simultaneously at every discrete time step, real-world controllers often have multiple frequencies.
To address this, the authors provide a formal framework and a scalable algorithm, Action-Persistent Actor-Critic (AP-AC), to handle these multi-frequency environments efficiently. By directly optimizing periodic, non-stationary policies, AP-AC significantly outperforms standard RL methods in complex control and traffic simulations.
The Challenge of Multiple Control Frequencies
In standard Markov Decision Processes (MDPs), we assume an agent chooses a full action vector at every discrete time step. However, consider a walking robot: the ankle motors might require high-frequency adjustments (e.g., every 1ms), while the hip joints only need updates every 4ms.
Existing RL approaches usually handle this discrepancy by either:
- Ignoring it: Treating everything as high-frequency, which is computationally expensive and often introduces noise.
- Action Repetition: Forcing all actions to repeat for $k$ steps, which is too restrictive and limits fine-grained control.
- State Augmentation: Adding "time-remaining" or "last-action" information to the state. This leads to exponential complexity as the number of variables grows.
The core challenge is that in these "multiple frequency" settings, the optimal policy is inherently non-stationary. Because some actions are "locked in" while others change, the best action to take in a given state $s$ depends heavily on where the agent is in the overall control cycle. A standard stationary policy—which always outputs the same action distribution for a given state—can be arbitrarily suboptimal.
Methods: Action-Persistent Policy Iteration and Actor-Critic
The authors propose a two-step solution to formally solve this problem:
1. Action-Persistent Policy Iteration (AP-PI)
First, they introduce a theoretical tabular algorithm called AP-PI. They define "c-persistent Bellman operators" that explicitly account for the fact that different actions persist for different durations. They prove that this approach converges to an optimal periodic policy with only a linear increase in complexity relative to the cycle length $L$ (where $L$ is the Least Common Multiple of all control frequencies).
2. Action-Persistent Actor-Critic (AP-AC)
To scale this theory to high-dimensional continuous control, they introduce AP-AC, a deep RL version based on Soft Actor-Critic (SAC).
- Multi-Head Architecture: The actor and critic networks are designed with $L$ "heads." Head $t$ is responsible for the decision-making logic at the $t$-th step of the periodic cycle.
- Head Sharing: While there are $L$ heads, they share the lower layers of the neural network. This allows the agent to learn general features of the environment while specializing the "output" logic for specific phases of the control cycle.
- Sample Efficiency: Crucially, the algorithm uses an "intra-option" style update. Every single transition $(s, a, r, s')$ in the replay buffer is used to update all $L$ heads simultaneously, making it much more efficient than simply training $L$ separate networks.
Datasets and Evaluation
The researchers evaluated their method on two main types of benchmarks to test its robustness and scalability:
Modified MuJoCo Environments: Standard continuous control tasks (Hopper, Walker2d, HalfCheetah, Ant) were modified so that different joints had different "persistence" values. For example, thighs might update every 4 steps, shins every 2, and feet every 1.
SUMO Traffic Control: A realistic traffic simulation of a 2x2 grid where different junctions had different re-timing frequencies (e.g., some junctions updated every 2 seconds, others every 3).
Key Results and Insights
- Superior Performance: AP-AC consistently outperformed standard SAC and SAC with state augmentation (SAC-L).
- Scaling with Complexity: Traditional ways to solve this involve augmenting the state space with the "history" of actions, causing the state space to explode. By using a periodic multi-head architecture, AP-AC keeps complexity linear. Consequently, the performance gap between AP-AC and baselines grew wider as the action dimension increased (e.g., in the "Ant" and "SUMO" tasks).
- Robustness: In the MuJoCo tasks, standard SAC (which ignores persistence during training) suffered severe performance degradation when evaluated in the multi-frequency environment. AP-AC thrived by directly modeling the periodicity.
Limitations and Future Work
While highly effective, the approach has a few limitations that pave the way for future research:
- The "L" Problem: The method relies on $L$, the Least Common Multiple of the frequencies. If you have two controllers with frequencies of 13 and 17, $L$ becomes 221. Having 221 heads in a neural network could become computationally burdensome or memory-intensive.
- Fixed Frequencies: The paper assumes the control frequencies are known and fixed by the system designer. A natural next step would be an agent that learns the optimal frequency for each action variable dynamically.
- Multi-Agent Extension: The authors suggest that this framework could be extended to multi-agent RL, where different agents naturally operate on different time scales.
Summary updated