Eliminating High Frequency Oscillations in Reinforcement Learning

61 concepts60 papers2 proposals

When a deep reinforcement learning (RL) agent is trained to control a physical system—such as a humanoid robot, a drone, or an autonomous vehicle—it often achieves remarkable success in simulation. Yet, deploying this same policy onto physical hardware frequently results in a catastrophic surprise: the robot shakes violently, its motors overheat, and its joints suffer severe mechanical wear.

This discrepancy stems from high-frequency control jitter (or action fluctuation). While model-free RL is exceptional at maximizing cumulative rewards, it is notoriously blind to the physical realities of hardware unless explicitly instructed otherwise. Understanding why these oscillations occur and how to structurally eliminate them is vital to bridging the sim-to-real gap.


Why Neural Policies Oscillate

To understand the solution, we must first diagnose why standard reinforcement learning actively encourages jittery behavior. The root causes are a combination of simulation idealism, action independence, and neural network sensitivity.

The Bang-Bang Incentive

In an idealized simulator, physics is simplified. Actuators can instantly change torque without stripping gears, drawing excessive current, or damaging components. As shown by Seyde et al. in Is Bang-Bang Control All You Need? Solving Continuous Control with Bernoulli Policies, standard continuous control benchmarks rarely penalize high-frequency changes or jerk.

This absence of physical constraints incentivizes policies to adopt "bang-bang" control strategies—rapidly switching between maximum and minimum actuator limits to minimize tracking errors as fast as possible. As Schäfer et al. observe in The Crucial Role of Problem Formulation in Real-World Reinforcement Learning, standard reward formulations prioritize immediate error reduction over physical longevity, making aggressive, oscillatory control mathematically optimal in a virtual world.

Step-Based Independence and Noise

Standard RL algorithms typically treat control as a sequence of independent, absolute decisions. At each time step, the policy evaluates the current state and outputs an absolute action command. Furthermore, during training, algorithms inject independent random noise at every single time step to explore the environment.

As argued by Rückstieß et al. in Exploring parameter space in reinforcement learning, this step-level perturbation introduces high-frequency noise directly into the action trajectory. Because the policy learns to map states to absolute values independently at each step, it lacks any inherent mechanism to ensure that action $a_t$ is smoothly related to action $a_{t-1}$.

High Non-Linear Sensitivity

Deep neural networks are highly non-linear function approximators. Without explicit regularization, a tiny fluctuation in the perceived state—such as sensor noise or a slight tracking error—can result in a disproportionately large jump in the control output.

Zuo et al. describe this vulnerability in Adversarially Regularized Policy Learning Guided by Trajectory Optimization, noting that this extreme sensitivity causes the controller to overreact to minor environmental variations, generating jerky, unstable motions that can cause physical hardware to diverge from safe operating parameters.


Redefining the Action Space

One of the most effective ways to eliminate high-frequency jitter is to change what the policy outputs. Instead of forcing a neural network to output raw, low-level actuator commands at maximum frequency, researchers restructure the action space to build in temporal consistency.

Residual Control

Instead of predicting absolute actions, the policy can be reframed to predict incremental changes. In Visual Reinforcement Learning with Residual Action, Liu et al. demonstrate that modeling the difference between consecutive actions simplifies the learning objective. By optimizing for "residual" adjustments, the policy is naturally biased toward temporal smoothness, as a zero output from the network means maintaining the previous action rather than abruptly stopping.

Movement Primitives and Trajectory Curves

Another approach is to decouple the policy from low-level execution entirely. Otto et al. achieve this in Deep Black-Box Reinforcement Learning with Movement Primitives by using episode-based control to optimize movement primitive parameters rather than step-by-step forces.

Similarly, in PathRL: An End-to-End Path Generation Method for Collision Avoidance via Deep Reinforcement Learning, Yu et al. configure the network to output smooth Bezier curve control points rather than raw motor signals. Because the underlying mathematical curves are inherently continuous, the resulting physical trajectories are completely free of high-frequency velocity spikes.


Enforcing Smoothness via Constraints and Architecture

When modifying the action space is not an option, researchers instead constrain the policy's mathematical properties or bake smoothing mechanisms directly into the neural network architecture.

Lipschitz-Constrained Policies

To prevent a neural network from overreacting to small state changes, we can bound its maximum rate of change—a property known as the Lipschitz constant. If a policy is Lipschitz-constrained, a small change in the input state can only ever result in a proportionally small change in the output action.

In Learning Smooth Humanoid Locomotion through Lipschitz-Constrained Policies, Chen et al. apply a differentiable gradient penalty during reinforcement learning. This mathematical boundary eliminates the need for tedious, robot-specific manual tuning of smoothness rewards, ensuring stable "sim-to-real" deployment on complex humanoid hardware.

Architectural Low-Pass Filters

Rather than relying on loss functions to penalize jitter after the fact, recent architectures embed physical smoothing layers directly into the neural network's structure.

[State Input] ---> [Fourier/ODE Filter Layer] ---> [Lipschitz Controller] ---> [Smooth Action]
                         (Removes Noise)             (Bounds Sensitivity)

Wang et al. introduce this in ODE-based Smoothing Neural Network for Reinforcement Learning Tasks by embedding the physics of ordinary differential equations (ODEs) and low-pass filters directly into the neurons of the policy.

Expanding on this concept, Song et al. present LipsNet++: Unifying Filter and Controller into a Policy Network, which explicitly decouples noise filtering from policy smoothing by combining a Fourier filter layer with a Lipschitz-constrained controller. Because the network architecture itself is mathematically incapable of expressing high-frequency oscillations, the output control signals remain clean, hardware-safe, and highly robust to real-world sensor noise.

Temporal Hierarchies

Finally, control signal jitter can be mitigated by separating the timescales of decision-making. Biological systems rarely operate on a single high-frequency loop; instead, they separate high-level planning from low-level reflexes.

Patel et al. implement this principle in Temporally Layered Architecture for Adaptive, Distributed and Continuous Control by stacking "fast" and "slow" controllers. By allowing the agent to adapt its response frequency—acting rapidly only when an environmental disruption demands it—the system avoids making minute, unnecessary adjustments at every single available timestep, drastically reducing actuator wear and power consumption.

Key Foundational Work

Fitted Q-iteration by Advantage Weighted Regression (2008) — contributed 1 concept elements to this direction.
Exploring parameter space in reinforcement learning (2010) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 57 papers
  2. Version 1 · 2026-10-04 · 56 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_control_signal_jitter_bbae1ac9,
  title = {control signal jitter},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/control-signal-jitter/rdr_bbae1ac93fe54cf1b65bff8afe94eeff},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_control_signal_jitter_bbae1ac9_v2,
  title = {control signal jitter},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/control-signal-jitter/rdr_bbae1ac93fe54cf1b65bff8afe94eeff/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake