Introduction
Reinforcement Learning (RL) has seen significant success in teaching robots complex behaviors using gradient-based methods like Proximal Policy Optimization (PPO). However, these methods often struggle with the "last mile" of policy refinement—the transition from a competent policy to a near-optimal one. Researchers have introduced Triangular-Distribution Evolution Strategies (TD-ES), a two-stage pipeline that leverages bounded triangular perturbations to achieve superior precision and stability in robotic manipulation tasks.
The Challenge of Policy Refinement
In the final stages of training, the gradient signal becomes increasingly small and noisy. Standard gradient-based RL can plateau or even suffer from "catastrophic regressions," where the policy's performance suddenly drops. While Evolution Strategies (ES) offer a gradient-free alternative, they typically rely on Gaussian noise. Because Gaussian distributions have infinite support, they frequently sample parameter changes far from the current solution, leading to inefficient exploration and high variance in the refinement phase.
TD-ES: Bounded Exploration for Stability
The core innovation of TD-ES is the replacement of Gaussian noise with a triangular distribution. This shift introduces a "hard radius" or bounded support, ensuring that the algorithm only explores within a specific distance from the current policy parameters.
This bounded nature creates a "soft trust region." Since the distribution peaks at zero, most samples remain very close to the anchor policy, while the hard boundaries prevent wild, destabilizing updates. This approach provides a compute-light gradient proxy that is robust to outliers and easily parallelizable.
Visualizing Convergence
The impact of bounded support is most evident when comparing TD-ES to Vanilla ES on a reward landscape. While Vanilla ES continues to explore broadly even in later generations, TD-ES focuses its search, converging tightly around the local optima.
Experimental Evaluation
The researchers evaluated TD-ES using the Isaac Lab physics simulator across three robotic manipulation tasks:
- Lift-Cube: Basic grasping and positioning.
- Open-Drawer: A task requiring specific directional force.
- Peg-Insert: A high-precision task with sub-millimeter clearances (0.114mm).
The training pipeline starts with PPO to reach a "competent" level (40-60% success) before handing off to TD-ES for final refinement.
Key Results and Insights
TD-ES demonstrated a clear advantage in both performance and reliability:
- Higher Success Rates: TD-ES achieved an aggregate Interquartile Mean (IQM) success rate of 85.0%, outperforming both Gaussian ES (73.7%) and PPO-only baselines (67.2%).
- Variance Reduction: The use of triangular perturbations resulted in an 83.1% average variance reduction compared to Gaussian noise, leading to much smoother learning curves.
- Precision: In the "Open-Drawer" task, TD-ES reached a 97.8% success rate, drastically reducing the performance variance seen in other methods.
The study highlights that localized exploration is critical for the refinement stage. By restricting the search space, TD-ES prevents the robot from "forgetting" established skills while seeking marginal improvements.
Limitations and Future Work
While TD-ES excels at refinement, it remains dependent on the initial "anchor" policy provided by PPO. If the first stage fails to find the correct reward basin, the second stage cannot compensate. Future research may explore alternative bounded distributions, such as Beta or truncated Gaussian distributions, and investigate scaling the method to more complex, long-horizon tasks.
Summary updated