PACE: Parameter Change for Unsupervised Environment Design

Fang Yuan, Quanjun Yin, Siqi Shen, Yuxiang Xie, Junqiang Yang, Long Qin, Junjie Zeng, Qinglun Li

TL;DR

Unsupervised Environment Design (UED) helps reinforcement learning agents generalize by automatically generating training curricula, but current methods rely on noisy, computationally expensive proxy signals to evaluate environment utility. This paper introduces PACE (Parameter Change Environment Design), which measures an environment's value by looking directly at the magnitude of policy parameter changes it induces. This simple, intrinsic metric provides a low-variance, zero-extra-rollout signal of actual learning progress, leading to state-of-the-art generalization performance on MiniGrid and Craftax benchmarks.

What Problem Does This Paper Address?

Reinforcement Learning (RL) agents are notoriously prone to overfitting to their training environments, often failing spectacularly when deployed in out-of-distribution (OOD) scenarios. Unsupervised Environment Design (UED) attempts to solve this by dynamically generating a curriculum of environments (or "levels") to challenge the agent.

However, the core bottleneck in UED is evaluation: how do we accurately measure whether a generated level is at the sweet spot for an agent's current learning capacity? Existing methods rely on indirect, external proxies like regret (the performance gap between the agent and an expert) or Generalized Advantage Estimation (GAE) errors. These proxies suffer from high estimation bias, massive variance, or require expensive extra environment rollouts, frequently causing the curriculum to trap the agent in either unsolvable "panic zones" or trivial "comfort zones."

Methods

The authors propose Parameter Change Environment Design (PACE), shifting the evaluation paradigm from looking outward at rewards and value functions to looking inward at the agent's neural network weights.

  1. Direct Intrinsic Evaluation: When a new level is proposed, the agent performs a temporary, provisional policy update based on a collected trajectory. PACE scores the level using the squared $\ell_2$ norm of the resulting parameter change: $$S(l) = \frac{1}{\alpha} \|\theta_{new} - \theta_{old}\|_2^2$$
  2. Theoretical Grounding: Using a first-order Taylor expansion of the policy optimization objective, the authors mathematically prove that under local gradient-aligned updates, the actual improvement in the policy objective is directly proportional to this squared parameter change.
  3. Curriculum Management: High-scoring levels (those causing significant parameter shifts) are saved into a level buffer. During training, levels are sampled from this buffer using rank-based prioritization combined with a staleness-aware mechanism to maintain coverage and prevent catastrophic forgetting.
Figure 2. Overview of the PACE framework. At training step k, the agent policy is πθk, and we set θold = θk. The framework consists of two decoupled components: level scoring (orange) and policy training (blue). In the level scoring stage, a level l is generated by a random level generator, and a trajectory τ is collected using the current policy πθold. A provisional policy update is then performed to obtain temporary parameters θnew, without writing back to the policy. The 3D illustration depicts the objective landscape induced by level l, highlighting that improvement in the policy optimization objective correlates with the magnitude of the policy parameter change. Motivated by the first-order analysis in Section 3.1, PACE uses policy parameter change as an intrinsic signal of realized learning progress, and uses this signal to construct the environment value of level l. Specifically, PACE assigns a level score S(l), defined in Eq. (6), based on the magnitude of the induced parameter update. If S(l) exceeds the lowest score in the level buffer Λ, the new level replaces that entry. In the training stage, levels are sampled from the level buffer Λ according to a score-based replay distribution and used to update the policy, advancing the policy from πθk to πθk+1. The updated parameters are then assigned to θold for subsequent level scoring. Overall, PACE iteratively leverages policy parameter change as a learning progress signal to select training levels that match the agent current learning capacity.

Datasets

The method was validated on two prominent reinforcement learning benchmarks:

  • MiniGrid: A gridworld navigation environment. Agents were trained on $15\times15$ mazes and evaluated on 12 human-designed OOD levels (e.g., FourRooms, Labyrinth) as well as a massive, procedurally generated $51\times51$ maze called PerfectMaze.
  • Craftax: A highly complex, open-ended RL benchmark inspired by NetHack and Crafter. It features non-stationary task distributions, multi-floor worlds, and long-horizon exploration objectives, tested under a massive budget of 1 billion environment steps.

Key Insights and Ideas

  • The Agent is the Best Metric: Traditional UED treats environment dynamics or reward structures as the source of truth for difficulty. PACE argues that an environment is only valuable if it forces the agent's internal representations (weights) to actually adapt.
  • Geometry Maps to Progress: The realization that objective function improvement correlates tightly with the geometric distance traveled in parameter space provides an elegant mathematical shortcut around expensive Monte Carlo return estimations.
  • Zero-Cost Evaluation: Because parameter differencing happens entirely in memory using standard tensor arithmetic, PACE eliminates the need for extra environment interactions during the scoring phase, dropping evaluation complexity from $O(k \cdot T \cdot P)$ to a negligible $O(P)$.

Key Results

  • MiniGrid Generalization: On the 12 human-designed OOD levels, PACE achieved an Interquartile Mean (IQM) solve rate of 96.4% and a tiny Optimality Gap of 17.2%, vastly outperforming the strongest baseline, PLR (80.8% IQM, 29.9% gap).
  • Robust Scaling: On the massive $51\times51$ PerfectMaze, PACE achieved a 72.3% solve rate, demonstrating superior generalization to large-scale environments.
  • Craftax Mastery: In the complex, long-horizon Craftax environment, PACE maintained steady learning progress without plateauing early, achieving the highest final evaluation reward (31.2) and outperforming all baselines across aggregate metrics.
Figure 4. Aggregate OOD test performance on maze environments. Overall, PACE consistently dominates the performance–robustness trade-off, achieving both higher central performance and tighter performance dispersion across OOD maze environments.

Limitations and Future Work

  • Simple Level Generation: PACE currently relies on a basic random level generator or simple mutations to propose new environments. Integrating it with advanced generative models (like adversarial RL teachers or LLM-based level designers) remains an open direction.
  • Idealized Theoretical Assumptions: The theoretical proof assumes a single-step gradient update, whereas practical RL algorithms like PPO use multiple epochs and minibatches, meaning the parameter change is an approximation of the true objective improvement.
  • Baseline Exclusions: The authors focused on replay-based methods and did not directly compare PACE against multi-agent teacher-based methods (like PAIRED or MBeDED) due to their fundamentally different computational models, leaving a systematic comparison to future work.

Summary updated

Abstract

Unsupervised Environment Design (UED) offers a promising paradigm for improving reinforcement learning generalization by adaptively shaping training environments, but it requires reliable environment evaluation to remain effective. However, existing UED methods evaluate environments using indirect proxy signals such as regret, value-based errors, or Monte Carlo, which suffer from bias, high variance, or substantial computational overhead and fail to reflect agent realized learning progress. To address these limitations, we propose Parameter Change Environment Design (PACE), which evaluates an environment through the policy parameter change induced by training on that environment, directly grounding environment selection in realized learning progress. Specifically, PACE assigns environment value using a first-order approximation of the policy optimization objective, where the improvement induced by an environment is proportional to the squared L2 norm of the corresponding parameter update, enabling low-variance and computation-efficient evaluation without additional rollouts. Experiments on MiniGrid and Craftax show that PACE consistently outperforms established UED baselines, achieving higher IQM and smaller Optimality Gap on OOD evaluations, including an IQM of 96.4% and an Optimality Gap of 17.2% on MiniGrid.
BibTeX
@inproceedings{yuan2026pace,
  author = {Fang Yuan and Quanjun Yin and Siqi Shen and Yuxiang Xie and Junqiang Yang and Long Qin and Junjie Zeng and Qinglun Li},
  title = {{PACE: Parameter Change for Unsupervised Environment Design}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=OVlAgNFMfS},
}
Figure