In traditional off-policy reinforcement learning (RL), the experience replay buffer is treated as a static vault. The agent interacts with the environment, drops its transition trajectories into the buffer, and later samples them under the assumption that what was true in the past remains true now. This assumption holds firm if the environment and the reward function are stationary.
However, modern RL paradigms frequently break this assumption. In preference-based RL, the reward function continuously shifts as human feedback arrives. In unsupervised or multi-task RL, the latent meaning of a "skill" or policy assignment evolves as the agent learns. In imitation learning, initial expert demonstrations can turn out to be suboptimal bottlenecks.
When the lens through which we evaluate history changes, the history itself must change. A growing body of research addresses this challenge through dynamic experience modification—either by retroactively relabeling the metadata (rewards, skills, or policies) of historical transitions or by actively replacing stale trajectories with superior self-generated ones. This approach ensures that off-policy learning remains stable, consistent, and capable of surpassing its original constraints.
Synchronizing History with Evolving Reward Models
When an agent learns from human preferences rather than a hand-engineered reward function, it must iteratively fit a surrogate reward model to pairwise comparisons. This introduces a severe non-stationarity problem: a transition sampled millions of steps ago contains a reward value calculated by an obsolete reward model. If the agent updates its policy using these stale rewards, it optimizes for an incorrect objective, leading to severe instability.
To solve this, frameworks like PEBBLE, introduced by Lee et al., implement retroactive reward relabeling. Whenever the preference-based reward model is refined, the framework updates the reward values of all historical transitions stored in the replay buffer.
[ Human Feedback ] ---> [ Update Reward Model (R_t) ]
|
v
[ Replay Buffer ] ----> [ Relabel Historical Rewards ] ---> [ Policy Optimization ]
(Old Transitions) (R_old -> R_t) (Stable & Up-to-date)
As reviewed by Abdelkareem et al., this mechanism allows offline and interactive preference-based RL to remain effective because the agent's policy is consistently trained against the most current representation of the user's goals.
This synchronization mechanism has been successfully extended across several domains:
- Social Navigation: Wang et al. utilized dynamic reward relabeling in their FAPL framework to ensure that a robot navigating crowded spaces continuously aligns its entire history of explored states with evolving human comfort preferences.
- Multi-Objective Optimization: In the Pb-MORL framework proposed by Mu et al., historical experience is periodically updated with revised reward estimates to maintain consistency across multiple competing objectives.
- Automated Reward Search: When Large Language Models (LLMs) are used to search for optimal reward functions over time, discarding data between reward iterations is highly sample-inefficient. The LaRes framework, developed by Li et al., leverages a shared experience buffer where stored rewards are relabeled whenever the LLM updates the reward structure, preventing the need to restart training from scratch.
Maintaining Semantic Consistency in Skills and Policies
The need for dynamic relabeling extends beyond rewards; it applies equally to latent intents and policy assignments.
In unsupervised reinforcement learning, agents often learn task-agnostic behaviors by conditioning their policies on a latent "skill" vector. However, as the policy updates, the actual behavior triggered by a specific skill vector shifts—a phenomenon known as semantic drift. To combat this, Park et al. developed GenDa, which introduces dynamic skill relabeling. By updating the latent labels of stored trajectories using the agent's most recent representation, they prevent the policy from learning from stale data where the mapping between the skill vector and the resulting behavior has decoupled.
A parallel issue arises in multi-task reinforcement learning when an agent "borrows" behaviors from other tasks. In the Cross-Task Policy Guidance (CTPG) framework by He et al., historical transitions are dynamically re-labeled to reflect which task-specific policy would have most likely generated that behavior under current conditions. This correction mechanism accounts for the non-stationarity of the guiding policies, stabilizing off-policy training. Similarly, Xia et al. demonstrated in SkillRL that allowing a skill library to dynamically co-evolve and refine itself alongside the agent's policy significantly outperforms static skill architectures.
Breaking Performance Ceilings via Dynamic Replacement
While relabeling modifies the metadata of existing transitions, a complementary strategy focuses on replacing entire trajectories within the buffer to break performance ceilings.
In imitation learning and self-imitation learning, agents are typically seeded with expert demonstrations. However, if these demonstrations are suboptimal or contain human errors, they act as a hard ceiling on the agent's potential.
To transform imitation from a rigid copying task into a bootstrapping process, several researchers have proposed dynamic demonstration zones. In Dynamic Experience Replay (DER), Luo et al. continuously update a dedicated demonstration buffer with the agent's own successful trajectories discovered during training. This creates a self-improving curriculum where the "expert" data evolves alongside the agent's capabilities.
[ Initial Suboptimal Expert Data ]
|
v
[ Demonstration Buffer ] <====== [ Agent Discovers Better Trajectory ]
| |
v | (Overwrites)
[ Policy Training ] ------------------------------+
This bootstrapping mechanism is highly effective at filtering out low-quality guidance:
- Surpassing Suboptimal Teachers: The SAIL framework by Zhu et al. and the MAGA-ISIL framework by Fang et al. both dynamically replace initial suboptimal demonstrations with superior agent-generated trajectories once environment rewards confirm the agent has found a better path.
- Robustness to Imperfect Data: Chao Li shows that an adaptive experience replay mechanism, which replaces initial expert data with superior online experiences as they occur, prevents performance collapses and accelerates self-imitation learning.
Taking this concept a step further, Du et al. introduced "Lucid Dreaming for Experience Replay" (LiDER). Instead of waiting for the agent to accidentally discover a better trajectory online, LiDER actively "refreshes" past experiences. It resets the environment to a previously visited historical state and re-runs the scenario using the current, improved policy. If this new trajectory yields a higher return than the historical memory, the buffer is updated with the higher-quality data.
Why It Works: Restoring the Off-Policy Contract
The core reason dynamic experience modification works is that it restores the fundamental mathematical contract of off-policy reinforcement learning.
Off-policy algorithms like Q-learning or Actor-Critic rely on the Bellman equation to propagate values backward through time. This propagation assumes that a transition tuple $(s, a, r, s')$ represents a true, unchanging relationship between an action and its outcome under the current objective.
When rewards shift or skills drift, that tuple becomes a lie; it represents a relationship from a different task or an obsolete understanding. By dynamically relabeling rewards and skills, the agent aligns its entire history with its present objective. By dynamically replacing suboptimal trajectories, the agent ensures its exploration gradients are pulled toward ever-improving horizons. Ultimately, treating the experience replay buffer as a fluid, evolving entity allows agents to learn continuously and efficiently in non-stationary worlds.
Key Foundational Work
PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training (2021) — contributed 5 concept elements to this direction.
Dynamic Experience Replay (2019) — contributed 2 concept elements to this direction.