Broad-Persistent Advising in Interactive Reinforcement Learning via Latent State Encoding

Zhuoran Liang, Francisco CruzarXiv 2026 · arXiv

TL;DR

Encoder BPA improves Interactive Reinforcement Learning by using neural network encoders to store and reuse expert advice across similar situations. By mapping state histories into a latent space rather than relying on simple clustering, the agent can more accurately retrieve relevant guidance, significantly accelerating learning in complex and partially observable environments.

What Problem Does This Paper Address?

In Reinforcement Learning (RL), an agent learns by trial and error. Interactive RL (IntRL) speeds this up by allowing a "teacher" (human or expert agent) to provide advice. However, traditional IntRL is often "forgetful"—it uses a piece of advice once and discards it.

Previous attempts to fix this, like Broad-Persistent Advising (BPA), used K-means clustering to group similar states so advice could be reused. But K-means fails in high-dimensional spaces (the "curse of dimensionality") and often produces clusters that are too coarse to distinguish between different required behaviors. This paper addresses the need for a more sophisticated way to generalize and retrieve advice in complex, real-world-like robotic simulations.

Methods

The authors introduce Encoder BPA, which replaces discrete clustering with representation learning. The workflow consists of two main phases:

  1. Encoder Pre-training: A teacher network interacts with the environment to collect data. An encoder (either an LSTM or a Mamba architecture) is trained to map a sequence of the 16 most recent states into a compact latent representation ($z_t$). The training objective is to predict the teacher's action from this latent vector.
  2. Latent Space Advice Retrieval: During the actual RL training, the agent stores (latent vector, action) pairs in a persistent memory. When the agent needs guidance but no live teacher is available, it performs a search:
  3. Similarity Filtering: It finds stored memories with a high cosine similarity to the current latent state.
  4. Top-(k) Aggregation: It selects the $k$ most similar memories.
  5. Consistency Condition: To ensure stability, it only reuses the advice if the candidate actions are consistent (e.g., a majority vote for discrete actions or a low mean distance for continuous actions).
Figure 2: The PPR mechanism in Encoder BPA. In En- coder BPA, a trained encoder is used to map a state sequence into a latent representation at every time step. When external advice is available, the learning agent ap- plies the advised action and stores the (latent, action) pair in the persistent memory. When no external advice is available, the associated action may be reused in la- tent space advice retrieval.

Datasets

The researchers evaluated their approach on four classic and complex RL benchmarks:

  • Cart-Pole: A discrete action task involving balancing a pole.
  • Inverted Pendulum: A continuous version of the balancing task.
  • MuJoCo Ant: A high-dimensional (105-dim state space) task where a four-legged robot must learn to walk.
  • POMDP Ant: A "Partially Observable" version of the Ant task where certain velocity data is hidden, forcing the agent to rely on state history.
Figure 3: The environments used in the experiments.

Key Insights and Ideas

  • History Matters: By encoding a sequence of states rather than just the current frame, the encoder captures temporal features. This is crucial for POMDP environments where a single snapshot doesn't tell the whole story.
  • Behavioral Features vs. Raw Pixels: The encoder learns to group states based on the action required, not just how the state looks. The authors found a strong negative correlation between latent similarity and action distance, proving that "similar" latents actually lead to similar expert actions.
  • Early-Stage Catalyst: Encoder BPA is most effective at the beginning of training. It provides a "warm start" by maximizing the utility of every piece of teacher advice, allowing the agent to reach higher reward levels much faster than baseline methods.

Key Results

  • Superior Efficiency: In the MuJoCo Ant and POMDP Ant environments, Encoder BPA (using both LSTM and Mamba) achieved significantly higher rewards in the early stages of training compared to original BPA and standard IntRL.
  • Higher Reuse Frequency: Encoder BPA reused advice far more often than previous methods. For example, in Cart-Pole, it achieved ~28,000 reuses compared to only ~6,000 for the original K-means BPA.
  • Robustness to Noise: Even when the teacher's accuracy was reduced to 90%, the latent retrieval mechanism remained effective at identifying and applying the available guidance.
Comparison of different IntRL approaches. Encoder BPA (Mamba/LSTM) shows the best early training performance across various tasks.

Limitations and Future Work

  • Late-Stage Convergence: In continuous action environments, the "averaging" strategy for retrieved advice can sometimes introduce inaccuracies in later training stages, occasionally hindering final convergence.
  • Hyperparameter Sensitivity: The system relies on specific thresholds for similarity ($\tau$) and consistency ($\delta$), which currently require manual tuning for different environments.
  • Future Directions: The authors suggest using contrastive learning to make the latent space even more structured and exploring dynamic retrieval conditions that allow the agent to rely less on stored advice as it becomes more confident in its own policy.

Summary updated

Abstract

Deep Interactive Reinforcement Learning (Deep IntRL) accelerates reinforcement learning training by incorporating external advice. However, conventional Deep IntRL uses each piece of advice only in the current step and discards it after a single interaction. Persistent Advising and Broad-Persistent Advising (BPA) extend this process by storing advice in persistent memory and reusing it in similar states through state generalisation based on K-means clustering. This research proposes Encoder BPA, which employs a latent encoder to retrieve and reuse advice. In this framework, a trained encoder is used to map a state sequence into a latent representation at every time step. Advice can be retrieved through latent comparison, together with a top-(k) aggregation and a consistency condition to improve reuse stability. Experimental results show that Encoder BPA improves learning efficiency and advice utilisation at an early stage of training compared with different Deep IntRL approaches, especially in complex and partially observable environments.
BibTeX
@misc{liang2026broad,
  author = {Zhuoran Liang and Francisco Cruz},
  title = {{Broad-Persistent Advising in Interactive Reinforcement Learning via Latent State Encoding}},
  year = {2026},
  eprint = {2611.13712},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13712},
}
Figure