Manifold Embeddings for Model-Based Reinforcement Learning under Partial Observability

Keith Bush, Joelle PineauNeurIPS 2009 · OpenReview

The Challenge of Partial Observability

Most Reinforcement Learning (RL) algorithms rely heavily on the Markov property: the assumption that the current state provides all the necessary information to predict the future. In the real world, however, this assumption rarely holds. We frequently encounter Partial Observability, where our sensors only capture a "shadow" of the system's true internal dynamics—such as observing a car's position but not its velocity, or reading a single electrode's voltage without knowing the full state of the brain.

When the Markov property is broken, standard RL fails. While researchers often turn to complex recurrent neural networks or Partially Observable Markov Decision Process (POMDP) frameworks to handle this, these methods can be computationally heavy or require "first-principles" physics models that simply do not exist for complex systems like biological tissue. This paper asks a fundamental question: Can we reconstruct a valid, Markovian state space directly from a stream of noisy, incomplete observations?

Methodology: Reconstructing the State Space

The authors bridge the gap between nonlinear dynamic analysis and RL to create a framework for off-line, model-based control. The approach relies on three main components:

  1. Takens’ Theorem & Delay Embeddings: This mathematical theory suggests that you can reconstruct the "topology" of a hidden state space by looking at a sliding window of past observations. Given a sequence of observations $\tilde{s}$, a new state vector $s_E(t)$ can be created by stacking current and past observations: $[\tilde{s}(t), \tilde{s}(t-1), \dots, \tilde{s}(t-(E-1))]$.
  2. Spectral Parameter Selection: Choosing the right "window size" and "dimension" ($E$) for these embeddings is notoriously difficult. The authors utilize a non-parametric spectral approach based on Singular Value Decomposition (SVD) to automatically extract the optimal parameters directly from the data.
  3. Generative Local Models: Once the state space is reconstructed, the authors build a local model rather than immediately learning a policy. Using a nearest-neighbor approach to estimate the system's derivatives, they simulate "surrogate trajectories." They then run Batch Q-learning on this model to find the optimal policy offline.

Key Insights: The Dynamic Nature of Manifolds

One of the most striking findings in this research is that the "shape" of the system's state space changes as the agent learns.

Dynamic Manifold: Random vs. Learned Policy

As shown above, a random policy might require a complex, higher-dimensional embedding to adequately describe the system's erratic dynamics. However, a highly optimized "learned" policy often restricts the system to a narrower set of behaviors, effectively collapsing the dynamics into a simpler, lower-dimensional manifold.

Furthermore, by ensuring the embedding preserves "locality" (meaning points that are close in the embedding correspond to similar physical states), the authors can use simple, robust nearest-neighbor models instead of complex global function approximators. Building this model first allows researchers to validate the embedding and predicted dynamics before deploying a policy on a sensitive system.

Real-World Application: Neurostimulation for Epilepsy

The paper evaluates the approach on a partially observable "Mountain Car" benchmark—successfully reconstructing hidden velocity to achieve near-optimal path-to-goal steps—but its primary contribution is a high-stakes real-world application: Neurostimulation for Epilepsy.

The dataset consists of field potential recordings from rat hippocampus brain slices. The goal is to learn a stimulation policy that suppresses seizures while minimizing the amount of electrical interference.

Neurostimulation Field Potentials and Embeddings

In a simulated environment built from this data, the learned policy reduced the time spent in a "seizure state" from 25.6% (uncontrolled) to just 5.2%. It also outperformed a standard fixed 1.0 Hz stimulation policy while using approximately 40% less electrical stimulation.

Crucially, when the policy learned in simulation was transferred to real animal brain slices, it successfully suppressed seizures across multiple trials, maintaining an effective stimulation frequency of roughly 0.65 Hz. This demonstrates that policies learned on manifold embeddings can successfully transfer to real, noisy biological systems.

Limitations and Future Work

While promising, the framework has a few limitations that pave the way for future research:

  • Data Density: The nearest-neighbor modeling approach requires a relatively dense dataset to avoid bias. In extremely high-dimensional systems, the "curse of dimensionality" might make this difficult.
  • Static Embeddings: Currently, the embedding parameters are chosen once before learning. Future work could involve adaptive embeddings that update as the policy changes, potentially using a policy gradient approach to optimize the state representation itself.
  • Computational Scaling: While SVD and nearest-neighbor lookups are efficient for the datasets tested, scaling to massive, real-time industrial streams would require more optimized data structures, such as KD-trees or approximate nearest neighbors.

Summary updated

Abstract

Interesting real-world datasets often exhibit nonlinear, noisy, continuous-valued states that are unexplorable, are poorly described by first principles, and are only partially observable. If partial observability can be overcome, these constraints suggest the use of model-based reinforcement learning. We experiment with manifold embeddings as the reconstructed observable state-space of an off-line, model-based reinforcement learning approach to control. We demonstrate the embedding of a system changes as a result of learning and that the best performing embeddings well-represent the dynamics of both the uncontrolled and adaptively controlled system. We apply this approach in simulation to learn a neurostimulation policy that is more efficient in treating epilepsy than conventional policies. We then demonstrate the learned policy completely suppressing seizures in real-world neurostimulation experiments on actual animal brain slices.
BibTeX
@inproceedings{bush2009manifold,
  author = {Keith Bush and Joelle Pineau},
  title = {{Manifold Embeddings for Model-Based Reinforcement Learning under Partial Observability}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2009},
  url = {https://openreview.net/forum?id=BJVExKZdZH},
}
Figure