The Challenge of Partial Observability
Most Reinforcement Learning (RL) algorithms rely heavily on the Markov property: the assumption that the current state provides all the necessary information to predict the future. In the real world, however, this assumption rarely holds. We frequently encounter Partial Observability, where our sensors only capture a "shadow" of the system's true internal dynamics—such as observing a car's position but not its velocity, or reading a single electrode's voltage without knowing the full state of the brain.
When the Markov property is broken, standard RL fails. While researchers often turn to complex recurrent neural networks or Partially Observable Markov Decision Process (POMDP) frameworks to handle this, these methods can be computationally heavy or require "first-principles" physics models that simply do not exist for complex systems like biological tissue. This paper asks a fundamental question: Can we reconstruct a valid, Markovian state space directly from a stream of noisy, incomplete observations?
Methodology: Reconstructing the State Space
The authors bridge the gap between nonlinear dynamic analysis and RL to create a framework for off-line, model-based control. The approach relies on three main components:
- Takens’ Theorem & Delay Embeddings: This mathematical theory suggests that you can reconstruct the "topology" of a hidden state space by looking at a sliding window of past observations. Given a sequence of observations $\tilde{s}$, a new state vector $s_E(t)$ can be created by stacking current and past observations: $[\tilde{s}(t), \tilde{s}(t-1), \dots, \tilde{s}(t-(E-1))]$.
- Spectral Parameter Selection: Choosing the right "window size" and "dimension" ($E$) for these embeddings is notoriously difficult. The authors utilize a non-parametric spectral approach based on Singular Value Decomposition (SVD) to automatically extract the optimal parameters directly from the data.
- Generative Local Models: Once the state space is reconstructed, the authors build a local model rather than immediately learning a policy. Using a nearest-neighbor approach to estimate the system's derivatives, they simulate "surrogate trajectories." They then run Batch Q-learning on this model to find the optimal policy offline.
Key Insights: The Dynamic Nature of Manifolds
One of the most striking findings in this research is that the "shape" of the system's state space changes as the agent learns.
As shown above, a random policy might require a complex, higher-dimensional embedding to adequately describe the system's erratic dynamics. However, a highly optimized "learned" policy often restricts the system to a narrower set of behaviors, effectively collapsing the dynamics into a simpler, lower-dimensional manifold.
Furthermore, by ensuring the embedding preserves "locality" (meaning points that are close in the embedding correspond to similar physical states), the authors can use simple, robust nearest-neighbor models instead of complex global function approximators. Building this model first allows researchers to validate the embedding and predicted dynamics before deploying a policy on a sensitive system.
Real-World Application: Neurostimulation for Epilepsy
The paper evaluates the approach on a partially observable "Mountain Car" benchmark—successfully reconstructing hidden velocity to achieve near-optimal path-to-goal steps—but its primary contribution is a high-stakes real-world application: Neurostimulation for Epilepsy.
The dataset consists of field potential recordings from rat hippocampus brain slices. The goal is to learn a stimulation policy that suppresses seizures while minimizing the amount of electrical interference.
In a simulated environment built from this data, the learned policy reduced the time spent in a "seizure state" from 25.6% (uncontrolled) to just 5.2%. It also outperformed a standard fixed 1.0 Hz stimulation policy while using approximately 40% less electrical stimulation.
Crucially, when the policy learned in simulation was transferred to real animal brain slices, it successfully suppressed seizures across multiple trials, maintaining an effective stimulation frequency of roughly 0.65 Hz. This demonstrates that policies learned on manifold embeddings can successfully transfer to real, noisy biological systems.
Limitations and Future Work
While promising, the framework has a few limitations that pave the way for future research:
- Data Density: The nearest-neighbor modeling approach requires a relatively dense dataset to avoid bias. In extremely high-dimensional systems, the "curse of dimensionality" might make this difficult.
- Static Embeddings: Currently, the embedding parameters are chosen once before learning. Future work could involve adaptive embeddings that update as the policy changes, potentially using a policy gradient approach to optimize the state representation itself.
- Computational Scaling: While SVD and nearest-neighbor lookups are efficient for the datasets tested, scaling to massive, real-time industrial streams would require more optimized data structures, such as KD-trees or approximate nearest neighbors.
Summary updated