Grounding State Representations in Observable Future Predictions

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2115 papers166 conceptsfrozen 2026-10-04

In reinforcement learning, the concept of "state" is often treated as a hidden variable—a secret code that the environment uses to decide what happens next. In Partially Observable Markov Decision Processes (POMDPs), agents must maintain a "belief" over these hidden variables, which are often abstract and difficult to learn.

Predictive State Representations (PSRs), introduced by Michael Littman and Satinder Singh 0001 in Predictive Representations of State, flip this perspective. Instead of imagining hidden variables that generate data, they define state as a collection of predictions about future observations. By grounding the internal representation in observable data, PSRs avoid the local optima and modeling challenges inherent in latent-variable systems.

The Intuition: State as a Forecast

Imagine you are navigating a dark room. A traditional latent-state model might try to estimate your $(x, y)$ coordinates—a variable you cannot directly see. A predictive state model, however, represents your "location" as a set of answers to questions: "If I take two steps forward, will I hit a wall?" or "If I reach to the left, will I feel a doorknob?"

In the PSR framework, these questions are called tests. A test is a sequence of future actions and observations. The state of the system at any moment is simply a vector of the probabilities that these tests will succeed. As David Wingate explains in Exponential Family Predictive Representations of State, this approach ensures that every component of the state has a direct physical meaning tied to the environment's dynamics.

Why Prediction Outperforms Latent Modeling

The primary advantage of predictive representations is verifiability. In latent models, the target values for hidden states are never observed. In PSRs, the target values for the state components—the actual outcomes of future actions—eventually appear in the agent's data stream.

This grounding provides several benefits:

  1. Avoiding Local Optima: Because the targets are observable, learning can be framed as a supervised task. Wen Sun demonstrates in Learning to Filter with Predictive State Inference Machines that using future observations as ground truth labels allows for more robust training than unsupervised latent-variable inference.
  2. Capturing Long-Term Dependencies: Traditional models often struggle with "aliasing," where different parts of the environment look identical. By maintaining a "profile" of future probabilities, as explored by Erik Talvitie in Maintaining Predictions over Time without a Model, agents can distinguish between similar-looking states based on their different future trajectories.
  3. Efficiency: Certain deterministic systems can be represented with exponentially fewer parameters using PSRs than with traditional POMDPs, as shown in A Nonlinear Predictive State Representation.

Integrating Predictions into Neural Networks

While early PSRs focused on linear systems, modern research integrates these ideas into deep learning. General Value Function Networks (GVFNs), proposed by Matthew Kyle Schlegel and Adam White 0001 in General Value Function Networks, are recurrent neural networks where the hidden units are explicitly constrained to be multi-step predictions about the future.

Similarly, Arun Venkatraman introduced Predictive-State Decoders (PSDs) in Predictive-State Decoders: Encoding the Future into Recurrent Networks. This method uses a regularizer that forces an RNN's internal state to be capable of decoding future observations. This "future-dependent" training signal helps the network learn more meaningful representations of the environment's underlying physics without requiring direct access to latent parameters.

Theoretical Foundations and Scalability

The field has recently moved toward establishing rigorous guarantees for these models. In Partially Observable RL with B-Stability, Fan Chen 0006 and Song Mei identify B-stability as a unified condition that makes PSRs tractable. This allows for algorithms that achieve polynomial sample complexity, meaning they can learn near-optimal policies in complex environments with a reasonable amount of data.

Furthermore, the work of Masatoshi Uehara in Future-Dependent Value-Based Off-Policy Evaluation in POMDPs shows that even short windows of future observations can serve as powerful proxies for hidden states. This suggests that the "predictive" approach is not just a theoretical curiosity but a practical necessity for scaling reinforcement learning to high-dimensional, partially observable worlds.

Papers in this version

Showing 60 of 115 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 143 papers
  2. Version 1 · 2026-10-04 · 115 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_predictive_state_representation_873bbe92,
  title = {predictive state representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/predictive-state-representation/rdr_873bbe92243d4349ae7557e0976c22a0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_grounding_state_representations_in_obser_873bbe92_v1,
  title = {Grounding State Representations in Observable Future Predictions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/grounding-state-representations-in-observable-future-predictions/rdr_873bbe92243d4349ae7557e0976c22a0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}