State Representation Through Observable Future Predictions

211 concepts135 papers1 proposal

When an artificial agent operates in a complex environment, it rarely has access to the complete state of the world. In a partially observable environment—whether it is a robot navigating a crowded room or an algorithm managing a patient's healthcare treatment—the agent must construct an internal representation of the world from a stream of incomplete observations.

Traditionally, AI has solved this in one of two ways:

  1. History-based representations: Remember everything that has happened up to the current moment.
  2. Latent-variable representations: Infer a hidden, unobservable state (such as a belief state in a POMDP) that generated the observations.

Both approaches have severe drawbacks. History-based methods suffer from an exponential explosion of unique paths; as noted by Eddie J. Rafols et al. in Using Predictive Representations to Improve Generalization in Reinforcement Learning, tracking raw history creates a massive number of distinct trajectories that actually lead to the exact same functional situation. Conversely, latent-variable models are notoriously difficult to learn from data. Algorithms like Expectation-Maximization frequently get trapped in local minima because they try to estimate parameters for variables that can never be directly observed.

Predictive State Representations (PSRs) offer a radical alternative: define the state of the world not by what happened in the past, and not by what the world is right now, but entirely by what will happen in the future if the agent takes specific actions.

The Core Intuition: State as a Vector of Tests

To understand why this works, imagine navigating a dark maze. Instead of representing your location as an abstract coordinate like $(X=4, Y=2)$, you represent your state as a collection of answers to hypothetical questions:

  • If I take two steps forward, will I hit a wall? (Probability: 0.9)
  • If I turn left and open a door, will I see a light? (Probability: 0.1)

In the language of PSRs, each of these action-observation sequences is called a test. A test is a sequence of future actions and their corresponding observations. The state of the system at any given moment is simply a vector containing the conditional probabilities that these tests will succeed.

As formalized by Masoumeh T. Izadi et al. in A Planning Algorithm for Predictive State Representations, you do not need to track an infinite number of future tests to understand the environment. For any finite-dimensional dynamical system, there exists a small, finite set of linearly independent tests—known as core tests—that form a sufficient statistic for the entire system. If you can accurately predict the outcomes of these core tests, you implicitly know everything there is to know about the environment's hidden dynamics.

This shifts the entire paradigm of state modeling. By substituting immediate perception or abstract hidden variables with learned expectations of future outcomes, as explored by Richard S. Sutton et al. in Temporal Abstraction in Temporal-difference Networks, the agent grounds its internal model entirely in measurable, verifiable data.

Bypassing Local Minima via Matrix Factorization

The true elegance of grounding state in future predictions is that it transforms a difficult unsupervised learning problem into a straightforward linear algebra problem.

Consider a massive, theoretical matrix called the History-Test Matrix. The rows of this matrix represent every possible history of actions and observations the agent could have experienced. The columns represent every possible future test. Each entry in the matrix is the conditional probability that a specific test will succeed given a specific history.

                  Future Tests
               T1      T2      T3   ...
            +------+------+------+
H1 (History)| 0.9  | 0.1  | 0.4  |
            +------+------+------+
H2 (History)| 0.2  | 0.8  | 0.5  |
            +------+------+------+
H3 (History)| 0.9  | 0.1  | 0.4  |
    ...     +------+------+------+

If the underlying environment can be generated by a system with $d$ hidden states, then the mathematical rank of this enormous matrix is at most $d$. This means the relationship between what has occurred (history) and what is likely to happen next (tests) is fundamentally low-rank.

Instead of using iterative, local-search algorithms like Expectation-Maximization to guess hidden parameters, we can collect data from the environment, populate a matrix of observable frequencies, and perform Singular Value Decomposition (SVD). As demonstrated by Yash Chandak et al. in Representations and Exploration for Deep Reinforcement Learning using Singular Value Decomposition, decomposing this history-test relationship allows an agent to extract a compact belief state using simple linear transformations. Because SVD provides a globally optimal solution, this spectral learning approach completely bypasses the local minima traps that plague traditional latent-variable models.

Furthermore, because these predictions are explicitly conditioned on future actions, they are policy-invariant. As Ahmed Hefny et al. show in An Efficient, Expressive and Local Minima-Free Method for Learning Controlled Dynamical Systems, formulating the system state as a conditional expectation ensures that the learned representation remains mathematically valid even when the agent's behavior policy changes drastically between training and deployment.

Modern Deep Learning Integration

While classical PSRs relied on exact linear operators, the core philosophy of observable-based state grounding has become a vital tool for regularizing modern deep neural networks.

Recurrent Neural Networks (RNNs) often struggle to discover meaningful internal representations in reinforcement learning because scalar reward signals are too sparse and indirect. To fix this, researchers use future-prediction as an auxiliary self-supervised task.

In Predictive-State Decoders: Encoding the Future into Recurrent Networks, Arun Venkatraman et al. introduced a regularizer that forces the hidden state of an RNN to directly decode into multi-step future observations. By forcing the network to forecast the consequences of its actions, the internal representation is compelled to capture the underlying physical properties and system dynamics without ever requiring direct access to ground-truth latent parameters.

+-----------------+     +-----------------+     +-----------------+
|  Past History   | --> |  Hidden State   | --> | Predicted Future|
| (Observations)  |     |     (RNN)       |     |  (Observations) |
+-----------------+     +-----------------+     +-----------------+
                                 |
                                 v
                        +-----------------+
                        | Policy / Action |
                        +-----------------+

Similarly, Amy Zhang 0001 et al. leverage this concept in Learning Causal State Representations of Partially Observable Environments by training models to compress history into "causal states" that are sufficient statistics for predicting immediate subsequent observations and rewards.

By shifting the target of representation learning from reconstructing the past to anticipating the future, these architectures ensure that the agent's internal mental model remains tightly bound to the observable reality of its environment.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 143 papers
  2. Version 1 · 2026-10-04 · 115 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_predictive_state_representation_873bbe92,
  title = {predictive state representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/predictive-state-representation/rdr_873bbe92243d4349ae7557e0976c22a0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_predictive_state_representation_873bbe92_v2,
  title = {predictive state representation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/predictive-state-representation/rdr_873bbe92243d4349ae7557e0976c22a0/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake