Minimalist State Representations in Reinforcement Learning

29 concepts25 papers1 proposal

When a reinforcement learning (RL) agent attempts to master a task, it must observe the world and decide on an action. Intuitively, it seems that giving the agent more information—more sensor readings, finer coordinate grids, or richer history—would always lead to better decisions.

In practice, the opposite is often true. High-dimensional state spaces suffer from the "curse of dimensionality." As the number of features grows, the volume of the state space increases exponentially. The agent becomes lost in a massive, sparse desert of data, struggling to distinguish meaningful signals from irrelevant noise. As Giorgi et al. demonstrate theoretically, value function approximation is fundamentally bottlenecked by the underlying topology of the state space, making direct learning on raw, uncompressed environments computationally infeasible.

To build sample-efficient and robust agents, researchers use State Representation Learning (SRL) and dimensionality reduction. By compressing the environment's state space into a minimalist representation, we simplify the optimization landscape, allowing agents to learn faster and generalize better.

The Intuition of State Compression

Why does reducing the state space help an agent learn? Imagine training a robot to kick a soccer ball. If the robot tracks its position using absolute global coordinates, its policy must learn how to kick the ball from every single spot on the field. The state space is massive, and the data is highly redundant.

However, if we transform the coordinate system to be robot-centric, the state space collapses. Park et al. demonstrated that this simple coordinate transformation reduces state dimensionality by 60%, preventing the policy from overfitting to global coordinates and drastically accelerating training stability.

[Absolute Coordinates]  --> High Dimensional, Sparse Data, Slow Convergence
[Relative Coordinates]  --> Low Dimensional, Dense Data, Fast Convergence

By forcing the agent to view the world through a compact lens, we restrict the search space for the policy optimizer. This concept manifests across three primary paradigms: feature selection, structural transformation, and learned latent embeddings.

1. Information-Theoretic Feature Selection

One of the most direct ways to compress a state space is to filter out variables that do not influence the optimal policy.

  • Mutual Information Filtering: Frameworks like the one proposed by Mussi et al. use automated feature selection based on mutual information to isolate the most informative variables. By filtering out features that share low mutual information with the reward or transition dynamics, the agent avoids the computational overhead of processing noise.
  • Heuristic Ranking: In educational data mining, Jordan et al. utilized feature selection methods to rank potential state variables for tutorial dialogue tactics. They discovered that an agent's performance metrics were less critical than its autonomy levels, allowing them to discard low-impact features.
  • Enabling Tabular Methods: For highly complex environments like online advertising networks, Refaei Afshar et al. showed that concentrating on a tiny subset of high-impact features (such as price floors and sequence positions) can compress a state space by several orders of magnitude. This extreme reduction makes simple, stable tabular RL methods viable in environments that typically require deep neural networks.

2. Task-Specific and Approximate Abstractions

Instead of just dropping features, we can fundamentally change how the state space is structured based on the task at hand.

Konidaris et al. introduced a framework where an autonomous robot dynamically selects task-specific state spaces for different sub-tasks (or options). When navigating a room, the robot ignores its fine-grained motor configurations; when manipulating an object, it ignores the room's global geometry. This decoupling ensures that the agent only faces high-dimensional optimization when absolutely necessary.

Similarly, in environments with partial observability (POMDPs), maintaining an exact belief state over all variables is paralyzing. Rodriguez et al. showed that by using factored, approximate belief states—effectively marginalizing out weak correlations between variables—agents can search the policy space significantly faster and avoid getting trapped in poor local optima.

In engineering applications, this abstraction can be driven by expert knowledge. For instance, Meng et al. simplified a high-dimensional digital twin synchronization problem by compressing raw trajectory data into a single scalar performance metric (Mean Squared Error), while Ao et al. designed a fixed-dimensional state representation using geometric coverage metrics to make robotic sensor calibration sample-efficient.

3. Learning Latent Embeddings

When dealing with high-dimensional sensory inputs like raw camera pixels, manual feature engineering becomes impossible. This is where State Representation Learning (SRL) excels. As surveyed by Lesort et al. and benchmarked by Raffin et al., SRL algorithms train auxiliary models (like autoencoders or forward dynamics models) to distill raw inputs into low-dimensional, continuous latent vectors.

[Raw Pixels (e.g., 256x256x3)] -> [SRL Encoder] -> [Latent Vector (e.g., dim=16)] -> [RL Policy]

This separation of concerns is powerful: the encoder learns what the world looks like, allowing the RL policy to focus entirely on how to act. Walsh et al. demonstrated that learning these compact representations (such as Factored MDPs) allows agents to maintain formal exploration guarantees while drastically reducing sample complexity.

The Sim-to-Real Advantage: Why Less is More

The benefits of minimalist representations extend beyond training speed; they are critical for transferring policies from simulation to the real world (Sim-to-Real).

When an agent is trained in simulation on a high-dimensional observation space, it often learns to exploit subtle, simulation-specific artifacts or noise. When deployed on a physical robot, these unmodeled dynamics cause the policy to fail.

Tan et al. demonstrated that reducing the dimensionality of the observation space forces quadrupedal locomotion policies to rely on generalized, fundamental features rather than simulation noise. This concept was comprehensively benchmarked by Dionigi et al. in quadrotor flight. The authors discovered that minimalist input configurations consistently outperformed high-dimensional state inputs, enabling robust, zero-shot transfer to the real world using only relative position data.

A Nuanced Counterpoint

While minimizing dimensionality is the dominant paradigm for sample efficiency, it is worth noting that the relationship between input size and optimization is not entirely linear. Ota et al. showed that intentionally increasing input dimensionality through a specialized feature extractor trained on auxiliary tasks can sometimes help neural network optimizers find better policies in continuous control.

However, this approach relies on structured expansion to aid gradient flow, rather than exposing the agent to uncompressed environmental noise. The overarching consensus remains clear: whether through information filtering, geometric transformations, or latent embeddings, distilling the environment into its most critical, minimalist essence is the key to building capable, efficient, and robust reinforcement learning agents.

Key Foundational Work

Reinforcement Learning Using Approximate Belief States (1999) — contributed 1 concept elements to this direction.
Reinforcement Learning-based Feature Seleciton For Developing Pedagogically Effective Tutorial Dialogue Tactics (2008) — contributed 1 concept elements to this direction.
Autonomous Robot Skill Acquisition (2008) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_state_space_dimensionality_reduction_c5442232,
  title = {state space dimensionality reduction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/state-space-dimensionality-reduction/rdr_c5442232e7004cfab10a457733e709b0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_state_space_dimensionality_reduction_c5442232_v1,
  title = {state space dimensionality reduction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/state-space-dimensionality-reduction/rdr_c5442232e7004cfab10a457733e709b0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake