Unsupervised State Discovery for Autonomous Agents

34 concepts30 papers1 proposal

When an artificial agent navigates the real world, it is bombarded by an overwhelming stream of high-dimensional sensory data—pixels, audio waveforms, and raw readings. To make decisions, the agent must compress this noise into a clean, actionable internal representation called a state space.

Traditionally, AI designers have bypassed this challenge by hand-coding state spaces, providing agents with pre-defined coordinate grids, symbolic features, or clean environment simulators. However, true autonomy requires tabula rasa learning. An agent placed in an unfamiliar environment must independently discover its own semantic building blocks and state abstractions directly from raw experience.

As Lihong Li et al. established, defining the mathematical properties of an optimal state abstraction does not provide a mechanism for discovering it when the underlying transition and reward functions are unknown. To bridge this gap, a rich body of research focuses on how agents can autonomously construct internal representations without human intervention.


Discovering Primitives from Statistical Regularities

The first step in autonomous state discovery is breaking down continuous experience into discrete, meaningful units. Instead of relying on a static, universal dictionary of concepts, agents can extract semantic primitives by identifying recurring substructures in their sensory-motor streams.

This concept dates back to early computational linguistics and cognitive modeling. Sharon C. Salveter demonstrated that an agent could learn the meanings of actions from scratch by tracking recurring patterns in its environmental interactions, discovering its own semantic building blocks rather than relying on pre-defined primitives. This idea scales to sequential data; Catherine J. Stevens et al. showed that recurrent networks can capture complex structural hierarchies purely by learning to predict the next element in a sequence, extracting the underlying "grammar" of the domain from raw statistical regularities.

Rather than treating the environment as a set of pre-packaged categories, true autonomy treats categorization as an emergent property of interaction. As Stephen W. Smoliar argued through the lens of Neural Darwinism, meaningful categories are not absorbed from external labels; they emerge through a selectionist process where internal neural groups compete and coordinate to map raw sensory-motor experiences into functional classifications.


Compression via Attention and Attractor Dynamics

Raw sensory data contains vast amounts of irrelevant information. To build a functional state space, an agent must aggressively filter out noise and compress its experiences into stable, well-formed representations.

One way to achieve this is through reward-driven attention. Hiroshi Ishiguro et al. demonstrated that robots can construct localized state spaces from raw visual inputs by using attention control to filter out features irrelevant to the immediate task. Similarly, Edwin D. de Jong showed that agents can organize continuous sensory experiences into discrete "situation concepts" using only sparse evaluative feedback, allowing them to form functional representations without explicit ground-truth labels.

Raw Sensory Data ───► [ Attention / Compression Filter ] ───► Stable Attractor State
       │                          ▲
       └─────── Evaluative ───────┘
                Feedback

From a dynamical systems perspective, these compressed states function as "clean-up" mechanisms. Donald W. Mathis et al. proposed that learning temporally persistent, well-formed representations—akin to computational attractor states—allows a network to filter out noise, resolve ambiguities, and keep multi-step reasoning on track. Modern deep learning implementations realize this by compressing internal network signals directly; for instance, Ameya Pore utilized Variational Autoencoders (VAEs) to compress a robot's own internal neural activations into abstract states, enabling autonomous self-monitoring and faster policy learning.


Predictive and Variational World Models

A powerful intuition in representation learning is that a state should not just represent the present—it should predict the future. By defining internal states based on transition dynamics, agents can simulate a "black box" environment simply by observing it.

Early architectures designed by C. L. Winter focused on abstracting significant associations and defining internal states by learning sequences of transitions from noisy data. A mathematically rigorous extension of this is the Predictive State Representation (PSR) framework. Matthew R. Rudary et al. advanced this by introducing nonlinear PSRs, which bypass unobservable hidden states entirely. Instead, they represent the system's state as a set of predictions about the outcomes of future actions, proving that deterministic systems can be modeled with exponentially fewer parameters than traditional hidden-state models.

Traditional POMDP:  [ Hidden State ] ───► Predicts ───► Future Observations
Predictive State:   [ Observable Future Tests ] ───────► Serves as State Representation

In deep learning, this predictive drive is often formulated as variational inference. Tim Verbelen et al. utilized deep active inference to let agents learn internal world models and plan complex actions directly from high-dimensional inputs. By minimizing a single objective—Variational Free Energy—the agent simultaneously optimizes its perception (its internal state representation) and its actions, ensuring its internal model accurately mirrors environmental physics.


Exploration, Topology, and Goal Generation

An agent cannot learn an abstraction for a state or concept it has never encountered. Therefore, autonomous state discovery is intimately tied to intrinsic motivation, exploration, and mapping.

To discover cause-and-effect relationships without a teacher, early autonomous systems like the one developed by Paul D. Scott et al. used Shannon uncertainty to drive curiosity, generating conjectures about the environment and retracting them when contradicted by experience. In modern reinforcement learning, this exploration is often driven by competitive dynamics. Sainbayar Sukhbaatar et al. introduced asymmetric self-play, where two minds of the same agent (Alice and Bob) compete to challenge one another. This adversarial interaction forces the agent to thoroughly explore its state space and discover useful skills before any external task is introduced.

As agents explore, they must organize their discoveries geometrically or topologically. Kuniaki Kawabata et al. combined Self-Organizing Maps (SOM) with reinforcement learning to let mobile robots discretize continuous visual environments into topological navigation states. Expanding on this, Laëtitia Matignon et al. developed frameworks to discover topological representations of environments, allowing agents to identify promising areas for skill acquisition in sparse-reward settings.

Raw Exploration ───► Topological Mapping ───► Sub-goal Identification ───► Policy Basis

Once the environment's structure is mapped, agents can identify critical transition points, exit states, or sub-goals. David Kuric et al. demonstrated that automatically discovering these transition states allows an agent to construct a reusable "policy basis," which can be instantly combined to solve complex, multi-stage planning problems with global optimality guarantees.

Ultimately, this enables the creation of autotelic agents. As surveyed by Cédric Colas et al., autotelic agents leverage intrinsic motivation to autonomously generate, represent, and pursue goals they have never previously encountered. Rather than being confined to human-defined coordinate grids to track progress—a limitation highlighted by Zhao Yang et al.—these agents learn to identify and represent target states entirely on their own.

This transition from hand-coded symbolic mappings to unsupervised state discovery is essential for true autonomy. As Som Sagar et al. note, discovering these concepts directly from an agent's experience not only empowers autonomous planning but also provides a foundation for generating trustworthy, human-understandable explanations of robot decision-making without relying on expert-defined concept dictionaries.

Key Foundational Work

On the Existence of Primitive Meaning Units (1980) — contributed 2 concept elements to this direction.
An Adaptive Network That Learns Sequences of Transitions (1988) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_autonomous_representation_learning_3ed01b1d,
  title = {autonomous representation learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/autonomous-representation-learning/rdr_3ed01b1d46ae4cd7a7933843147eb4cb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_autonomous_representation_learning_3ed01b1d_v1,
  title = {autonomous representation learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/autonomous-representation-learning/rdr_3ed01b1d46ae4cd7a7933843147eb4cb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake