Reinforcement Learning with Perceptual Aliasing: The Perceptual Distinctions Approach

Lonnie ChrismanAAAI 1992

Abstract

This paper introduces the Predictive Distinctions approach, a method for reinforcement learning (RL) in environments where different physical states appear identical to an agent's sensors—a phenomenon known as perceptual aliasing. By learning a probabilistic predictive model from experience, the system autonomously discovers hidden state distinctions. This allows the agent to utilize memory to make optimal decisions even when immediate sensory input is incomplete or ambiguous.

Introduction

In traditional reinforcement learning, agents typically map current sensory inputs (percepts) directly to actions. However, real-world robotic applications often encounter perceptual aliasing, where two distinct environmental configurations produce identical sensor readings but require different optimal actions. For instance, a robot may need to remember its previous location to decide whether to move toward a charger that is currently out of sight. Without internal state or memory, such agents cannot resolve these ambiguities. This research proposes a framework for learning a world model that accounts for hidden features without prior specification of the state space.

Methods: The Predictive Distinctions Architecture

The proposed system utilizes a two-layered architecture to bridge the gap between raw perception and action selection.

Architecture of the Predictive Distinctions approach, showing the interaction between the predictive model and the reinforcement learner.
  1. Predictive Model: Instead of acting on raw percepts, the agent maintains an internal "belief" regarding its current state, modeled as a Partially Observable Markov Decision Process (POMDP). This model tracks transition probabilities (how actions change states) and observation probabilities (likelihood of percepts in specific states).
  2. Reinforcement Learner: The agent performs Q-learning over its "belief vector"—the probability distribution over possible states—rather than raw sensory inputs.

Dynamic State Discovery

The core innovation lies in how the model evolves. Starting with a minimal model, the agent monitors its predictive accuracy. When the current model fails to be "Markovian" (i.e., the current state is insufficient to predict the next state), the agent employs Chi-squared tests on historical data. If significant differences in transition or observation distributions are detected between segments of experience, the agent "splits" a state into two, effectively discovering a hidden distinction in the environment.

Experimental Results

The approach was evaluated in a Simulated Space Station Docking environment. The task required a robot to transport supplies between two identical-looking stations. To receive a reward, the robot had to visit the station it had least recently visited, necessitating memory of past actions.

Auto-estimated performance (Q-values) over 16,000 time steps, showing convergence as the agent discovers hidden states.

The system successfully discovered an 11-state model from an initial 2-state configuration, accurately capturing the hidden complexities of the task. As shown in Figure 2, the Q-values stabilized over 16,000 time steps, indicating the convergence of an optimal policy. The agent successfully filtered out 30% sensor noise to maintain a stable belief state and executed a complex 5-step operational loop.

Discussion and Conclusion

The "Predictive Distinctions" approach demonstrates that predictiveness can serve as a powerful proxy for utility. By focusing on improving its predictive model, the agent naturally identifies the hidden states necessary for high-performance decision-making. This dynamic growth allows agents to start with minimal knowledge and incrementally add complexity as demanded by their experience.

However, limitations remain. Perceptual aliasing complicates exploration, as the agent may misidentify its location. Furthermore, the system may "oversplit" states or fail to discover distinctions that affect rewards but not immediate percepts. Future work may explore the "Utile-Distinction Conjecture," investigating whether agents can learn to prioritize distinctions that directly impact utility over general environmental prediction.

Summary updated

BibTeX
@inproceedings{chrisman1992reinforcement,
  author = {Lonnie Chrisman},
  title = {{Reinforcement Learning with Perceptual Aliasing: The Perceptual Distinctions Approach}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1992},
}
Figure