Abstract
Psychlab is a simulated laboratory environment built on DeepMind Lab, designed to bridge the gap between artificial intelligence and experimental psychology. By recreating classical psychological paradigms, Psychlab enables researchers to evaluate the cognitive faculties of reinforcement learning (RL) agents with the same precision used for human subjects. Initial experiments using the state-of-the-art UNREAL agent revealed significant visual and temporal processing deficits compared to humans. These findings led to the development of a bio-inspired "foveal vision" model that significantly improves agent performance across standard RL benchmarks.
Introduction
While deep reinforcement learning (RL) agents have achieved remarkable success in complex 3D environments, understanding the specific reasons for their success or failure remains a challenge. It is often unclear whether an agent's failure is due to poor memory, limited visual acuity, or an inability to track moving objects. Standard benchmarks are typically too "noisy" to isolate these specific cognitive faculties.
To address this, the authors introduce Psychlab, a platform that brings 150 years of experimental psychology into the realm of AI research. Psychlab recreates controlled laboratory conditions within a 3D game world, allowing for the use of "psychophysics"—the quantitative study of the relationship between physical stimuli and sensation—to benchmark agents with high precision.
The Psychlab Platform
Psychlab is built upon the DeepMind Lab (DM-Lab) environment. In this setup, the agent is positioned before a virtual computer monitor and interacts with stimuli by "saccading" (moving its gaze) toward targets via a simple API.
Key features of the platform include:
- The Psychlab API: A Lua-based framework that allows for the creation of tasks by placing "widgets" on the virtual screen. These widgets can trigger rewards or callbacks based on agent interaction.
- Adaptive Staircase Procedures: Psychlab utilizes standard psychological methods where task difficulty (e.g., stimulus size or contrast) adjusts automatically based on performance. For RL agents, this serves as an automated curriculum.
- Foveal Vision Model: Based on discovered visual weaknesses in standard agents, the authors implemented a non-uniform subsampling method. This "fovea" mimics the human eye by maintaining high pixel density in the center of the gaze while discarding information at the periphery.
Experimental Paradigms
Psychlab introduces a suite of classic psychological tasks as new benchmarks for AI agents. These tasks are designed to isolate specific cognitive functions:
- Visual Search: Identifying a target among a set of distractors.
- Landolt C Identification: Measuring visual acuity and contrast sensitivity.
- Glass Pattern Detection: Testing the perception of global forms.
- Change Detection: Evaluating visual working memory.
- Random Dot Motion & Multiple Object Tracking: Testing motion perception and temporal integration.
The platform's validity was confirmed by comparing human performance on these tasks with existing psychology literature, showing a close match.
Analysis of Agent Performance
Testing the state-of-the-art UNREAL agent in Psychlab yielded several key insights into the current limitations of deep RL architectures:
Acuity and Resolution
Surprisingly, the UNREAL agent demonstrated significantly worse visual acuity than humans, even when both were tested at the same low resolution (84x84). This suggests that standard Convolutional Neural Network (CNN) architectures may suffer from aliasing or lose critical information about small targets.
The "Large Object" Bias
Due to the nature of convolutional weight sharing, agents receive larger gradient signals from larger stimuli. Consequently, UNREAL learns to identify large objects much faster than small ones. This bias can be detrimental, as agents might focus on large, low-reward lures while ignoring smaller, more valuable targets.
Failure of Temporal Integration
While UNREAL performed well on static tasks like visual search, it struggled significantly with tasks requiring integration over time. In motion discrimination and multiple object tracking tasks, the agent performed at chance levels. This highlights a major limitation in how current agents process temporal information, despite the inclusion of Long Short-Term Memory (LSTM) units.
Search Patterns
In visual search tasks, human reaction times typically increase with the number of distractors (serial search). In contrast, UNREAL showed constant reaction times regardless of the number of items, reflecting the parallel processing nature of CNNs.
Results and Validation
The insights gained from Psychlab led to tangible improvements in agent performance. By introducing the foveal vision model, the authors were able to scale the UNREAL agent to higher resolutions without modifying its internal architecture. This foveated UNREAL agent outperformed the standard uniform-resolution agent on all eight DeepMind Lab "laser tag" levels, demonstrating that improvements in controlled psychological tasks translate to broader RL performance.
Discussion and Future Directions
The current fovea model is a simplified approximation of the biological eye. Furthermore, the agent's failure on motion-based tasks indicates that standard LSTMs are insufficient for replicating human-like temporal processing or working memory.
Future work will utilize Psychlab to probe other cognitive domains, such as reinforcement learning schedules and complex decision-making. By tightening the loop between cognitive science and AI design, Psychlab provides a rigorous framework for developing more capable and human-like artificial agents.
Summary updated