Psychlab: A Psychology Laboratory for Deep Reinforcement Learning Agents

Joel Z. Leibo, Cyprien de Masson d'Autume, Daniel Zoran, David Amos, Charles Beattie, Keith Anderson, Antonio García Castañeda, Manuel Sanchez, Simon Green, Audrunas Gruslys, Shane Legg, Demis Hassabis, Matthew BotvinickarXiv 2018 · OpenReview

Abstract

Psychlab is a simulated laboratory environment built on DeepMind Lab, designed to bridge the gap between artificial intelligence and experimental psychology. By recreating classical psychological paradigms, Psychlab enables researchers to evaluate the cognitive faculties of reinforcement learning (RL) agents with the same precision used for human subjects. Initial experiments using the state-of-the-art UNREAL agent revealed significant visual and temporal processing deficits compared to humans. These findings led to the development of a bio-inspired "foveal vision" model that significantly improves agent performance across standard RL benchmarks.

Introduction

While deep reinforcement learning (RL) agents have achieved remarkable success in complex 3D environments, understanding the specific reasons for their success or failure remains a challenge. It is often unclear whether an agent's failure is due to poor memory, limited visual acuity, or an inability to track moving objects. Standard benchmarks are typically too "noisy" to isolate these specific cognitive faculties.

To address this, the authors introduce Psychlab, a platform that brings 150 years of experimental psychology into the realm of AI research. Psychlab recreates controlled laboratory conditions within a 3D game world, allowing for the use of "psychophysics"—the quantitative study of the relationship between physical stimuli and sensation—to benchmark agents with high precision.

The Psychlab Platform

Psychlab is built upon the DeepMind Lab (DM-Lab) environment. In this setup, the agent is positioned before a virtual computer monitor and interacts with stimuli by "saccading" (moving its gaze) toward targets via a simple API.

Key features of the platform include:

  • The Psychlab API: A Lua-based framework that allows for the creation of tasks by placing "widgets" on the virtual screen. These widgets can trigger rewards or callbacks based on agent interaction.
  • Adaptive Staircase Procedures: Psychlab utilizes standard psychological methods where task difficulty (e.g., stimulus size or contrast) adjusts automatically based on performance. For RL agents, this serves as an automated curriculum.
  • Foveal Vision Model: Based on discovered visual weaknesses in standard agents, the authors implemented a non-uniform subsampling method. This "fovea" mimics the human eye by maintaining high pixel density in the center of the gaze while discarding information at the periphery.

Experimental Paradigms

Psychlab introduces a suite of classic psychological tasks as new benchmarks for AI agents. These tasks are designed to isolate specific cognitive functions:

  • Visual Search: Identifying a target among a set of distractors.
  • Landolt C Identification: Measuring visual acuity and contrast sensitivity.
  • Glass Pattern Detection: Testing the perception of global forms.
  • Change Detection: Evaluating visual working memory.
  • Random Dot Motion & Multiple Object Tracking: Testing motion perception and temporal integration.
Overview of the various psychological tasks implemented in Psychlab, including visual search, motion perception, and object tracking.

The platform's validity was confirmed by comparing human performance on these tasks with existing psychology literature, showing a close match.

Analysis of Agent Performance

Testing the state-of-the-art UNREAL agent in Psychlab yielded several key insights into the current limitations of deep RL architectures:

Acuity and Resolution

Surprisingly, the UNREAL agent demonstrated significantly worse visual acuity than humans, even when both were tested at the same low resolution (84x84). This suggests that standard Convolutional Neural Network (CNN) architectures may suffer from aliasing or lose critical information about small targets.

The "Large Object" Bias

Due to the nature of convolutional weight sharing, agents receive larger gradient signals from larger stimuli. Consequently, UNREAL learns to identify large objects much faster than small ones. This bias can be detrimental, as agents might focus on large, low-reward lures while ignoring smaller, more valuable targets.

Failure of Temporal Integration

While UNREAL performed well on static tasks like visual search, it struggled significantly with tasks requiring integration over time. In motion discrimination and multiple object tracking tasks, the agent performed at chance levels. This highlights a major limitation in how current agents process temporal information, despite the inclusion of Long Short-Term Memory (LSTM) units.

The Multiple Object Tracking task, one of the temporal integration paradigms where standard RL agents performed poorly.

Search Patterns

In visual search tasks, human reaction times typically increase with the number of distractors (serial search). In contrast, UNREAL showed constant reaction times regardless of the number of items, reflecting the parallel processing nature of CNNs.

Results and Validation

The insights gained from Psychlab led to tangible improvements in agent performance. By introducing the foveal vision model, the authors were able to scale the UNREAL agent to higher resolutions without modifying its internal architecture. This foveated UNREAL agent outperformed the standard uniform-resolution agent on all eight DeepMind Lab "laser tag" levels, demonstrating that improvements in controlled psychological tasks translate to broader RL performance.

Discussion and Future Directions

The current fovea model is a simplified approximation of the biological eye. Furthermore, the agent's failure on motion-based tasks indicates that standard LSTMs are insufficient for replicating human-like temporal processing or working memory.

Future work will utilize Psychlab to probe other cognitive domains, such as reinforcement learning schedules and complex decision-making. By tightening the loop between cognitive science and AI design, Psychlab provides a rigorous framework for developing more capable and human-like artificial agents.

Summary updated

Abstract

Psychlab is a simulated psychology laboratory inside the first-person 3D game world of DeepMind Lab (Beattie et al. 2016). Psychlab enables implementations of classical laboratory psychological experiments so that they work with both human and artificial agents. Psychlab has a simple and flexible API that enables users to easily create their own tasks. As examples, we are releasing Psychlab implementations of several classical experimental paradigms including visual search, change detection, random dot motion discrimination, and multiple object tracking. We also contribute a study of the visual psychophysics of a specific state-of-the-art deep reinforcement learning agent: UNREAL (Jaderberg et al. 2016). This study leads to the surprising conclusion that UNREAL learns more quickly about larger target stimuli than it does about smaller stimuli. In turn, this insight motivates a specific improvement in the form of a simple model of foveal vision that turns out to significantly boost UNREAL's performance, both on Psychlab tasks, and on standard DeepMind Lab tasks. By open-sourcing Psychlab we hope to facilitate a range of future such studies that simultaneously advance deep reinforcement learning and improve its links with cognitive science.
BibTeX
@misc{leibo2018psychlab,
  author = {Joel Z. Leibo and Cyprien de Masson d'Autume and Daniel Zoran and David Amos and Charles Beattie and Keith Anderson and Antonio García Castañeda and Manuel Sanchez and Simon Green and Audrunas Gruslys and Shane Legg and Demis Hassabis and Matthew Botvinick},
  title = {{Psychlab: A Psychology Laboratory for Deep Reinforcement Learning Agents}},
  year = {2018},
  eprint = {1801.08116},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1801.08116},
}
Figure