Design and evaluation of a global workspace agent embodied in a realistic multimodal environment

Rousslan Fernand Julien Dossa, Kai Arulkumaran, Arthur Juliani, Shuntaro Sasai, Ryota KanaiFrontiers Comput. Neurosci 2024 · DOI

TL;DR

This paper introduces a novel embodied artificial agent designed based on Global Workspace Theory (GWT) to explore higher-level cognitive processes like consciousness in AI. The agent, trained in a realistic audiovisual navigation task, demonstrates improved robustness and performance with smaller working memory capacities compared to a standard recurrent architecture, suggesting that a constrained workspace can incentivize more dynamic attentional strategies.

What Problem Does This Paper Address?

As artificial neural networks (ANNs) become increasingly sophisticated, researchers are drawing parallels between their capabilities and human cognitive functions. While ANNs have shown success in mimicking specific brain functions like visual processing or language understanding, exploring higher-level cognitive processes, such as conscious information access, has been challenging due to a lack of concrete, testable hypotheses. Global Workspace Theory (GWT) offers a framework for understanding consciousness, but previous computational implementations have often been limited to unimodal inputs and lacked embodiment in realistic environments. This paper aims to bridge this gap by designing and evaluating an embodied agent that adheres to GWT's "indicator properties" within a multimodal, realistic 3D environment.

Methods

The core of this research is the development of a "global workspace agent" that explicitly incorporates the indicator properties of GWT. This agent is designed with specialized input modules for visual and acoustic modalities, each employing recurrent neural networks (RNNs) with gated recurrent units (GRUs) to process temporal information. A key innovation is the integration of a cross-attention mechanism that allows the agent to selectively attend to information from these sensory inputs and its own working memory. This mechanism facilitates a "global broadcast" of information, making it accessible to downstream modules, a central tenet of GWT.

The agent is trained using imitation learning, specifically behavioral cloning, on expert trajectory data. This approach was chosen to ensure a high baseline performance and reduce the variance often associated with reinforcement learning in complex tasks. The researchers also developed a baseline GRU agent that mirrors the global workspace agent but omits the GWT-specific features, particularly the cross-attention mechanism and global broadcast, to allow for direct comparison.

Datasets

The agents were trained and evaluated using the SoundSpaces library, a simulator that provides realistic 3D environments with audiovisual navigation tasks. Specifically, the SAVi (Semantic Audiovisual Navigation) extension was utilized. SAVi offers a richer variety of target object categories and their associated acoustic properties compared to the original SoundSpaces, and crucially, it provides acoustic cues for a variable duration, making the task more challenging and requiring better working memory capabilities. The researchers generated a dataset of 500,000 tuples of (observation, action, environment termination) samples from an oracle agent within this simulator.

Key Insights and Ideas

  • Embodiment and Multimodality are Crucial: The study highlights that testing GWT in a realistic, multimodal, embodied environment is essential for uncovering the potential benefits of such architectures, moving beyond unimodal or abstract tasks.
  • Bottlenecks Drive Dynamic Attention: A smaller, more constrained global workspace (bottleneck) appears to force the agent to develop more dynamic and mixed attentional patterns, integrating information from different modalities more effectively.
  • GWT Agents are More Robust: The global workspace agent demonstrated greater robustness, particularly at smaller working memory sizes, compared to the baseline GRU agent.
  • Global Broadcast Prioritizes Past Information: Analysis of the agent's internal weights suggests that the global broadcast mechanism effectively prioritizes information from the previous time step (working memory) over current sensory inputs for the recurrent encoders, indicating its role in propagating information over time.
  • SAVi Task May Not Fully Stress Working Memory: Despite the design of the SAVi task to challenge working memory, the results suggest that the task might not be sufficiently complex to fully reveal the advantages of a global workspace, as performance differences between agents were not drastic across all working memory sizes.

Key Results

  • The global workspace agent achieved a relatively robust success rate of approximately 80% across various working memory sizes, outperforming the baseline GRU agent which achieved around 60% and only became competitive at higher working memory sizes.
  • Probing accuracy for the target object category was highest when conditioned on acoustic features, with visual features also providing significant information.
  • The global workspace agent showed better integration of information into its working memory for predicting the target object category, especially when acoustic cues were present.
  • Attention analysis revealed that agents largely perform cross-attention between visual and audio modalities, with the working memory being matched with one or both inputs. The working memory and null keys were largely unattended, suggesting that current inputs are prioritized, but the global broadcast ensures past information is still utilized.
  • The normalized importance weights indicated that the previous working memory was prioritized over current sensory inputs in the recurrent encoders, supporting the role of the global broadcast in information propagation.

Limitations and Future Work

The authors acknowledge that the SAVi task, while more complex than previous benchmarks, might not be challenging enough to fully stress the working memory capacity of the agents. This suggests that future work should investigate even more complex environments and tasks to better evaluate higher-level cognitive mechanisms. They also propose refining the "indicator properties" of GWT with more insights from neuroscience.

Future research directions include:

  • Further investigating the role of the global workspace as an informational bottleneck.
  • Exploring other quantitative and qualitative properties, such as connectivity patterns and the emergence of cross-modal analogies within the shared workspace.
  • Developing more complex simulator environments for training and testing artificial agents.

Summary updated

BibTeX
@article{dossa2024design,
  author = {Rousslan Fernand Julien Dossa and Kai Arulkumaran and Arthur Juliani and Shuntaro Sasai and Ryota Kanai},
  title = {{Design and evaluation of a global workspace agent embodied in a realistic multimodal environment}},
  journal = {Frontiers Comput. Neurosci},
  year = {2024},
  doi = {10.3389/fncom.2024.1352685},
}
Figure