Gamer in the scanner : Event-related analysis of fMRI activity during retro videogame play guided by automated annotations of game content

Yann Harel, Basile Pinsard, Julie A. Boyle, André Cyr, Maximilien Le Clei, Paul-Henri Mignot, Marie St‐Laurent, Karim Jerbi, Pierre BellecPsyArXiv 2023 · DOI

TL;DR

This paper introduces a novel method for automatically annotating gameplay events directly from a retro videogame's memory states, enabling more scalable and interpretable fMRI studies of naturalistic cognition. They demonstrate that these automated annotations can reliably map brain activity to specific in-game events, revealing both shared and individual neural signatures.

What Problem Does This Paper Address?

Studying cognition in naturalistic settings, like playing video games, offers rich insights but presents significant challenges for traditional neuroimaging analysis. The complex, dynamic nature of these environments makes it difficult to precisely define and model cognitive events. Previous approaches relied on manual annotations, which are time-consuming and limited in scope, or modified game versions, which restrict the choice of games and are also labor-intensive. This paper tackles the problem of creating a flexible, scalable, and automated method for annotating gameplay events to enable robust fMRI analysis of naturalistic cognitive processes.

Methods

The core innovation is an automated annotation framework built using the gym-retro Python library. This library allows for the emulation of retro video games and provides access to the game's internal memory states (RAM). By "mining" this RAM, the researchers could automatically identify and timestamp specific player actions (e.g., jumping, hitting) and game events (e.g., killing an enemy, losing health) without altering the game itself. These automated annotations were then used to construct event-related general linear models (GLMs) for fMRI data analysis. They also employed multivariate pattern analysis (MVPA) to assess the discriminability of brain responses to different game events.

Datasets

The study utilized data from four participants who played Shinobi III: Return of the Ninja Master (Sega, 1993) for over 7 hours each while undergoing fMRI. This gameplay data, along with corresponding automated annotations, formed the primary dataset. For comparison and validation, they also used data from the Human Connectome Project (HCP) test-retest dataset, which includes brain activation maps from more conventional functional localizer tasks.

Key Insights and Ideas

  • Automated Annotation is Scalable and Flexible: The gym-retro approach bypasses the need for manual annotation or game modification, making it applicable to a vast library of retro games and enabling large-scale data collection.
  • Naturalistic Events Elicit Distinct Brain Patterns: Even within a single, continuous gameplay session, different game events (like "Kill" vs. "HealthLoss") and player actions (like "JUMP" vs. "HIT") elicited distinct patterns of brain activation.
  • Individual Neural Signatures Emerge: While some brain responses were consistent across participants, there was also significant individual variability, suggesting that gameplay experiences shape unique neural representations.
  • Game Events are Highly Discriminable: MVPA demonstrated that brain activity patterns associated with different game events could be accurately decoded, highlighting the richness of information captured by these automated annotations.

Key Results

  • Robust Brain Activation Maps: GLM analyses using automated annotations produced interpretable brain activation maps, primarily engaging visual and sensorimotor cortices, consistent with the demands of the game.
  • High Classification Accuracy: MVPA achieved classification accuracies typically around or above 90% for distinguishing between different game events, demonstrating the discriminative power of the neural signals.
  • Within-Participant Consistency: Brain activation patterns showed robust consistency within participants across different sessions (r ≈ .25-.55), even for rarer events like HealthLoss.
  • Between-Participant Correlations Lower: Correlations between participants were lower, underscoring the individual nature of neural responses.

Limitations and Future Work

  • Limitations:
    • The approach relies on reverse-engineering memory addresses, which can be incomplete or tedious for some variables.
  • While temporal resolution is high (game framerate), aligning it perfectly with other high-temporal-resolution modalities (MEG, EEG) requires precise logging.
  • The GLM approach assumes low collinearity between regressors, necessitating separate analyses for each annotation. This might be a limitation for highly granular annotations.
  • The study involved only four participants, limiting the ability to draw firm conclusions about group-level effects.
  • Future Work:
    • Extending the framework to more complex cognitive processes like learning, planning, and affective evaluation.
  • Integrating physiological data (cardiac, respiratory) and phenomenological questionnaires for a more comprehensive understanding.
  • Exploring nonlinear modeling approaches (e.g., AI models) to better capture the dynamics of brain responses.
  • Leveraging the gym-retro platform to study a wider range of games and populations, including children and clinical populations.
  • Using automated annotations to train and validate brain-aligned artificial neural networks.

Summary updated

Abstract

In recent years, videogames have gathered interest in cognitive neuroscience for their potential to study cognition in dynamical and naturalistic contexts. Yet, inherent game complexity can push traditional modeling to its limits, and current annotation methodologies may not fully capture the depth of their content. Neuroimaging studies have resorted to manual annotations or employed modified game versions that offer greater experimental control, both approaches limiting and labor-intensive. Here, we introduce an innovative approach using the gym-retro Python library to emulate a retro action-platformer, Shinobi III: Return of the Ninja Master (Sega, 1993), which enables automated annotations from the game's memory states. We demonstrate how this method facilitates the study of the neural basis of game events, including player actions (jumping, hitting) and feedback events (killing an enemy or being hit). Four individuals played Shinobi while undergoing functional magnetic resonance magnetic imaging (fMRI). We used automated game annotation to derive event-based general linear models of the BOLD signal in each subject, and showed common and subject-specific activation patterns in visual and sensori-motor cortices. Multi-voxel pattern analysis (MVPA) conducted on session-level annotation-derived activation maps achieved highly accurate brain decoding across all annotations. Given that the gym-retro emulation platform supports over a thousand games, this study underscores the potential of automated game annotations to study complex cognitive processes in dynamic settings across a broad game spectrum.
BibTeX
@misc{harel2023gamer,
  author = {Yann Harel and Basile Pinsard and Julie A. Boyle and André Cyr and Maximilien Le Clei and Paul-Henri Mignot and Marie St‐Laurent and Karim Jerbi and Pierre Bellec},
  title = {{Gamer in the scanner : Event-related analysis of fMRI activity during retro videogame play guided by automated annotations of game content}},
  year = {2023},
  doi = {10.31234/osf.io/uakq9},
  howpublished = {PsyArXiv},
}
Figure