Magnetic resonance-based eye tracking using deep neural networks

Markus Frey, Matthias Nau, Christian F. DoellerNature Neuroscience 2021 · DOI

TL;DR

Researchers have developed DeepMReye, a novel deep learning system that tracks eye movements using only the magnetic resonance (MR) signal from the eyeballs, eliminating the need for expensive and cumbersome camera-based eye trackers. This breakthrough enables cameraless eye tracking in fMRI studies, even with existing datasets and when participants' eyes are closed, opening new avenues for understanding brain function and neurological disorders.

What Problem Does This Paper Address?

Functional magnetic resonance imaging (fMRI) is a powerful tool for studying brain activity, but interpreting its results is often complicated by eye movements. Viewing behavior is a critical factor that can influence brain activity, either as a variable of interest or as a confound. While MR-compatible camera eye trackers exist, they are expensive, require specialized setup, and cannot be used in all situations (e.g., with blind participants or in retrospective analyses). This limits the widespread adoption of eye tracking in fMRI research, potentially leading to misinterpretations of brain activity. The authors aim to bridge this gap by providing a freely available, cameraless, and versatile solution for eye tracking directly from MR signals.

Methods

The core of DeepMReye is a convolutional neural network (CNN) designed to decode gaze position from the MR signal of the eyeballs. The CNN takes 3D MR data from segmented eye masks as input. It employs a series of 3D convolutional layers, group normalization, and residual blocks for feature extraction. The model is trained using a combination of two loss functions: the Euclidean Error (EE) between predicted and true gaze positions, and a "predicted error" (PE) which acts as an unsupervised measure of decoding confidence. This PE is crucial for identifying unreliable decoding instances and participants. The system can perform "cameraless" eye tracking, meaning it doesn't require external cameras. It achieves sub-TR (repetition time) temporal resolution, meaning it can estimate gaze position at a finer temporal scale than the fMRI acquisition itself.

Datasets

The researchers trained and evaluated DeepMReye on a substantial dataset comprising 268 participants from five existing fMRI datasets. These datasets covered a variety of viewing behaviors, including:

  • Fixations: Participants fixated on specific points on a screen.
  • Smooth Pursuit: Participants tracked a moving target.
  • Free Viewing: Participants freely viewed images.
  • Visual Search: Participants searched for a target letter.

The data were acquired on five different 3T MRI scanners using 14 distinct scanning protocols, with TRs ranging from 800 to 2,500 ms and voxel sizes from 1.5 to 2.5 mm. This diverse dataset allowed for robust testing of the model's generalizability across different scanners, protocols, and viewing behaviors.

Key Insights and Ideas

  • Cameraless Eye Tracking from MR Signals: The most significant insight is the successful demonstration that eye movements can be reliably decoded directly from the MR signal of the eyeballs, bypassing the need for cameras. This is a paradigm shift for fMRI research.
  • Robustness Across Diverse Conditions: The model performs well across a wide range of scanning parameters (voxel size, TR) and viewing tasks, highlighting its broad applicability.
  • Works on Existing Datasets: A critical advantage is its ability to decode gaze from previously acquired fMRI data, unlocking the potential of vast existing datasets.
  • Eyes-Closed Tracking: The ability to track eye movements even when the eyes are closed opens up entirely new research possibilities, such as studying resting-state or sleep-related eye movements without visual confounds.
  • Unsupervised Reliability Estimation: The predicted error (PE) provides a built-in mechanism to assess decoding reliability, allowing researchers to filter out unreliable data points or participants.

Key Results

  • High Accuracy: DeepMReye achieved high decoding accuracy across various viewing behaviors, with median Pearson correlations (r) around 0.89 and R-squared values around 0.78 for the most reliable participants.
  • Sub-TR Resolution: The system can decode gaze at a temporal resolution of approximately 3 Hz, which is a moderate sub-TR resolution, enabling the capture of more dynamic eye movements than standard fMRI acquisition.
  • Minimal Training Data Required: A well-trained model can be achieved with as few as 6-8 participants and just 5-10 minutes of training data, significantly reducing the burden of data acquisition.
  • Explains Brain Activity: Decoded eye movements from DeepMReye successfully explained network-wide brain activity, including regions not typically associated with oculomotor function, comparable to camera-based eye tracking.
  • Robust to Scan Parameters: Decoding performance remained high across different voxel sizes and TRs, demonstrating its applicability in diverse fMRI settings.

Limitations and Future Work

  • Temporal Resolution: While achieving sub-TR resolution, DeepMReye's temporal resolution is still lower than that of camera-based systems, limiting its use for studying very rapid individual saccades.
  • Eyeball Inclusion: The method requires the eyeballs to be visible in the MR images, which may not always be the case. However, the authors note that brain activity decoding is still possible in such scenarios.
  • PE as a Guide, Not Absolute: The predicted error (PE) is a valuable indicator of reliability but should be used in conjunction with other quality assessment measures.
  • Future Work: The authors suggest exploring the impact of specific scanning sequence parameters (like slice timing) on sub-TR decoding resolution. Further validation of eyes-closed tracking in larger samples is also needed. They also envision extending the CNN architecture to decode other behavioral or stimulus features from brain activity.

Summary updated

Abstract

Viewing behavior provides a window into many central aspects of human cognition and health, and it is an important variable of interest or confound in many functional magnetic resonance imaging (fMRI) studies. To make eye tracking freely and widely available for MRI research, we developed DeepMReye, a convolutional neural network (CNN) that decodes gaze position from the magnetic resonance signal of the eyeballs. It performs cameraless eye tracking at subimaging temporal resolution in held-out participants with little training data and across a broad range of scanning protocols. Critically, it works even in existing datasets and when the eyes are closed. Decoded eye movements explain network-wide brain activity also in regions not associated with oculomotor function. This work emphasizes the importance of eye tracking for the interpretation of fMRI results and provides an open source software solution that is widely applicable in research and clinical settings.
BibTeX
@inproceedings{frey2021magnetic,
  author = {Markus Frey and Matthias Nau and Christian F. Doeller},
  title = {{Magnetic resonance-based eye tracking using deep neural networks}},
  booktitle = {Nature Neuroscience},
  year = {2021},
  doi = {10.1038/s41593-021-00947-w},
}
Figure