Development and validation of a nonverbal consensus-based semantic memory paradigm in patients with epilepsy

Edwina B. Tran, Jet M. J. Vonk, Kaitlin B. Casaletto, Da Zhang, Raphaël Christin, Siddharth Marathe, Maria Luisa Gorno‐Tempini, Edward F. Chang, Jonathan K. KleenJournal of the International Neuropsychological Society 2024 · DOI

Abstract

This study demonstrated performance impairments in patients with epilepsy (PWE) that may reflect dysfunction of nonverbal semantic memory circuits, such as seizure onset zones overlapping with key semantic regions (e.g., anterior temporal lobe). The Visual-based Semantic Association Task (ViSAT) paradigm avoids language-based confounds, is repeatable for longitudinal studies, captures detailed behavioral data, and is open-source. We propose it as a strong alternative for clinical and research assessment of nonverbal semantic memory.

Introduction

Semantic memory is our internal library of general knowledge, allowing us to associate concepts like "cherry" with "pie" or "space shuttle" with "moon." In clinical neurology, assessing this function is vital because many conditions, particularly epilepsy, affect the anterior temporal lobe—a critical hub for semantic processing.

However, traditional assessments face significant hurdles. Most rely on verbal naming, making it difficult to distinguish between a loss of the concept and a language deficit (aphasia). Furthermore, legacy nonverbal tests like the Pyramids & Palm Trees (PPT) test are often proprietary, use limited black-and-white drawings, and lack the ability to track reaction times or account for the subjective nature of human associations.

The ViSAT Paradigm

To address these limitations, the researchers developed the Visual-based Semantic Association Task (ViSAT). This open-source tool consists of 100 unique trials featuring real-life color photographs. In each trial, participants view a stimulus image and must select the most related image from four choices.

The ViSAT task structure (B) and the distribution of semantic categories across stimuli (D), answers (E), and relations (F).

To ensure the task measures meaning rather than simple visual matching, the team used a deep learning model (ResNet-18) to confirm that correct answers were not more visually similar to the stimulus than distractors. The task was administered via a MATLAB-based interface that separately timed the perception stage (viewing the target) and the association stage (choosing the answer).

Validation and Consensus-Based Scoring

Unlike traditional tests that assume a single binary "correct" answer, ViSAT uses consensus-based scoring derived from crowdsourced data. This treats semantic association as a probabilistic human agreement. Validation against established benchmarks showed that ViSAT achieved a consensus of over 90% in the vast majority of its trials, significantly outperforming older instruments like the Camels & Cactus Test (CCT).

Comparison of stimulus types in traditional tests (A) and the distribution of percent consensus (C) for PPT, CCT, and ViSAT.

Clinical Findings in Epilepsy

The study compared healthy controls against a group of patients with epilepsy (PWE). The results revealed specific deficits in the clinical group:

  • Accuracy: PWE had significantly lower accuracy (91.4%) compared to controls (94.4%).
  • The "Semantic Delay": While PWE were slower overall, their delay was disproportionately larger during the association phase (choosing the answer) compared to the perception phase. This suggests a specific bottleneck in semantic retrieval rather than a general slowing of visual processing.
  • Anatomical Correlation: Patients with seizure onset zones in the temporal lobe were particularly affected, reinforcing the role of these regions in nonverbal semantic memory.

Conclusion and Future Directions

ViSAT provides a modernized, culturally neutral, and rigorous method for assessing semantic memory. While the current study's sample size limits definitive mapping of specific brain regions to error types, the tool's open-source nature allows for wide adoption and future cross-cultural validation. As a nonverbal, repeatable paradigm, ViSAT is well-positioned for longitudinal tracking of cognitive health in patients with epilepsy and other neurological disorders.

Summary updated

Abstract

This study demonstrated performance impairments in PWE that may reflect dysfunction of nonverbal semantic memory circuits, such as seizure onset zones overlapping with key semantic regions (e.g., anterior temporal lobe). The ViSAT paradigm avoids confounds, is repeatable/longitudinal, captures behavioral data, and is open-source, thus we propose it as a strong alternative for clinical and research assessment of nonverbal semantic memory.
BibTeX
@article{tran2024development,
  author = {Edwina B. Tran and Jet M. J. Vonk and Kaitlin B. Casaletto and Da Zhang and Raphaël Christin and Siddharth Marathe and Maria Luisa Gorno‐Tempini and Edward F. Chang and Jonathan K. Kleen},
  title = {{Development and validation of a nonverbal consensus-based semantic memory paradigm in patients with epilepsy}},
  journal = {Journal of the International Neuropsychological Society},
  year = {2024},
  doi = {10.1017/s1355617724000158},
}
Figure