Abstract
Human working memory is inherently limited, often leading to the loss of critical spatial and visual information during complex navigation tasks. While traditional "lifelogging" solutions rely on continuous video recording, they suffer from significant privacy concerns, high computational costs, and a lack of user-specific personalization. This paper introduces Memento, a multimodal wearable framework that leverages physiological signals—specifically EEG, skin conductance (GSR), and heart rate (PPG)—to detect "moments of significance." By identifying these high-attention episodes and utilizing them as personalized memory cues, Memento improves short-term route recall by up to 23% while reducing computational overhead by 75% compared to computer-vision-based approaches.
1. Introduction
Whether navigating an unfamiliar city or searching for specific items in a cluttered retail environment, humans frequently encounter more information than their working memory can retain. Existing technological interventions, such as wearable cameras that record every moment, present a "brute force" approach to memory augmentation. These methods are often socially invasive and computationally expensive, requiring power-intensive deep learning models to process high-resolution video. Perhaps most importantly, computer vision (CV) models typically prioritize "global saliency"—what is visually prominent to an average observer—rather than "local saliency," which reflects what a specific individual actually found important.
Memento addresses these challenges by shifting the focus from the environment to the user. By capturing the brain and body's natural signatures of attention—Event-Related Potentials (ERPs) and physiological arousal—the system creates a personalized record of significant moments, or "mementos," without the need for constant, invasive recording.
2. The Memento Framework
The Memento architecture integrates data from off-the-shelf (COTS) wearable sensors, including a 5-channel EEG headset and a wrist-worn sensor for GSR and PPG. The technical pipeline is divided into three critical stages:
2.1 Signal Reconstruction and Cleaning
Low-density COTS EEG signals are notoriously susceptible to noise and motion artifacts. Memento employs a functional connectivity-based reconstruction method combined with Independent Component Analysis (ICA). This process leverages correlations between different brain regions to "clean" the data, ensuring high signal fidelity even in mobile contexts.
2.2 Multimodal Fusion
To build a robust representation of the user’s cognitive state, the system fuses EEG features with the phasic component of GSR (indicative of emotional arousal) and systolic blood pressure estimates derived from PPG. This multimodal approach compensates for the limitations of any single sensor; for instance, GSR can help distinguish genuine attentional shifts from EEG artifacts like eye blinks.
2.3 Event Extraction via Change Point Detection
The system identifies "mementos" using a combination of Continuous Wavelet Transform (CWT) and a sliding-window Change Point Detection (CPD) algorithm. This allows the framework to pinpoint abrupt shifts in physiological activity that correspond to high-attentional episodes. These identified timestamps are then used to extract short video snippets from a wearable camera, providing the user with a curated set of memory cues.
3. Experimental Evaluation
The researchers validated Memento using the newly developed WoM (Wayfinding and Memory) dataset, collected through two distinct user studies:
- Study 1 (Validation): 15 participants navigated four diverse virtual environments (ranging from a quiet campus to a dense NYC streetscape). This study confirmed that physiological sensors could accurately detect attention when participants were cued to look at specific objects.
- Study 2 (Recall Performance): 25 participants performed navigation tasks and were subsequently tested on their ability to retrace routes. Their performance was compared across four conditions: No Aid (free recall), CV-based aid, Memento-EEG, and Memento-EGP (the full multimodal version).
4. Key Results and Insights
4.1 Enhanced Recall and Personalization
Participants using the full Memento-EGP system demonstrated a 20–23% improvement in route recall compared to free recall. Notably, Memento outperformed state-of-the-art CV models by 8–12%. This suggests that physiological cues are more effective than visual saliency models at capturing the specific landmarks a user relied on for navigation.
4.2 Computational Efficiency and Privacy
By processing 1D physiological time-series data instead of 2D video frames, Memento achieved a 75% reduction in computation time (3.86s vs. 15.35s for CV methods). Furthermore, because the system only triggers video recording or storage during "moments of significance," it significantly reduces the privacy risks associated with traditional lifelogging.
4.3 The Value of Multimodality
The integration of GSR and PPG proved vital. The multimodal fusion reduced noise artifacts by 22.8% and increased the detection rate of attentional instances from 49.8% (EEG only) to 72.6%.
5. Conclusion
Memento demonstrates that wearable physiological sensing offers a practical, efficient, and private alternative to traditional memory augmentation tools. The system is particularly effective in complex, "cluttered" environments where human cognitive load is highest. Future work will focus on transitioning the system from virtual environments to real-world AR/VR applications and exploring its potential to support individuals with cognitive decline or to enhance student learning through personalized review cues.
Summary updated