Augmenting Personalized Memory via Practical Multimodal Wearable Sensing in Visual Search and Wayfinding Navigation

Indrajeet Ghosh, Kasthuri Jayarajah, Nicholas R. Waytowich, Nirmalya Roy

Abstract

Human working memory is inherently limited, often leading to the loss of critical spatial and visual information during complex navigation tasks. While traditional "lifelogging" solutions rely on continuous video recording, they suffer from significant privacy concerns, high computational costs, and a lack of user-specific personalization. This paper introduces Memento, a multimodal wearable framework that leverages physiological signals—specifically EEG, skin conductance (GSR), and heart rate (PPG)—to detect "moments of significance." By identifying these high-attention episodes and utilizing them as personalized memory cues, Memento improves short-term route recall by up to 23% while reducing computational overhead by 75% compared to computer-vision-based approaches.

1. Introduction

Whether navigating an unfamiliar city or searching for specific items in a cluttered retail environment, humans frequently encounter more information than their working memory can retain. Existing technological interventions, such as wearable cameras that record every moment, present a "brute force" approach to memory augmentation. These methods are often socially invasive and computationally expensive, requiring power-intensive deep learning models to process high-resolution video. Perhaps most importantly, computer vision (CV) models typically prioritize "global saliency"—what is visually prominent to an average observer—rather than "local saliency," which reflects what a specific individual actually found important.

Memento addresses these challenges by shifting the focus from the environment to the user. By capturing the brain and body's natural signatures of attention—Event-Related Potentials (ERPs) and physiological arousal—the system creates a personalized record of significant moments, or "mementos," without the need for constant, invasive recording.

2. The Memento Framework

The Memento architecture integrates data from off-the-shelf (COTS) wearable sensors, including a 5-channel EEG headset and a wrist-worn sensor for GSR and PPG. The technical pipeline is divided into three critical stages:

2.1 Signal Reconstruction and Cleaning

Low-density COTS EEG signals are notoriously susceptible to noise and motion artifacts. Memento employs a functional connectivity-based reconstruction method combined with Independent Component Analysis (ICA). This process leverages correlations between different brain regions to "clean" the data, ensuring high signal fidelity even in mobile contexts.

2.2 Multimodal Fusion

To build a robust representation of the user’s cognitive state, the system fuses EEG features with the phasic component of GSR (indicative of emotional arousal) and systolic blood pressure estimates derived from PPG. This multimodal approach compensates for the limitations of any single sensor; for instance, GSR can help distinguish genuine attentional shifts from EEG artifacts like eye blinks.

2.3 Event Extraction via Change Point Detection

The system identifies "mementos" using a combination of Continuous Wavelet Transform (CWT) and a sliding-window Change Point Detection (CPD) algorithm. This allows the framework to pinpoint abrupt shifts in physiological activity that correspond to high-attentional episodes. These identified timestamps are then used to extract short video snippets from a wearable camera, providing the user with a curated set of memory cues.

3. Experimental Evaluation

The researchers validated Memento using the newly developed WoM (Wayfinding and Memory) dataset, collected through two distinct user studies:

  • Study 1 (Validation): 15 participants navigated four diverse virtual environments (ranging from a quiet campus to a dense NYC streetscape). This study confirmed that physiological sensors could accurately detect attention when participants were cued to look at specific objects.
  • Study 2 (Recall Performance): 25 participants performed navigation tasks and were subsequently tested on their ability to retrace routes. Their performance was compared across four conditions: No Aid (free recall), CV-based aid, Memento-EEG, and Memento-EGP (the full multimodal version).

4. Key Results and Insights

4.1 Enhanced Recall and Personalization

Participants using the full Memento-EGP system demonstrated a 20–23% improvement in route recall compared to free recall. Notably, Memento outperformed state-of-the-art CV models by 8–12%. This suggests that physiological cues are more effective than visual saliency models at capturing the specific landmarks a user relied on for navigation.

4.2 Computational Efficiency and Privacy

By processing 1D physiological time-series data instead of 2D video frames, Memento achieved a 75% reduction in computation time (3.86s vs. 15.35s for CV methods). Furthermore, because the system only triggers video recording or storage during "moments of significance," it significantly reduces the privacy risks associated with traditional lifelogging.

4.3 The Value of Multimodality

The integration of GSR and PPG proved vital. The multimodal fusion reduced noise artifacts by 22.8% and increased the detection rate of attentional instances from 49.8% (EEG only) to 72.6%.

5. Conclusion

Memento demonstrates that wearable physiological sensing offers a practical, efficient, and private alternative to traditional memory augmentation tools. The system is particularly effective in complex, "cluttered" environments where human cognitive load is highest. Future work will focus on transitioning the system from virtual environments to real-world AR/VR applications and exploring its potential to support individuals with cognitive decline or to enhance student learning through personalized review cues.

Summary updated

Abstract

Working memory involves the temporary retention of information over short periods. It is a critical cognitive function that enables humans to perform various online processing tasks, such as dialing a phone number, recalling misplaced items' locations, or navigating through a store. However, inherent limitations in an individual's capacity to retain information often result in forgetting important details during such tasks. Although previous research has successfully utilized wearable and assistive technologies to enhance long-term memory functions (e.g., episodic memory), their application to supporting short-term recall in daily activities remains underexplored. To address this gap, we present Memento, a framework that uses multimodal wearable sensor data to detect significant changes in cognitive state and provide intelligent in situ cues to enhance recall. Through two user studies involving 15 and 25 participants in visual search navigation tasks, we demonstrate that participants receiving visual cues from Memento achieved significantly better route recall, improving approximately 20-23% compared to free recall. Furthermore, Memento reduced cognitive load and review time by 46% while also substantially reducing computation time (3.86 seconds vs. 15.35 seconds), offering an average of 75% effectiveness compared to computer vision-based cue selection approaches.
BibTeX
@inproceedings{ghosh2025augmenting,
  author = {Indrajeet Ghosh and Kasthuri Jayarajah and Nicholas R. Waytowich and Nirmalya Roy},
  title = {{Augmenting Personalized Memory via Practical Multimodal Wearable Sensing in Visual Search and Wayfinding Navigation}},
  booktitle = {UMAP},
  year = {2025},
  doi = {10.1145/3699682.3728340},
}
Figure