Abstract
Lifelogging involves passively capturing personal daily events via wearable devices, creating a visual diary of one's life. LifeSeeker is an interactive retrieval system designed to navigate these massive datasets. By combining textual descriptions, spatial relationships, location information, and image similarities, LifeSeeker aids users in recollecting specific moments. This paper details the system's architecture, its performance in the Lifelog Search Challenge (LSC), and the observed performance gap between expert and novice users.
Introduction
The practice of lifelogging generates hundreds of thousands of egocentric images, presenting a significant information retrieval challenge known as the "semantic gap." Users often remember the context of an event—such as the time of day or a nearby landmark—better than specific visual details. LifeSeeker addresses this by providing a multi-modal search interface that aligns technical queries with human associative memory.
System Architecture and Methods
LifeSeeker employs a multi-tiered approach to bridge the gap between raw data and user queries. The system architecture is divided into three main layers: Database, Indexing, and Retrieval.
Multi-Modal Indexing
The system processes data into searchable metadata across four primary categories:
- Temporal Data: Indexed by date, month, and specific periods (e.g., morning).
- Location: GPS coordinates are converted into semantic addresses and categorized into 32 place types (e.g., "office").
- Visual Concepts: Utilizes YOLOv4 for object detection, Bottom-up Attention for attributes, and PlacesCNN for scene recognition.
- OCR: Extracts text from the environment, such as signs or screens, to provide additional context.
Search and Retrieval
The system uses a Weighted Bag-of-Words (BoW) model that prioritizes time and location filters. For advanced users, an Elastic Search integration allows for complex syntax-based queries. A key feature of the UI is the temporal browsing panel, which allows users to view images immediately preceding or following a search result, facilitating navigation through "landmark" events.
Results and Evaluation
LifeSeeker was evaluated using the LSC’21 dataset, comprising over 183,000 images. In the 2021 Lifelog Search Challenge, the system ranked 3rd overall and achieved the highest recall, solving 20 out of 23 queries (a recall score of 0.8696).
A follow-up user study highlighted a significant "Expert Gap." While the system developer achieved a score of 523.80, the top novice user scored 346.95. This discrepancy stems from the expert's familiarity with the system's specific vocabulary, whereas novices often struggled with synonyms not present in the fixed concept index.
Conclusion and Future Work
LifeSeeker demonstrates the power of combining multi-modal indexing with temporal browsing for lifelog retrieval. However, the reliance on a fixed library of visual tags remains a bottleneck. Future developments will focus on query expansion using NLP to bridge the vocabulary gap for novice users and integrating biometric data to filter moments based on the user's physical or emotional state.
Summary updated