LifeSeeker: an interactive concept-based retrieval system for lifelog data

Thao-Nhu Nguyen, Tu-Khiem Le, Van-Tu Ninh, Annalina Caputo, Graham Healy, Sinéad Smyth, Minh-Triet Tran, Binh T. NguyenMultimedia Tools and Applications 2023 · OpenReview

Abstract

Lifelogging involves passively capturing personal daily events via wearable devices, creating a visual diary of one's life. LifeSeeker is an interactive retrieval system designed to navigate these massive datasets. By combining textual descriptions, spatial relationships, location information, and image similarities, LifeSeeker aids users in recollecting specific moments. This paper details the system's architecture, its performance in the Lifelog Search Challenge (LSC), and the observed performance gap between expert and novice users.

Introduction

The practice of lifelogging generates hundreds of thousands of egocentric images, presenting a significant information retrieval challenge known as the "semantic gap." Users often remember the context of an event—such as the time of day or a nearby landmark—better than specific visual details. LifeSeeker addresses this by providing a multi-modal search interface that aligns technical queries with human associative memory.

An example of an egocentric lifelog image where OCR can extract environmental text like "NO JUNK MAIL".

System Architecture and Methods

LifeSeeker employs a multi-tiered approach to bridge the gap between raw data and user queries. The system architecture is divided into three main layers: Database, Indexing, and Retrieval.

The LifeSeeker system architecture, illustrating the flow from database indexing to the interactive search interface.

Multi-Modal Indexing

The system processes data into searchable metadata across four primary categories:

  • Temporal Data: Indexed by date, month, and specific periods (e.g., morning).
  • Location: GPS coordinates are converted into semantic addresses and categorized into 32 place types (e.g., "office").
  • Visual Concepts: Utilizes YOLOv4 for object detection, Bottom-up Attention for attributes, and PlacesCNN for scene recognition.
  • OCR: Extracts text from the environment, such as signs or screens, to provide additional context.

Search and Retrieval

The system uses a Weighted Bag-of-Words (BoW) model that prioritizes time and location filters. For advanced users, an Elastic Search integration allows for complex syntax-based queries. A key feature of the UI is the temporal browsing panel, which allows users to view images immediately preceding or following a search result, facilitating navigation through "landmark" events.

Results and Evaluation

LifeSeeker was evaluated using the LSC’21 dataset, comprising over 183,000 images. In the 2021 Lifelog Search Challenge, the system ranked 3rd overall and achieved the highest recall, solving 20 out of 23 queries (a recall score of 0.8696).

A follow-up user study highlighted a significant "Expert Gap." While the system developer achieved a score of 523.80, the top novice user scored 346.95. This discrepancy stems from the expert's familiarity with the system's specific vocabulary, whereas novices often struggled with synonyms not present in the fixed concept index.

Conclusion and Future Work

LifeSeeker demonstrates the power of combining multi-modal indexing with temporal browsing for lifelog retrieval. However, the reliance on a fixed library of visual tags remains a bottleneck. Future developments will focus on query expansion using NLP to bridge the vocabulary gap for novice users and integrating biometric data to filter moments based on the user's physical or emotional state.

Summary updated

Abstract

Lifelogging was introduced as the process of passively capturing personal daily events via wearable devices. It ultimately creates a visual diary encoding every aspect of one’s life with the aim of future sharing or recollecting. In this paper, we present LifeSeeker, a lifelog image retrieval system participating in the Lifelog Search Challenge (LSC) for 3 years, since 2019. Our objective is to support users to seek specific life moments using a combination of textual descriptions, spatial relationships, location information, and image similarities. In addition to the LSC challenge results, a further experiment was conducted in order to evaluate the power retrieval of our system on both expert and novice users. This experiment informed us about the effectiveness of the user’s interaction with the system when involving non-experts.
BibTeX
@article{nguyen2023lifeseeker,
  author = {Thao-Nhu Nguyen and Tu-Khiem Le and Van-Tu Ninh and Annalina Caputo and Graham Healy and Sinéad Smyth and Minh-Triet Tran and Binh T. Nguyen},
  title = {{LifeSeeker: an interactive concept-based retrieval system for lifelog data}},
  journal = {Multimedia Tools and Applications},
  year = {2023},
  url = {https://openreview.net/forum?id=f-vY72h6pb},
}
Figure