Semantics-based selection of everyday concepts in visual lifelogging

Peng Wang, Alan F. SmeatonInternational Journal of Multimedia Information Retrieval 2012 · DOI

Introduction

Lifelogging involves the continuous digital recording of a person's daily life, often utilizing wearable cameras like the Microsoft SenseCam. While this technology creates a rich "digital memory," it also generates a massive volume of low-quality, noisy images. Traditional computer vision approaches for concept detection—identifying objects or scenes—are typically optimized for high-quality media like broadcast TV. These models struggle with the mundane, blurry, and varied nature of everyday lifelogs. Furthermore, a significant "semantic gap" exists: a disconnect between low-level image features (pixels and edges) and the high-level semantic meaning of an activity (e.g., "having a meeting"). This paper proposes a framework to intelligently select which concepts are most relevant for representing specific daily activities to improve searching and browsing.

The Microsoft SenseCam and its application in wearable lifelogging.

Methodology

The authors introduce a multi-stage approach to build an Event Semantic Space (EES):

  1. Concept Selection via Semantic Density: Instead of selecting individual keywords, the authors use a density-based approach. They apply Principal Component Analysis (PCA) and hierarchical clustering to group related concepts, identifying "clusters" that naturally appear together in specific activities.
  2. Ontology Comparison: The study compares two types of semantic networks: WordNet, a lexical database focusing on "is-a" relationships (e.g., a "cup" is a "vessel"), and ConceptNet, a common-sense knowledge base capturing contextual relationships (e.g., a "cup" is used for "drinking").
  3. Global Ranking with Random Walk: Once "seed" concepts are identified, a Random Walk algorithm (similar to PageRank) propagates importance across the semantic network. This allows the system to discover and rank relevant concepts that might not have been initially detected but are semantically close to the activity.

Experimental Setup

The study relied on a custom-built dataset derived from user experiments:

  • Target Activities: 23 everyday activities were selected based on frequency (e.g., eating, cycling, using a phone).
  • User Study: 13 participants (researchers familiar with lifelogging) reviewed SenseCam image streams and listed concepts helpful for retrieving those activities.
  • Concept Pool: This resulted in a ground-truth set of 171 everyday concepts. Algorithms were evaluated on both a "SmallSet" (85 concepts with high user agreement) and a "LargeSet" (all 171 concepts).

Key Insights and Results

The most significant finding is that contextual similarity (ConceptNet) consistently outperforms lexical similarity (WordNet) for lifelogging. For daily life, knowing that a "plate" is found on a "table" during "eating" is far more useful than knowing a "plate" is a type of "dishware."

By looking for "dense" clusters of concepts rather than isolated detections, the system becomes more robust to the noise inherent in automatic image recognition. Furthermore, the Random Walk ranking allows the system to "fill in the blanks." If a camera misses a clear shot of a "plate" but detects a "fork" and a "table," the system can still correctly infer that "eating" is the likely activity.

Evaluation using metrics like Group Consistency and Rank Correlation showed that ConceptNet-based selection achieved the highest median scores. The performance gap between the proposed contextual approach and traditional lexical methods widened as the number of concepts increased, demonstrating the system's robustness.

Conclusion

While the system performs well, some activities remain challenging. Procedural tasks like "cooking" or general concepts like "shopping" are difficult to define with a consistent set of visual concepts. Future work will involve integrating temporal information and exploring more complex event-based reasoning to further bridge the semantic gap in personal digital archives.

Summary updated

BibTeX
@article{wang2012semantics,
  author = {Peng Wang and Alan F. Smeaton},
  title = {{Semantics-based selection of everyday concepts in visual lifelogging}},
  journal = {International Journal of Multimedia Information Retrieval},
  year = {2012},
  doi = {10.1007/s13735-012-0010-8},
}
Figure