Summarization of LifeLog Data Based on Implicit Feedback from Personal Browsing History

Koki Hatada, Toshihiko Yamasaki, Kiyoharu AizawaThe Journal of The Institute of Image Information and Television Engineers 2010 · DOI

Abstract

LifeLog data is summarized by extracting the data that has most attracted users from vast amounts of information in the LifeLog data. However, because this information is personal and its characteristics vary from person to person, a new technique for estimating individual user's interests is needed. We propose a method that infers the importance of each LifeLog photo based on the user's photo browsing history. We developed a browsing system that displays photos captured with wearable cameras and records the photo browsing history of the user. Using visual techniques such as HSV histograms and bag-of-features in combination with the RankBoost algorithm enabled us to successfully estimate the importance of unknown photos.

Introduction

The "Lifelog" concept—recording every moment of one's life via wearable sensors and cameras—promises a perfect digital memory. However, the reality is a data deluge. A device like the Microsoft SenseCam can capture thousands of images a day, many of which are redundant, blurry, or mundane. The core challenge is summarization: extracting the "highlights" of a day. While previous research relied on objective sensor data (like changes in light or movement) to segment events, "importance" is deeply subjective. This paper addresses the gap between generic event segmentation and personalized importance estimation by leveraging a user's natural browsing behavior.

Methodology

The proposed system consists of a closed-loop pipeline for data collection, feedback gathering, and predictive modeling.

Data Collection and Hardware

Researchers utilized a SenseCam, a wearable device equipped with a camera and multiple sensors, including an accelerometer, thermometer, light sensor, and infrared sensor. This device captures a continuous stream of first-person perspective images throughout the user's day.

The SenseCam wearable device used for continuous data collection.

Implicit Feedback Browser

To capture subjective interest without tedious manual tagging, a custom photo-viewing application was developed. As the user browses their day, the system records "implicit feedback." If a user clicks on a thumbnail to view a larger version or zooms in, the system assumes that image is more important than those ignored. This behavioral data serves as the ground truth for training the importance model.

The custom browsing interface displays scenes chronologically and records user interactions as implicit feedback.

Feature Extraction and RankBoost

The model analyzes three distinct feature sets:

  1. Sensor Data: An 8-dimensional vector (acceleration, temperature, etc.).
  2. HSV Histograms: A 128-dimensional vector representing color distribution.
  3. Bag-of-Features (SIFT): A 100-dimensional vector representing local visual patterns, robust to lighting and angle changes.

These features are processed using RankBoost, an algorithm designed to learn preferences from pairwise comparisons. If Image A was interacted with more than Image B, the model learns the specific visual or sensor features that distinguish A as more desirable.

Experimental Results

The study used a dataset collected by a single user over 13 days. The first 9 days were used for training, and the final 4 days served as the evaluation set.

Key findings include:

  • High Precision: When predicting the single most important image for an event, the system achieved approximately 75% accuracy.
  • Visual Dominance: Visual features (HSV and SIFT) were significantly more effective than raw sensor data. The sensor-only model performed only slightly better than random selection, suggesting that environmental factors like temperature or movement are poor proxies for personal meaning.
  • Feature Synergy: The best performance was achieved by combining all three feature types (Sensor + HSV + SIFT), allowing the system to recognize specific scene types (e.g., a dinner table or a face) as consistently important.

Conclusion and Future Work

This research demonstrates that implicit feedback is a scalable and effective way to personalize lifelog summarization. By learning from what users actually do rather than what they say, the system bridges the gap between raw data and human interest.

Future work will address the current limitations, such as the small sample size (one user) and the ambiguity of browsing intent (e.g., clicking a photo because it is blurry vs. because it is interesting). The ultimate goal is to integrate these predictions into automated "daily digest" generators for personalized digital memories.

Summary updated

Abstract

LifeLog data is summarized by extracting the data that has most attracted users from vast amounts of information in the LifeLog data. However, because this information is personal and its characteristics vary from person to person, a new technique for estimating individual user's interests is needed. We propose a method that infers the importance of each LifeLog photo based on the user's photo browsing history. We developped a browsing system that displays photos captured with wearable cameras and records the photo browsing history of the user. Using visual techniques such as HSV histograms and bag-of-features in combination with the RankBoost algorithm enabled us to successfully estimate the importance of unknown photos.
BibTeX
@article{hatada2010summarization,
  author = {Koki Hatada and Toshihiko Yamasaki and Kiyoharu Aizawa},
  title = {{Summarization of LifeLog Data Based on Implicit Feedback from Personal Browsing History}},
  journal = {The Journal of The Institute of Image Information and Television Engineers},
  year = {2010},
  doi = {10.3169/itej.64.237},
}
Figure