ConvLogMiner: A Real-Time Conversational Lifelog Miner

Pei-Wei Kao, An-Zi Yen, Hen-Hsen Huang, Hsin-Hsi ChenIJCAI 2021 · DOI

TL;DR

This paper introduces ConvLogMiner, a novel system that automatically detects personal life events from real-time human conversations. By analyzing spoken or written dialogue, ConvLogMiner aims to enrich lifelogging data, offering a more comprehensive view of an individual's experiences beyond traditional methods like social media posts or photos.

What Problem Does This Paper Address?

Lifelogging, the practice of digitally capturing daily life experiences, has gained traction for its applications in lifestyle analysis and memory assistance. However, existing lifelogging methods primarily focus on extracting information from social media, photos, or videos. This leaves a significant gap: the rich, nuanced information embedded within everyday human conversations. Conversations not only convey information but also reveal personal activities, experiences, and even emotional states that are crucial for a complete lifelog. The challenge lies in accurately and automatically extracting these "life events" from the dynamic and often ambiguous nature of spoken dialogue, which involves challenges like pronoun resolution (co-reference) and inferring events that depend on multiple turns of conversation.

Methods

ConvLogMiner is built upon the XLNet framework and comprises three key modules designed to address the complexities of conversational lifelogging:

  • Real-Time Event Detector: This module focuses on identifying new life events triggered by the most recent utterance in a conversation. It processes a sliding window of past turns to capture immediate context.
  • Speaker Attentive Event Detector: This auxiliary module aims to extract life events solely based on the utterances of a specific speaker, filtering out potential noise from the other participant.
  • Entire Conversation Analyzer: This module considers the entire conversation to infer life events, recognizing that some events might only become clear when viewed in the broader context of the dialogue.

The system operates in two modes: an online service that provides instant detection of new events as they occur, and an offline service that refines these detections using an ensemble mechanism combining the predictions of all three modules. This ensemble approach uses majority voting to produce a final, more robust set of detected life events, which are then stored in a personal knowledge base.

Datasets

The authors introduce a new dataset called DiaLog, created by extending the DailyDialog dataset. DiaLog consists of 600 multi-turn conversations (4-6 utterances) across four topics: Ordinary Life, Attitude & Emotion, Relationship, and Tourism. These conversations are annotated with life events, with 780 events in total. The dataset was annotated by three linguistically trained annotators, achieving a high inter-annotator agreement (Krippendorff's alpha of 0.81). The dataset is intended to facilitate research in life event detection from daily dialogues.

Key Insights and Ideas

  • Conversations as a Rich Source of Lifelogs: The paper strongly advocates for conversations as an underutilized but highly valuable source of personal life event data, complementing existing lifelogging methods.
  • The Importance of Context: The necessity of considering different levels of context (recent turns, individual speaker's turns, entire conversation) is highlighted by the performance of the different modules.
  • Ensemble for Robustness: Combining predictions from multiple specialized modules (real-time, speaker-specific, and full-conversation) significantly improves the overall accuracy of life event detection.
  • Challenges in Conversational Analysis: The paper explicitly calls out challenges like pronoun resolution and the need to infer events from multiple utterances, underscoring the complexity of the task.

Key Results

The ConvLogMiner system demonstrated promising performance on the DiaLog dataset. The ensemble mechanism, which combines the predictions of the three modules, achieved the highest macro F1 score of 45.30%. While the real-time event detector (XLNetAggre) performed well with an F1 of 44.21%, the ensemble approach provided a better refinement. The speaker attentive event detector (XLNetSelfTurns) and the entire conversation analyzer (XLNetAllTurns) had F1 scores of 42.33% and 39.56%, respectively, with XLNetAllTurns achieving the highest recall (48.27%).

Limitations and Future Work

The authors acknowledge several limitations and outline future research directions:

  • Focus on Event Types: Currently, ConvLogMiner only detects the types of life events. Future work aims to extract the subject, predicate, and object of these events to form structured triples for better organization in personal knowledge bases.
  • Privacy Concerns: Detecting life events from conversations raises significant privacy concerns. The authors plan to investigate privacy-aware lifelogging frameworks to address this issue.
  • Real-World Deployment: While demonstrated as a pilot system, further work is needed for robust real-world deployment and handling of diverse conversational styles and noise.

Summary updated

BibTeX
@inproceedings{kao2021convlogminer,
  author = {Pei-Wei Kao and An-Zi Yen and Hen-Hsen Huang and Hsin-Hsi Chen},
  title = {{ConvLogMiner: A Real-Time Conversational Lifelog Miner}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/710},
}
Figure