Situation-Based Multiparticipant Chat Summarization: a Concept, an Exploration-Annotation Tool and an Example Collection

Anna Smirnova, Evgeniy Slobodkin, George ChernishevACL 2021

TL;DR

This paper introduces "situation-based summarization" for multiparticipant chats, a novel approach to condense conversations around specific events rather than just chronological order. They developed a dedicated tool, Chat Corpora Annotator (CCA), and a custom query language (Matcher) to facilitate the creation of specialized datasets for this task, addressing the lack of existing resources.

What Problem Does This Paper Address?

Modern text-based communication, especially in group chats, generates vast amounts of information. While these chats are crucial for professional and personal interactions, navigating them to find specific information or catch up after an absence is incredibly difficult. Existing search functionalities are often basic, and the sheer volume of messages makes manual review time-consuming and inefficient. The authors identify a gap in summarization techniques tailored to the unique characteristics of chat data (short messages, informal language, emoticons) and the need for specialized tools and datasets to address this.

Methods

The core contribution is the concept of situation-based summarization. Instead of a linear summary, they propose identifying and summarizing "situations" – contiguous subsets of messages revolving around a single event (e.g., planning a meeting, solving a technical issue).

To enable this, they developed:

  • Chat Corpora Annotator (CCA): A standalone, user-friendly tool designed for exploring and annotating chat logs. It offers statistical analysis, search capabilities (including N-gram and Concordancer), and importantly, integrates their custom query language. CCA aims to be accessible without requiring coding skills.
  • Matcher Query Language: A SQL-like language for rule-based detection of situations. It allows users to define patterns of messages based on keywords, time, users, and proximity, facilitating both data exploration and the generation of annotation recommendations. This approach is inspired by information retrieval and Boolean retrieval models.

Datasets

The paper introduces a newly created gold-standard dataset for situation-based summarization, derived from the freeCodeCamp datasets. This corpus contains:

  • 967,038 messages spanning 381 days.
  • 236 tagged situations, totaling 4146 messages.
  • An average situation length of 17 messages.
  • A tagset of 6 manually devised tags (JobSearch, CodeHelp, SoftwareSupport, OSSelection, Meeting, FCCBug), reflecting common situations in the freeCodeCamp data.

Key Insights and Ideas

  • Situations over Chronology: The most significant insight is the shift from linear summarization to event-centric "situations." This aligns better with how humans recall and process conversations, focusing on actionable or informative clusters of messages.
  • The Need for Specialized Tools: The paper highlights that existing annotation tools are often too general or require significant customization for chat data. CCA is presented as a tailored solution.
  • Query Language for Annotation: Using a query language like Matcher for annotation recommendations is a novel approach, offering transparency and user control compared to "black box" machine learning models.
  • Data Scarcity is a Bottleneck: The authors emphasize that the lack of suitable datasets is a major hurdle for developing and evaluating chat summarization models. Their work directly addresses this by providing a tool and a dataset.

Key Results

  • CCA Responsiveness: Experiments show CCA is highly responsive, with crucial operations like indexing and querying taking milliseconds to seconds on a large dataset (500MB, ~1M messages).
  • Usability Study: A small usability study with three volunteers indicated that CCA is "easy enough to use" and convenient for reading and searching large chat volumes, achieving a "OK" level of usability (50th percentile on a modified SUS questionnaire). Users suggested improvements like more concise documentation and better feature integration.
  • Dataset Creation Efficiency: A single annotator created the corpus in approximately 20 work hours, largely leveraging the Matcher query language for efficiency.

Limitations and Future Work

  • Single Annotator: The corpus was created by a single annotator, meaning inter-annotator agreement was not assessed. Future work includes implementing features for comparing outputs from multiple annotators and calculating agreement metrics (e.g., Cohen's Kappa).
  • Assistance, Not Automation: The query processor (Matcher) is intended as an annotation assistant, not a fully automated situation detector. The results require manual checking due to the inherent variability and noise in chat data.
  • Future Tool Development: The authors plan to improve CCA's usability, efficiency, and language support.
  • Future Summarization Work: The ultimate goal is to develop actual situation-based summarization models, moving beyond the annotation and exploration phases.

Summary updated

BibTeX
@inproceedings{smirnova2021situation,
  author = {Anna Smirnova and Evgeniy Slobodkin and George Chernishev},
  title = {{Situation-Based Multiparticipant Chat Summarization: a Concept, an Exploration-Annotation Tool and an Example Collection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure