Topic Segmentation and Labeling in Asynchronous Conversations

Shafiq Rayhan Joty, Giuseppe Carenini, Raymond T. NgJAIR 2013 · DOI · OpenReview

TL;DR

This paper tackles the challenge of automatically identifying and labeling topics within asynchronous conversations like emails and blogs. They introduce new annotated datasets and a comprehensive framework that leverages conversational structure, outperforming existing methods and achieving results highly correlated with human annotations.

What Problem Does This Paper Address?

Asynchronous conversations, such as emails and blog comments, are ubiquitous in modern communication. However, their inherent characteristics—interleaved topics, lack of strict sequentiality, and informal language—make traditional topic segmentation and labeling techniques, developed for more structured text like news articles or synchronous dialogues, perform poorly. This paper addresses the need for robust methods to automatically understand the thematic structure of these conversations, which is crucial for applications like summarization, information retrieval, and understanding group dynamics.

Methods

The authors propose a complete framework for topic segmentation and labeling in asynchronous conversations.

Topic Segmentation: * Unsupervised Models: They extend existing methods like LCSeg (Lexical Cohesion-based Segmenter) and LDA (Latent Dirichlet Allocation) by incorporating a "fine-grained conversational structure" derived from a Fragment Quotation Graph (FQG). This FQG captures reply-to relationships and quotation structures, providing richer context than simple temporal ordering. * Supervised Model: A novel graph-theoretic supervised model is introduced, combining lexical, conversational (e.g., speaker, gap), and topic features. This model uses a sentence-pair classifier and graph partitioning to identify topic boundaries.

Topic Labeling: * Unsupervised Models: Two novel random walk models are proposed. These models leverage conversation-specific clues: 1. Leading Sentences: The idea is that the initial sentences of a topic segment often contain the most informative clues for its label. 2. Fine-grained Conversational Structure: The FQG is also used to inform the ranking of words for label generation. * Phrase Generation: They extract keyphrases from ranked words and re-rank them considering the entire conversation to capture relevant terms that might appear outside the immediate segment. * Redundancy Checking: A Maximum Marginal Relevance (MMR) criterion is used to select diverse and non-redundant labels.

Datasets

  • BC3 email corpus: Previously available, contains 40 email conversations.
  • New Blog corpus: A newly introduced dataset of 20 conversations from Slashdot.
  • Both corpora were manually annotated for topics by three annotators.

Key Insights and Ideas

  • Asynchronous conversations are different: The interleaved nature of topics in asynchronous communication fundamentally breaks the sequentiality assumption of traditional topic segmentation models, necessitating new approaches.
  • Conversational structure is key: Leveraging the fine-grained structure (like quotation and reply-to relationships captured in the FQG) significantly improves both segmentation and labeling performance.
  • Leading sentences are informative: The initial sentences of a topic segment are disproportionately important for understanding and labeling the topic.
  • Domain-specific features matter: Features like speaker and reply-to relationships, which are specific to conversational contexts, are crucial for accurate segmentation.
  • Abstractive labels are challenging: Human annotators often generate abstractive labels that go beyond simple keyphrases, highlighting the difficulty in replicating human-like labeling.

Key Results

  • Segmentation:
    • The supervised segmentation model achieved the best performance, outperforming unsupervised models even with limited labeled data.
  • LCSeg+FQG was the best unsupervised model, showing significant improvements over LCSeg by incorporating conversational structure.
  • The supervised model's segmentation decisions were highly correlated with human annotations.
  • Labeling:
    • The proposed biased random walk models (BiasRW and BiasRW+) significantly outperformed baselines and existing methods like TextRank.
  • Incorporating conversational structure (CorGen, CorBias) further improved performance, especially on blogs.
  • CorBias+ (combining biased random walk with conversation structure and conversation-level phrases) achieved the best labeling performance.
  • End-to-End System: The combined system showed promising results, with performance getting closer to human agreement as the number of output labels (k) increased.

Limitations and Future Work

  • Limitations:
    • The performance of LDA models was limited, possibly due to the bag-of-words assumption and insufficient data for training.
  • Human-authored labels are often abstractive, making it difficult for extractive methods to fully match them.
  • The FQG construction can be less accurate when quotation cues are absent.
  • Future Work:
    • Develop more abstractive labeling frameworks that can generate human-like labels by synthesizing information.
  • Explore more sophisticated information extraction, semantic understanding, and data-to-text generation techniques.
  • Conduct more extrinsic evaluations by using the methods to support other NLP tasks (e.g., summarization, visualization).
  • Adapt the approach to other similar domains.
  • Extend the work to both synchronous and asynchronous domains.

Summary updated

Abstract

Topic segmentation and labeling is often considered a prerequisite for higher-level conversation analysis and has been shown to be useful in many Natural Language Processing (NLP) applications. We present two new corpora of email and blog conversations annotated with topics, and evaluate annotator reliability for the segmentation and labeling tasks in these asynchronous conversations. We propose a complete computational framework for topic segmentation and labeling in asynchronous conversations. Our approach extends state-of-the-art methods by considering a fine-grained structure of an asynchronous conversation, along with other conversational features by applying recent graph-based methods for NLP. For topic segmentation, we propose two novel unsupervised models that exploit the fine-grained conversational structure, and a novel graph-theoretic supervised model that combines lexical, conversational and topic features. For topic labeling, we propose two novel (unsupervised) random walk models that respectively capture conversation specific clues from two different sources: the leading sentences and the fine-grained conversational structure. Empirical evaluation shows that the segmentation and the labeling performed by our best models beat the state-of-the-art, and are highly correlated with human annotations.
BibTeX
@article{joty2013topic,
  author = {Shafiq Rayhan Joty and Giuseppe Carenini and Raymond T. Ng},
  title = {{Topic Segmentation and Labeling in Asynchronous Conversations}},
  journal = {J. Artif. Intell. Res.},
  year = {2013},
  doi = {10.1613/jair.3940},
}
Figure