Automatic Rephrasing of Transcripts-based Action Items

Amir Dn Cohen, Amir Kantor, Sagi Hilleli, Eyal KolmanACL 2021

TL;DR

This paper tackles the challenge of automatically rephrasing action items (AIs) from meeting transcripts into clear, self-contained statements. They introduce a new dataset, AIR, and demonstrate that while standard models struggle with the nuances of spoken language, incorporating contextual "hints" significantly improves rephrasing quality, approaching human-level performance.

What Problem Does This Paper Address?

In today's work environment, especially with the rise of remote work, online meetings are ubiquitous. These meetings often generate valuable information, including action items – commitments to perform tasks. While automatic speech recognition (ASR) has improved, making meeting transcripts readily available, the NLP community has paid less attention to processing this spoken language compared to written text. A key challenge is that spoken language is often fragmented, informal, and relies heavily on implicit context, making it difficult for machines to extract and understand action items accurately. This paper focuses on the specific problem of taking a raw action item from a meeting transcript and rephrasing it into a coherent, self-contained statement that is easy to understand, even without the full meeting context.

Methods

The core of the paper involves fine-tuning pre-trained transformer models, specifically BART, for the task of action item rephrasing. The authors propose two key innovations to improve upon a baseline BART model:

  • Coreference Resolution (CR) Hints: They leverage pre-trained coreference resolution models to identify mentions of the same entity across the transcript. By adding "hints" that mark these coreferent spans, they help the rephrasing model understand relationships between different parts of the conversation.
  • Context Detection Model Hints: They train a separate model to explicitly identify spans of text in the transcript that are relevant to the action item. These "context hints" are then fed into the rephrasing model, guiding it to focus on the most pertinent information.

The input to the rephrasing model typically includes the action item sentence itself, along with the preceding seven and following two sentences from the transcript, along with speaker information.

Datasets

The authors introduce a novel dataset called AIR (Action Item Rephrasing). This dataset is compiled from various sources of manually transcribed meeting records, including:

  • ICSI Meeting Corpus: Transcripts from conference room meetings.
  • Augmented Multi-party Interaction (AMI) Meeting Corpus: Multimodal data with manually annotated transcripts.
  • Board Meetings (LSC): Publicly available transcripts of legal services corporation board meetings.
  • Internal Dataset (ID): Manually transcribed meetings primarily focused on software development.

The AIR dataset contains action items extracted from these meetings, along with human-annotated rephrasings. For evaluation, they also used a subset of the Avocado dataset (used in prior work on email AI rephrasing) to compare performance on emails versus transcripts.

Key Insights and Ideas

  • Spoken Language is Fundamentally Different: The paper strongly emphasizes that spoken language, with its inherent informality, fragmentation, and reliance on implicit context, poses a significantly harder challenge for NLP tasks than written text.
  • Context is King (Especially for AIs): Action items are often underspecified without their surrounding conversational context. Simply rephrasing the AI sentence in isolation is insufficient.
  • Hints are Crucial for Contextual Understanding: Standard models struggle to identify relevant context. Providing explicit "hints" derived from coreference resolution and context detection models dramatically improves the model's ability to leverage this context.
  • Human Agreement is Challenging: The authors found surprisingly low agreement among human annotators for context detection, highlighting the inherent ambiguity and subjectivity in interpreting spoken conversations. This underscores the difficulty of the task.
  • N-gram Metrics Can Be Misleading: The paper points out the limitations of traditional metrics like ROUGE for evaluating text generation, especially when dealing with paraphrasing. Human evaluation is deemed more reliable.

Key Results

  • Baseline Performance: The base BART model, even when trained on the AIR dataset, performed worse on transcripts than on email data, highlighting the difficulty of spoken language.
  • Hint-Based Improvement: Incorporating context hints (from the context detection model) led to near-human-level performance on a constrained subset of the problem.
  • Coreference Hints Also Beneficial: Coreference resolution hints also improved performance, though generally to a lesser extent than context detection hints.
  • Human Performance Benchmark: Human rephrasing consistently achieved the highest scores, but the best model (with context hints) came remarkably close, achieving statistically significant improvements over the base model.
  • Email vs. Transcript Differences: The paper demonstrates that models trained for email AI rephrasing are less effective on meeting transcripts, primarily due to the lack of a "subject" equivalent and the greater number of utterances providing context.

Limitations and Future Work

  • Low Annotator Agreement: The authors acknowledge the low agreement among annotators for context detection, which poses challenges for training robust models.
  • Dataset Size Dependency: Performance, particularly for the context model, was affected by dataset size when moving from the internal dataset to the smaller public AIR dataset.
  • Constrained Subset Performance: While near-human performance was achieved, it was on a "constrained subset of the problem," implying that the task's full complexity might still be a challenge.
  • Future Work: The authors suggest that their approach of using "hints" to guide models in leveraging context could be applicable to other NLP tasks dealing with noisy or context-dependent data. Further exploration into more sophisticated ways to integrate contextual information and handle the inherent ambiguity of spoken language is also implied.

Summary updated

BibTeX
@inproceedings{cohen2021automatic,
  author = {Amir DN Cohen and Amir Kantor and Sagi Hilleli and Eyal Kolman},
  title = {{Automatic Rephrasing of Transcripts-based Action Items}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure