Energy-based Self-attentive Learning of Abstractive Communities for Spoken Language Understanding

Guokan Shang, Antoine Jean-Pierre Tixier, Michalis Vazirgiannis, Jean-Pierre Lorré

Summarizing spoken conversations, such as business meetings or customer service calls, presents a unique set of challenges compared to summarizing structured text like news articles. Spoken language is often fragmented, interrupted, and non-linear, with multiple topics discussed simultaneously. This paper introduces a novel approach to Abstractive Community Detection (ACD), a critical sub-task designed to group related utterances into "communities" that can be summarized by a single abstractive sentence.

The Challenge of Spoken Dialogue

Traditional extractive summarization methods, which select the most important sentences from a text, often fail in conversational settings. Individual utterances like "Yeah, that works" or "I agree" are meaningless without their surrounding context. The goal of ACD is to identify specific utterances that together support a summary point, even if they are separated by time or other topics.

Example of utterances linked to abstractive summary points in a meeting corpus.

As shown in the figure above, different participants may contribute to various summary points (e.g., decisions about casing material) throughout a meeting. ACD aims to link these disparate fragments into coherent clusters.

Methodology: Hierarchical Self-Attention and Energy-Based Learning

The authors propose a sophisticated neural utterance encoder trained within an Energy-Based Modeling (EBM) framework to address the contextual and messy nature of human speech.

1. Hierarchical Utterance Encoder

The core of the model is a hierarchical encoder that processes utterances not in isolation, but within their "neighborhood." It employs three distinct self-attention mechanisms:

  • Word Encoder (WE): Combines word embeddings with discourse features such as the speaker's role and the type of speech act (e.g., question vs. statement).
  • Content-Aware Self-Attention: Analyzes a window of surrounding context (approximately 11 utterances before and after) to identify complementary information that clarifies the current utterance's meaning.
  • Time-Aware Self-Attention: Incorporates a trainable decay mechanism to weight context based on temporal proximity, recognizing that closer utterances are generally more relevant.
  • Utterance-Level Self-Attention: Produces a final, compact vector representation (embedding) of the utterance.
The proposed hierarchical self-attentive utterance encoder architecture.

2. Energy-Based Learning Framework

Rather than using a standard classifier, the model is trained using Siamese and Triplet architectures. The Triplet Network approach proved particularly effective. By training on a trio of utterances (Anchor, Positive, and Negative), the model learns to minimize the "energy" (distance) between related utterances while maximizing the distance to unrelated ones. This allows the model to organize the embedding space based on relative semantic relationships rather than absolute constraints.

3. Overlapping Clustering

Once utterances are projected into the embedding space, the authors apply Fuzzy c-Means (FCM) clustering. This choice is vital because it allows for overlapping communities, reflecting how a single utterance can contribute to multiple summary points.

Key Insights and Results

  • Context is Crucial: The model's performance improved steadily as the context window increased, peaking at around 11 utterances. This confirms that meeting utterances are often "ill-formed" and require significant history and future context to resolve.
  • Triplet > Siamese: The Triplet architecture provided more robustness and flexibility in organizing the embedding space compared to Siamese networks.
  • Superior Performance: The proposed model significantly outperformed baselines, including Hierarchical Attention Networks (HAN) and traditional TF-IDF approaches, particularly on the Omega Index, which measures the quality of overlapping clusters.

Conclusion and Future Work

This research demonstrates that energy-based self-attentive learning can effectively capture the complex, overlapping structure of human dialogue. While the current model excels at grouping utterances, future work remains to bridge the "extraction gap"—integrating the initial selection of summary-worthy utterances—and improving robustness against noise from Automatic Speech Recognition (ASR) systems.

Summary updated

BibTeX
@inproceedings{shang2020energy,
  author = {Guokan Shang and Antoine Jean-Pierre Tixier and Michalis Vazirgiannis and Jean-Pierre Lorré},
  title = {{Energy-based Self-attentive Learning of Abstractive Communities for Spoken Language Understanding}},
  booktitle = {AACL/IJCNLP},
  year = {2020},
  doi = {10.18653/v1/2020.aacl-main.34},
}
Figure