Improving Online Forums Summarization via Unifying Hierarchical Attention Networks with Convolutional Neural Networks

Sansiri Tarnpradab, Fereshteh Jafariakinabad, Kien A. HuaarXiv 2021 · OpenReview

TL;DR

This paper introduces a novel "Hierarchical Unified Deep Neural Network" (HUDNN) for summarizing online forum discussions. By combining Bi-LSTM and CNN with an attention mechanism, HUDNN effectively captures both sentence-level context and thread-level hierarchy, outperforming existing methods in generating concise and informative summaries of lengthy forum threads.

What Problem Does This Paper Address?

Online discussion forums are a rich source of information, but their lengthy and often unstructured nature makes it challenging for users to quickly grasp the main ideas. Traditional summarization methods struggle with the hierarchical and context-dependent nature of forum discussions, where key information can be scattered across multiple posts. This paper aims to solve this by developing an automatic text summarizer specifically designed for online forums, enabling users to efficiently understand the gist of a discussion.

Methods

The core of the proposed solution is the "Hierarchical Unified Deep Neural Network" (HUDNN). This architecture leverages a hierarchical approach to process forum threads:

  • Sentence Encoder: This component first uses a Bidirectional Long Short-Term Memory (Bi-LSTM) network to capture sequential information within a sentence, followed by a Convolutional Neural Network (CNN) to extract salient features. An attention mechanism is applied to these CNN features to highlight important word-level units within a sentence.
  • Thread Encoder: Similar to the sentence encoder, this part uses Bi-LSTM and CNN to process sequences of sentence vectors. It then employs another attention mechanism to weigh sentence pairs, capturing the hierarchical structure of the entire thread.
  • Unified Representation: The final representation for classification is formed by concatenating the sentence-level representation with the thread-level representation. This allows the model to consider both local sentence context and global thread structure.
  • Extractive Summarization: The model is trained to classify sentences as either part of the summary or not. Sentences are then selected based on their predicted saliency scores, with an additional constraint to ensure novelty and avoid redundancy.

The key novelty lies in the unified approach, combining Bi-LSTM and CNN at both sentence and thread levels, and applying attention to the high-level features extracted by CNN, rather than directly to words or sentences as in some prior work.

Datasets

The authors evaluated their model on three datasets:

  • Trip advisor: Forum data crawled from TripAdvisor.
  • Reddit: Forum data crawled from Reddit.
  • Newsroom: A dataset of news articles and their summaries.

The Trip advisor and Reddit datasets represent real-life online forums, while Newsroom serves as a benchmark for comparison. For training, they created labels by identifying sentences that contribute to a summary based on ROUGE scores, aiming for a 20% compression ratio.

Key Insights and Ideas

  • Hierarchy is Crucial: The paper strongly emphasizes that understanding the hierarchical structure of forum discussions (words -> sentences -> posts -> thread) is vital for effective summarization. Simply processing sentences in isolation is insufficient.
  • Bi-LSTM + CNN Synergy: Combining Bi-LSTM for sequential context and CNN for feature extraction at both sentence and thread levels proves more effective than using either alone. This combination leverages the strengths of both architectures.
  • Attention on High-Level Features: Applying the attention mechanism to the features extracted by the CNN layer (rather than raw word or sentence embeddings) leads to improved performance. This suggests that attention is more effective when applied to more abstract, compressed representations.
  • Contextual Embeddings' Mixed Impact: While contextual embeddings like BERT and ELMo show promise, their impact on recall can be mixed, especially on datasets like Reddit. This highlights the ongoing challenge of effectively utilizing these advanced embeddings in summarization tasks.

Key Results

The proposed Hierarchical Unified Deep Neural Network (HUDNN) consistently outperformed several baselines, including traditional machine learning methods (SVM, LogReg), non-hierarchical deep learning models (LSTM, CNN), and other hierarchical models like HAN and HCNN.

  • ROUGE Scores: HUDNN achieved the best ROUGE-L scores across all datasets, indicating its strength in preserving the semantic meaning and order of sentences. For example, on the Reddit dataset, HUDNN achieved an ROUGE-L score of 53.34%, significantly higher than HAN (44.52%) and HCNN (44.61%) when using static embeddings.
  • Sentence-Level Classification: The model also showed competitive performance in sentence-level classification, with F1 scores often exceeding other hierarchical methods.
  • Ablation Studies: Removing components like CNN or Bi-LSTM individually led to a significant drop in performance, underscoring the importance of each part of the HUDNN architecture. The attention mechanism also consistently improved performance across all datasets.

Limitations and Future Work

  • Contextual Embeddings: The paper notes that while contextual embeddings like BERT and ELMo can improve precision, they sometimes lead to a drop in recall for forum datasets. Further research is needed to optimize their use.
  • Dataset Size: The performance gains from contextual embeddings were more noticeable on larger datasets like Reddit and Newsroom, suggesting that model performance might be sensitive to dataset size.
  • Future Work: The authors suggest exploring the effect of different attention mechanisms and further investigating the integration of contextual embeddings for improved summarization. They also highlight the potential for applying this hierarchical approach to other sequence-to-sequence tasks.

Summary updated

Abstract

Online discussion forums are prevalent and easily accessible, thus allowing people to share ideas and opinions by posting messages in the discussion threads. Forum threads that significantly grow in length can become difficult for participants, both newcomers and existing, to grasp main ideas. To mitigate this problem, this study aims to create an automatic text summarizer for online forums. We present Hierarchical Unified Deep Neural Network to build sentence and thread representations for the forum summarization. In this scheme, Bi-LSTM derives a representation that comprises information of the whole sentence and whole thread; whereas, CNN captures most informative features with respect to context from sentence and thread. Attention mechanism is applied on top of CNN to further highlight high-level representations that carry important information contributing to a desirable summary. Extensive performance evaluation has been conducted on three datasets, two of which are real-life online forums and one is news dataset. The results reveal that the proposed model outperforms several competitive baselines.
BibTeX
@misc{tarnpradab2021improving,
  author = {Sansiri Tarnpradab and Fereshteh Jafariakinabad and Kien A. Hua},
  title = {{Improving Online Forums Summarization via Unifying Hierarchical Attention Networks with Convolutional Neural Networks}},
  year = {2021},
  eprint = {2103.13587},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.13587},
}
Figure