Bidirectional Context-Aware Hierarchical Attention Network for Document Understanding

Jean-Baptiste Remy, Antoine Jean-Pierre Tixier, Michalis VazirgiannisarXiv 2019 · arXiv

TL;DR

Standard Hierarchical Attention Networks (HANs) encode sentences in isolation, often leading to redundant focus on repetitive features. This paper introduces CAHAN (Context-Aware HAN), which allows the sentence encoder to incorporate surrounding context via a context vector, resulting in more diverse and accurate document representations.

The Problem: Attentional Redundancy

In natural language processing, documents are typically processed hierarchically: words form sentences, and sentences form documents. The classic Hierarchical Attention Network (HAN) mirrors this structure, using word-level attention to build sentence vectors, which are then aggregated by a document-level encoder.

However, a significant "blind spot" exists: at the first level, every sentence is encoded in complete isolation. In a review where multiple sentences repeat the same sentiment (e.g., "Terrible value"), a standard HAN might exhaust its attentional budget on those redundant words in every sentence. Because the encoder is unaware that it has already captured the negative sentiment, it fails to highlight the specific details—like "cold food" or "rude staff"—that provide additional information.

Methodology: The CAHAN Architecture

The authors propose the Context-Aware Hierarchical Attention Network (CAHAN). The core innovation is the injection of a context vector ($c_i$) into the word-level attention mechanism. Instead of calculating word importance based solely on the current sentence, the model uses the context vector to guide its decisions. If the context already contains certain information, the model can shift its attention to new, complementary details.

The CAHAN architecture integrates context-aware self-attention to inform sentence encoding with document-level information.

Key Technical Components:

  • Two Context Strategies: The authors explored CAHAN-SUM (where context is the sum or centroid of surrounding sentence representations) and CAHAN-RNN (where context is derived from the hidden states of a document-level RNN).
  • Bidirectional Encoding: To provide context from both the past and the future, a bidirectional document encoder is used. This ensures that when encoding sentence i, the model is aware of the content in sentences $1$ to $i-1$ and $i+1$ to $N$.
  • Gated Context: A trainable "gate" ($\lambda$) allows the model to dynamically weight the context against the local sentence content, preventing context from overwhelming specific, high-impact details in shorter sentences.

Key Insights and Results

The models were evaluated on three massive benchmarks: Amazon (3.6M reviews), Yelp (610k reviews), and Yahoo! Answers (1.46M entries).

Performance and Qualitative Improvements

The bidirectional CAHAN-SUM (with gating) consistently outperformed the baseline HAN across all datasets. For instance, on the Yelp dataset, it achieved 67.02% accuracy compared to the baseline's 66.55%.

More importantly, visualizations confirm that CAHAN successfully implements a "coverage" mechanism. While the baseline HAN gets "stuck" on repeated phrases, CAHAN shifts its attention to secondary, informative topics once the primary sentiment is established.

Comparison of attention weights: HAN (top) focuses repeatedly on "terrible," while CAHAN (bottom) shifts focus to specific details like "appetizer" and "defrosted" as the document progresses.

Observations on Context Strategies

  • Centroids vs. Sums: In unidirectional models, using a "centroid" (average) of previous sentences proved more effective than raw sums, which can grow too large and "blind" the attention mechanism in long documents.
  • The Failure of RNN Context: Surprisingly, CAHAN-RNN performed poorly. The authors suggest that the "forgetting" bias inherent in RNNs is counterproductive for document-level logic, where sentences do not necessarily follow a strict temporal decay.
  • Gating is Essential: Gating provided the most significant boosts in sentiment analysis, suggesting that sentiment is often concentrated in specific "bursts" that require careful regulation.

Limitations and Future Work

While CAHAN provides richer representations, the performance margins on simple texts are modest, as context-blindness is less detrimental in short documents. Additionally, making sentence encoding dependent on surrounding context introduces a sequential bottleneck, reducing the parallelization benefits of the original HAN. Future work will explore applying CAHAN to abstractive summarization, where non-redundant document understanding is critical.

Summary updated

BibTeX
@misc{remy2019bidirectional,
  author = {Jean-Baptiste Remy and Antoine Jean-Pierre Tixier and Michalis Vazirgiannis},
  title = {{Bidirectional Context-Aware Hierarchical Attention Network for Document Understanding}},
  year = {2019},
  eprint = {1908.06006},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.06006},
}
Figure