Hierarchical Attentional Hybrid Neural Networks for Document Classification

Jader Abreu, Luis Fred, David Macêdo, Cleber Zanchettin

In the world of Natural Language Processing (NLP), document classification is a foundational challenge. Whether it's sorting legal filings, routing customer emails, or analyzing movie reviews, the goal is the same: understand the "vibe" and topic of a large block of text. While deep learning has pushed the boundaries of what's possible, many models still struggle to "read" like humans do—by recognizing that some words matter more than others and that sentences form a logical hierarchy.

A recent paper by Abreu et al., titled "Hierarchical Attentional Hybrid Neural Networks for Document Classification," introduces a new architecture designed to bridge this gap by combining the best of three worlds: Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Attention mechanisms.

TL;DR

The authors propose HAHNN, a hybrid model that uses convolutional layers to extract abstract features and a hierarchical attention mechanism to weigh the importance of specific words and sentences. By mirroring the natural structure of documents, HAHNN outperforms standard attention-based models on sentiment analysis tasks, particularly on the IMDb and Yelp datasets.

What Problem Does This Paper Address?

Traditional text classifiers often treat a document as a flat sequence of words. However, documents are inherently hierarchical: words form sentences, and sentences form documents.

Existing models, like the Hierarchical Attention Network (HAN), successfully use RNNs to capture this structure but often miss out on the rich, local feature extraction that CNNs provide. Conversely, pure CNN models are great at finding local patterns (like "not good") but struggle with long-range dependencies and the relative importance of different sentences. The authors set out to build a "hybrid" that captures both local nuances and global document structure.

Methods

The proposed HAHNN (Hierarchical Attentional Hybrid Neural Network) architecture consists of several specialized layers:

  1. Embedding Layer: Uses FastText to represent words as dense vectors. FastText is chosen because it considers subword information, helping the model handle rare or misspelled words better than standard Word2Vec.
  2. Convolutional Layer: This is the "hybrid" twist. Before the sequence is processed by an RNN, a stack of CNN layers (with varying window sizes) extracts meaningful, abstract features. The authors also experimented with a Temporal Convolutional Network (TCN) variant, which uses "dilated convolutions" to expand the model's receptive field without a massive increase in parameters.
  3. Hierarchical Attention:
    • Word Encoder & Attention: A Bidirectional Gated Recurrent Unit (Bi-GRU) processes the word features, and an attention mechanism assigns weights to the most "informative" words to create a sentence vector.
  4. Sentence Encoder & Attention: Another Bi-GRU processes these sentence vectors, and a second attention layer decides which sentences are most critical to the final classification.
  5. Softmax Output: The final document representation is fed into a softmax layer to produce the class probability.

Datasets

The model was evaluated on two popular benchmarks:

  • IMDb Movie Reviews: A binary classification dataset (positive/negative) containing 50,000 highly polar reviews.
  • Yelp 2018: A multi-class dataset (1–5 stars) for business reviews. The authors used a subset of 500,000 samples for their experiments.

Key Insights and Ideas

  • The Power of the Hybrid: The core "aha" moment is that CNNs and RNNs aren't competitors; they are complementary. CNNs act as excellent "feature hunters" at the word level, while the hierarchical RNN/Attention structure acts as the "architect" that organizes those features into a coherent document summary.
  • Dilated Convolutions for Context: By using TCNs with dilated convolutions, the model can look at a wider range of words simultaneously. This simulates a "long memory" similar to an RNN but with the computational efficiency of a CNN.
  • Visualizing Importance: The attention weights allow us to "peek" inside the black box. The model successfully identifies sentiment-heavy words like "loving" or "disappointment" and gives them higher weight when calculating the final score.

Key Results

The HAHNN model showed clear improvements over existing baselines:

Method Yelp 2018 (Acc %) IMDb (Acc %)
HN-ATT (Baseline) 72.73 89.02
CNN 71.81 91.34
Our model (CNN) 73.28 92.26
Our model (TCN) 72.63 95.17

The most striking result was on the IMDb dataset, where the TCN variant achieved 95.17% accuracy, a significant jump over the 89.02% achieved by the standard Hierarchical Attention Network (HN-ATT).

Limitations and Future Work

While the model is highly effective, the authors noted some "noise" in the attention weights. Occasionally, neutral words (like "translate") were assigned high importance for reasons that weren't immediately clear.

Future work will likely focus on refining these attention mechanisms to better distinguish between truly informative words and neutral ones that happen to appear in specific contexts. Additionally, the authors suggest that further tuning of the TCN parameters could yield even better results for long-form document classification.

Summary updated

Abstract

Document classification is a challenging task with important applications. The deep learning approaches to the problem have gained much attention recently. Despite the progress, the proposed models do not incorporate the knowledge of the document structure in the architecture efficiently and not take into account the contexting importance of words and sentences. In this paper, we propose a new approach based on a combination of convolutional neural networks, gated recurrent units, and attention mechanisms for document classification tasks. We use of convolution layers varying window sizes to extract more meaningful, generalizable and abstract features by the hierarchical representation. The proposed method in improves the results of the current attention-based approaches for document classification.
BibTeX
@inproceedings{abreu2019hierarchical,
  author = {Jader Abreu and Luis Fred and David Macêdo and Cleber Zanchettin},
  title = {{Hierarchical Attentional Hybrid Neural Networks for Document Classification}},
  booktitle = {ICANN (Workshop)},
  year = {2019},
  url = {https://openreview.net/forum?id=veJn_7-JGqW},
}

Loading related content...

Figure