Abstract
Document classification is a challenging task with important applications. The deep learning approaches to the problem have gained much attention recently. Despite the progress, the proposed models do not incorporate the knowledge of the document structure in the architecture efficiently and not take into account the contexting importance of words and sentences. In this paper, we propose a new approach based on a combination of convolutional neural networks, gated recurrent units, and attention mechanisms for document classification tasks. The main contribution of this work is the use of convolution layers to extract more meaningful, generalizable and abstract features by the hierarchical representation. The proposed method in this paper improves the results of the current attention-based approaches for document classification.
Introduction
In the world of Natural Language Processing (NLP), document classification is a foundational challenge. Whether it's sorting legal filings, routing customer emails, or analyzing movie reviews, the goal is to understand the "vibe" and topic of a large block of text. While deep learning has pushed the boundaries of what's possible, many models still struggle to "read" like humans do—by recognizing that some words matter more than others and that sentences form a logical hierarchy.
Traditional text classifiers often treat a document as a flat sequence of words. However, documents are inherently hierarchical: words form sentences, and sentences form documents. Existing models, like the Hierarchical Attention Network (HAN), successfully use RNNs to capture this structure but often miss out on the rich, local feature extraction that CNNs provide. The authors of "Hierarchical Attentional Hybrid Neural Networks for Document Classification" introduce a new architecture, HAHNN, to bridge this gap by combining the best of three worlds: Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and Attention mechanisms.
Methodology: The HAHNN Architecture
The proposed HAHNN (Hierarchical Attentional Hybrid Neural Network) architecture consists of several specialized layers designed to capture both local nuances and global document structure.
- Embedding Layer: Uses FastText to represent words as dense vectors. FastText is chosen because it considers subword information, helping the model handle rare or misspelled words better than standard Word2Vec.
- Convolutional Layer: This is the "hybrid" twist. Before the sequence is processed by an RNN, a stack of CNN layers (or Temporal Convolutional Networks - TCN) extracts meaningful, abstract features. TCNs use dilated convolutions to expand the model's receptive field without a massive increase in parameters.
- Hierarchical Attention:
- Word Encoder & Attention: A Bidirectional Gated Recurrent Unit (Bi-GRU) processes the word features, and an attention mechanism assigns weights to the most "informative" words to create a sentence vector.
- Sentence Encoder & Attention: Another Bi-GRU processes these sentence vectors, and a second attention layer decides which sentences are most critical to the final classification.
- Softmax Output: The final document representation is fed into a softmax layer to produce the class probability.
Experimental Results
The model was evaluated on two popular benchmarks: IMDb Movie Reviews (binary classification) and Yelp 2018 (multi-class sentiment analysis). The HAHNN model showed clear improvements over existing baselines:
| Method | Yelp 2018 (Acc %) | IMDb (Acc %) |
|---|---|---|
| HN-ATT (Baseline) | 72.73 | 89.02 |
| CNN | 71.81 | 91.34 |
| Our model (CNN) | 73.28 | 92.26 |
| Our model (TCN) | 72.63 | 95.17 |
The most striking result was on the IMDb dataset, where the TCN variant achieved 95.17% accuracy, a significant jump over the 89.02% achieved by the standard Hierarchical Attention Network (HN-ATT).
Interpretability and Visualization
A key advantage of the hierarchical attention mechanism is its interpretability. By visualizing the attention weights, we can "peek" inside the black box to see how the model identifies sentiment-heavy words and sentences.
The model successfully identifies words like "loving" or "disappointment" and gives them higher weight when calculating the final score. This confirms the model is learning meaningful linguistic patterns rather than just statistical noise.
Conclusion
The HAHNN model demonstrates that CNNs and RNNs are complementary: CNNs act as excellent "feature hunters" at the word level, while the hierarchical RNN/Attention structure acts as the "architect" that organizes those features into a coherent document summary. While some noise remains in the attention weights—where neutral words occasionally receive high importance—future work on refining these mechanisms and further tuning TCN parameters promises even greater accuracy for long-form document classification.
Summary updated