Hierarchical models vs. transfer learning for document-level sentiment classification

Jeremy Barnes, Vinit Ravishankar, Lilja Øvrelid, Erik VelldalarXiv 2020 · arXiv

Introduction

Sentiment analysis at the document level presents a unique challenge: the relevant emotional signal is often dispersed across multiple paragraphs and sentences. A single negative sentence might not shift the overall positive tone of a long review, requiring models to understand context and structure. This paper provides a systematic head-to-head comparison between two dominant paradigms: Hierarchical Modeling, which mirrors document structure (e.g., Hierarchical Attention Networks or HAN), and Transfer Learning, which leverages massive pre-training (e.g., BERT or ULMFit).

Methodology

The researchers evaluated seven models across three categories to determine their effectiveness under varying conditions such as document length and data availability:

  • Flat Models: Bag-of-Words (SVM), standard CNN, and an Attention Network (AN) using a BiGRU with dot-product attention.
  • Hierarchical Models: Hierarchical CNN (HCNN) and the Hierarchical Attention Network (HAN).
  • Transfer Learning: ULMFit (AWD-LSTM based) and Multilingual BERT (mBERT).

To assess the importance of document structure, the authors also conducted experiments where the sentence order of test data was shuffled, testing how much each architecture relies on the natural flow of text.

Experimental Setup

The study utilized sentiment datasets across five languages to ensure linguistic diversity:

  • Amazon Customer Reviews: English (En), French (Fr), German (De), and Japanese (Ja), featuring 5-class classification.
  • NoReC (Norwegian Review Corpus): A 6-class classification task.

Datasets were standardized by sampling 50,000 documents per language (where possible) to maintain a fair comparison.

Results and Discussion

Overall Performance

The Hierarchical Attention Network (HAN) emerged as the most robust model overall, achieving the highest average accuracy across all languages. While mBERT showed strength in English, French, and German, HAN's ability to model document structure gave it a significant edge in Norwegian and Japanese.

Table 1: Accuracy of various models across five languages, including the impact of sentence shuffling.

The Impact of Document Length

A key finding of this research is that hierarchical models act as a buffer for document length. While transfer learning models like mBERT excel on shorter snippets, their performance tends to degrade as documents exceed 10 sentences. In contrast, HAN remains robust, consistently outperforming mBERT on longer texts. There is a clear positive correlation (0.41) between document length and the performance advantage of hierarchical modeling.

Figure 1: Accuracy vs. Document size (number of sentences) comparing HAN and mBERT across different languages.

Low-Resource and Order Sensitivity

Contrary to the common assumption that transfer learning is mandatory for low-resource settings, hierarchical models performed competitively even with as few as 64 training examples. Furthermore, the shuffling experiments revealed that flat CNNs are the most sensitive to word/sentence order, losing approximately 8.3 percentage points in accuracy when structure was disrupted, whereas hierarchical models were significantly more resilient.

Conclusion

While transfer learning has revolutionized NLP, this study demonstrates that hierarchical architectures remain highly relevant for document-level tasks. HAN's superior performance on long documents and its robustness in low-resource scenarios suggest that explicitly modeling document structure provides benefits that pre-training alone cannot yet replicate. Future work may involve integrating these paradigms, such as using BERT as a base encoder within a hierarchical structure.

Summary updated

Abstract

Documents are composed of smaller pieces - paragraphs, sentences, and tokens - that have complex relationships between one another. Sentiment classification models that take into account the structure inherent in these documents have a theoretical advantage over those that do not. At the same time, transfer learning models based on language model pretraining have shown promise for document classification. However, these two paradigms have not been systematically compared and it is not clear under which circumstances one approach is better than the other. In this work we empirically compare hierarchical models and transfer learning for document-level sentiment classification. We show that non-trivial hierarchical models outperform previous baselines and transfer learning on document-level sentiment classification in five languages.
BibTeX
@misc{barnes2020hierarchical,
  author = {Jeremy Barnes and Vinit Ravishankar and Lilja Øvrelid and Erik Velldal},
  title = {{Hierarchical models vs. transfer learning for document-level sentiment classification}},
  year = {2020},
  eprint = {2002.08131},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2002.08131},
}
Figure