Hierarchical Inter-Attention Network for Document Classification with Multi-Task Learning

Bing Tian, Yong Zhang, Jin Wang, Chunxiao Xing

Introduction

Document classification is a fundamental NLP task, but deep learning models often struggle in low-resource settings where labeled data is scarce. While models like the Hierarchical Attention Network (HAN) capture document structure effectively, they rely on intra-attention—looking only within a single document. This limitation prevents them from leveraging informative patterns that might be rare in one domain but common in another.

The MT-HIA Framework

The Multi-Task Hierarchical Inter-Attention Network (MT-HIA) addresses this by sharing attention knowledge across related tasks. The architecture consists of three primary components:

  1. Hierarchical Bi-LSTM: Processes text at both word and sentence levels to preserve document structure.
  2. Global Shared Layer: A shared LSTM that captures task-invariant semantic features across all datasets.
  3. Inter-Attention Mechanism: Instead of task-specific attention, a shared MLP informed by the global layer determines importance, allowing the model to "import" knowledge from other tasks.
The MT-HIA architecture featuring task-specific encoders and a global shared layer.

A gating mechanism is also employed to regulate information flow, ensuring that task-specific nuances are not drowned out by global data.

Key Insights and Results

The core philosophy of MT-HIA is that rare patterns in one task are often abundant in others. By "looking outside" the document via inter-attention, the model builds more robust representations.

In evaluations across 15 sentiment classification datasets (including 14 Amazon domains and IMDB), MT-HIA achieved:

  • State-of-the-Art Accuracy: An average of 89.9%, outperforming single-task models like HAN by 4.3%.
  • Robustness: Superior performance over other multi-task baselines such as DA-MTL and CG-MTL.

Qualitative Analysis

The benefits of inter-attention are most visible in complex linguistic structures. For instance, in a review containing the phrase "although she was happy with it...", standard intra-attention models often focus on the word "happy" and misclassify the sentiment as positive. MT-HIA, having learned from other tasks that "although" typically signals a contrast followed by the true (negative) sentiment, correctly identifies the document's negative tone.

Comparison of attention weights between Multi-task Driven Inter Attention and standard Intra Attention.

Conclusion

MT-HIA demonstrates that maintaining hierarchical structure and sharing attention mechanisms across tasks significantly improves document classification. While future work may explore broader task diversity and computational scaling, the current results provide a strong foundation for cross-domain knowledge sharing in NLP.

Summary updated

Abstract

Document classification is an essential task in many real world applications. Existing approaches adopt both text semantics and document structure to obtain the document representation. However, these models usually require a large collection of annotated training instances, which are not always feasible, especially in low-resource settings. In this paper, we propose a multi-task learning framework to jointly train multiple related document classification tasks. We devise a hierarchical architecture to make use of the shared knowledge from all tasks to enhance the document representation of each task. We further propose an inter-attention approach to improve the task-specific modeling of documents with global information. Experimental results on 15 public datasets demonstrate the benefits of our proposed model.
BibTeX
@inproceedings{tian2019hierarchical,
  author = {Bing Tian and Yong Zhang and Jin Wang and Chunxiao Xing},
  title = {{Hierarchical Inter-Attention Network for Document Classification with Multi-Task Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/495},
}
Figure