Dual-Attention Graph Convolutional Network

Xueya Zhang, Tong Zhang, Wenting Zhao, Zhen Cui, Jian Yang

Abstract

Graph Convolutional Networks (GCNs) have emerged as powerful tools for text classification by modeling entire corpora as graphs. However, standard GCNs often struggle with the diverse and irregular structures of natural language due to their reliance on static adjacency matrices and uniform treatment of multi-hop features. This paper introduces the Dual-Attention GCN (DAGCN), a model that incorporates two novel attention mechanisms: "connection-attention" to dynamically weigh local word neighborhoods and "hop-attention" to scale the influence of broader, multi-hop contexts. By adaptively capturing both short-term dependencies and long-term semantics, the DAGCN achieves state-of-the-art results across several standard text classification benchmarks.

Introduction

Text classification requires models to synthesize information from both local word sequences and global document semantics. While sequence-based models like LSTMs and local feature extractors like CNNs excel at capturing local context, they often fail to account for global word co-occurrence patterns across a corpus. GCNs address this by representing words and documents as nodes in a massive heterogeneous graph, allowing for the propagation of information across the entire dataset.

Despite their strengths, standard GCNs face a "graph-invariant" problem when applied to text. They typically utilize uniform connection weights defined by a static adjacency matrix and treat features from different diffusion steps (hops) with equal importance. This rigidity is problematic because local word dependencies vary significantly between sentences, and the optimal context size—the distance a word's influence should travel in the graph—depends heavily on the text's length and semantic complexity. The Dual-Attention GCN aims to overcome these limitations by providing a framework that dynamically adapts to both local connections and global context scopes.

Methodology

The DAGCN is built upon a heterogeneous text graph where unique words and documents serve as nodes. Edges between word nodes are established using Pointwise Mutual Information (PMI), while edges between word and document nodes are weighted using Term Frequency-Inverse Document Frequency (TF-IDF).

Architecture of the Dual-Attention Graph Convolutional Network (DAGCN).

The core innovation of the DAGCN lies in its two-stage attention mechanism, which replaces standard graph convolutions:

  1. Connection-Attention: Inspired by Graph Attention Networks (GATs), this mechanism learns to assign unique weights to a node's immediate neighbors. By focusing on the most relevant adjacent words rather than treating all connections equally, the model effectively captures nuanced short-term dependencies.
  2. Hop-Attention: This mechanism controls long-term semantics by scaling features extracted from different hops (distances) within the graph. The authors utilize coefficients inspired by the Chebyshev inequality ($q_k = 1 - \frac{k-1}{c}$) to decay the influence of nodes as their distance from the target node increases. This allows the model to adaptively control its "receptive field" based on the distribution of context.

These mechanisms are integrated into a multi-head dual-attention layer. The resulting node embeddings are then passed through a standard softmax classifier, optimized using cross-entropy loss.

Experimental Evaluation

The model was evaluated on five standard text classification benchmarks: 20-Newsgroups (20NG), Ohsumed, R52, R8, and Movie Review (MR). These datasets represent a mix of long-form documents (20NG, Ohsumed) and shorter texts (R52, R8, MR).

Key Results

The DAGCN demonstrated superior performance, achieving state-of-the-art accuracy on four of the five datasets. It consistently outperformed strong baselines such as TextGCN, fastText, and various CNN/LSTM architectures.

Dataset DAGCN Accuracy TextGCN Accuracy
20NG 87.00% 86.34%
Ohsumed 69.19% 68.36%
R52 93.58% 93.56%
R8 97.36% 97.07%
MR 77.14% 76.74%

On the MR dataset, while DAGCN outperformed TextGCN, it was slightly edged out by pre-trained CNNs (77.75%), likely due to the specific nature of short sentiment-heavy snippets where local sequence features are paramount.

Ablation Study

To validate the contribution of the dual-attention layers, the authors conducted an ablation study. Removing these mechanisms and reverting to standard graph convolutions led to significant performance declines. For instance, accuracy on the Ohsumed dataset dropped from 69.19% to 64.95%, and on 20NG, it fell from 87.00% to 84.46%. These results underscore the critical role of adaptive weighting in processing complex graph-based text representations.

Discussion and Conclusion

A key insight from the study is that the optimal context size (number of hops, $K$) is intrinsically linked to text length. For longer documents like those in the Ohsumed dataset, a larger context ($K=3$) was beneficial. Conversely, for shorter texts like R52, a smaller context ($K=2$) was optimal. This suggests that excessive graph context can introduce noise, a problem the hop-attention mechanism successfully mitigates.

While the DAGCN sets a new bar for graph-based text classification, it faces challenges regarding scalability. The transductive nature of the model requires constructing a massive graph of the entire corpus, which is memory-intensive and complicates online learning or the addition of new documents. Future research may focus on inductive learning approaches to allow for real-time classification without full graph reconstruction and exploring fully learnable decay functions for the hop-attention mechanism.

Summary updated

BibTeX
@inproceedings{zhang2019dual,
  author = {Xueya Zhang and Tong Zhang and Wenting Zhao and Zhen Cui and Jian Yang},
  title = {{Dual-Attention Graph Convolutional Network}},
  booktitle = {ACPR},
  year = {2019},
  doi = {10.1007/978-3-030-41299-9\_19},
}
Figure