Abstract
Graph neural networks have shown significant success in the field of graph representation learning. Graph convolutions perform neighborhood aggregation and represent one of the most important graph operations. Nevertheless, one layer of these neighborhood aggregation methods only consider immediate neighbors, and the performance decreases when going deeper to enable larger receptive fields. Several recent studies attribute this performance deterioration to the over-smoothing issue, which states that repeated propagation makes node representations of different classes indistinguishable. In this work, we study this observation systematically and develop new insights towards deeper graph neural networks. First, we provide a systematical analysis on this issue and argue that the key factor compromising the performance significantly is the entanglement of representation transformation and propagation in current graph convolution operations. After decoupling these two operations, deeper graph neural networks can be used to learn graph node representations from larger receptive fields. We further provide a theoretical analysis of the above observation when building very deep models, which can serve as a rigorous and gentle description of the over-smoothing issue. Based on our theoretical and empirical analysis, we propose Deep Adaptive Graph Neural Network (DAGNN) to adaptively incorporate information from large receptive fields. A set of experiments on citation, co-authorship, and co-purchase datasets have confirmed our analysis and insights and demonstrated the superiority of our proposed methods.
Introduction
In the world of deep learning, "deeper is better" is a common mantra. However, Graph Convolutional Networks (GCNs) have historically been the exception. When researchers try to stack more than a few layers, performance usually falls off a cliff. The community has largely blamed this on over-smoothing: the idea that as you aggregate information from neighbors over many iterations, every node’s representation eventually converges to the same value, making them indistinguishable.
This paper argues that while over-smoothing is a real theoretical limit, the immediate cause of failure in shallow-to-medium depths is actually the entanglement of representation transformation (the weights and non-linearities) and propagation (the graph-based averaging). Because each layer in a standard GCN couples these two, adding layers exponentially increases the number of parameters and makes the model incredibly difficult to train.
The DAGNN Architecture
The authors propose a two-pronged solution embodied in the Deep Adaptive Graph Neural Network (DAGNN):
- Decoupling Transformation and Propagation: Instead of interleaving weight matrices with graph aggregations, DAGNN first applies a multi-layer perceptron (MLP) to the node features to produce a "base" representation ($Z$). Then, it performs $k$ steps of graph propagation on $Z$ without any additional trainable weights or non-linearities in the propagation steps. This allows the model to reach a massive receptive field (e.g., 100+ hops) without a parameter explosion.
- Adaptive Adjustment Mechanism: Not every node needs information from 100 hops away; some are better defined by their immediate neighbors. DAGNN learns a "retainment score" for each hop, calculating a weighted sum of the representations from all hops (from 0 to $k$). This allows the model to adaptively balance local and global information for every individual node.
Key Insights and Ideas
- The Entanglement Trap: GCNs fail early not just because of the graph topology, but because we are trying to learn complex transformations at the same time as we are diffusing information. By doing the "thinking" (MLP) first and the "diffusing" (propagation) second, we can go much deeper.
- Over-smoothing is a Choice: Visualizations show that even as depth increases, decoupled models can produce distinct, clusterable representations, whereas standard GCNs quickly become an indistinguishable "blob."
- Receptive Fields Matter for Small Data: Deeper models are significantly more robust when training labels are scarce. If you only have one labeled node per class, you need a large receptive field to propagate that signal across the graph.
Experimental Results
The researchers evaluated DAGNN on seven widely used datasets across citation networks (Cora, CiteSeer, PubMed), co-authorship graphs, and co-purchase graphs. Key findings include:
- Superior Accuracy: DAGNN outperformed state-of-the-art baselines (like GCN, GAT, APPNP, and SGC) across all seven datasets.
- Low-Label Performance: On the Cora dataset with only one training node per class, DAGNN improved accuracy over GCN by a staggering 23.7%.
- Depth Stability: While GCN performance plummeted after 3 layers, DAGNN remained stable or even improved as the number of hops increased to 100 or 200.
- Efficiency: Because the propagation steps have no trainable parameters, DAGNN is more parameter-efficient than stacking standard GCN layers.
Conclusion
By decoupling representation transformation from propagation, DAGNN overcomes the "3-layer ceiling" that has long limited Graph Neural Networks. While graph topology still dictates ultimate limits—especially in densely connected graphs where information spreads rapidly—this approach provides a robust framework for leveraging large receptive fields in graph representation learning.
Summary updated