Can GCNs Go as Deep as CNNs?

Guohao Li, Matthias Müller, Ali K. Thabet, Bernard Ghanem

Introduction

In the domain of computer vision, the success of deep learning has largely been driven by the ability to scale Convolutional Neural Networks (CNNs) to hundreds of layers. However, Graph Convolutional Networks (GCNs), which operate on non-Euclidean data like 3D point clouds or social networks, have historically been restricted to very shallow architectures—typically just 3 to 4 layers deep. This limitation stems from two primary issues: vanishing gradients and the phenomenon of "over-smoothing."

This paper introduces DeepGCNs, a framework that successfully adapts architectural innovations from the image domain—specifically residual/dense connections and dilated convolutions—to graph-structured data. By doing so, the authors demonstrate that GCNs can indeed go deep, achieving state-of-the-art performance in 3D point cloud segmentation.

The Challenge: Why GCNs Struggle with Depth

When stacking multiple GCN layers, researchers encounter two significant hurdles:

  1. Vanishing Gradients: As in early CNNs, the gradient signal used for training tends to dissipate as it back-propagates through many layers, making it difficult for the network to learn.
  2. Over-smoothing: This is a unique challenge for graphs. As information is aggregated from neighbors over successive layers, the features of all nodes eventually converge toward a global average. This leads to a loss of discriminative power, where every node in the graph starts to look identical, rendering the network ineffective for tasks like semantic segmentation.
Training loss comparison between GCNs with and without residual connections across different depths.

As shown in the figure above, while shallow GCNs might start with lower loss, they fail to improve as depth increases. In contrast, DeepGCNs with residual connections maintain stability and continue to minimize loss even at significant depths.

Proposed Methods: The DeepGCN Architecture

To overcome these limitations, the authors "borrow" three key architectural pillars from successful deep CNNs:

1. Residual and Dense Connections

The authors implement vertex-wise addition (ResNet-style) and concatenation (DenseNet-style) between GCN layers. These skip connections allow the network to preserve original node features and ensure a smoother gradient flow, directly addressing both over-smoothing and vanishing gradients.

2. Dilated Graph Convolutions

To increase the "receptive field" of a node—the amount of surrounding graph information it can perceive—without a massive increase in parameters, the authors introduce Dilated k-NN. Similar to dilated convolutions in images, this method samples neighbors at specific intervals rather than just the immediate $k$ nearest neighbors.

3. Stochastic Dilation

To improve generalization and act as a form of regularization, the network employs stochastic dilation during training. This involves randomly sampling different neighbors within the dilated range, preventing the model from over-fitting to specific local structures.

Overview of the DeepGCN architecture, showcasing the GCN Backbone (Plain, Residual, and Dense variants), Fusion Block, and MLP Prediction Block.

The final architecture consists of a GCN Backbone, a Fusion Block for multi-scale feature integration, and an MLP Prediction Block for final classification.

Experimental Results

The primary evaluation was conducted on the S3DIS (Stanford Large-Scale 3D Indoor Spaces) dataset, which requires semantic segmentation of 3D point clouds representing indoor environments.

Performance Gains

The authors successfully trained a 56-layer GCN, the deepest reported at the time. Their ResGCN-28 model achieved a mean Intersection over Union (mIoU) of 60.0%, outperforming the previous state-of-the-art (DGCNN) by 3.7%.

Qualitative Improvements

DeepGCNs showed a marked improvement in identifying "hard" classes that require broader context, such as beams, bookcases, and doors. Shallow models often confuse these with walls because they lack the receptive field to understand the global geometry.

Conclusion and Future Work

The DeepGCN framework proves that depth is as beneficial for graphs as it is for images. By mitigating over-smoothing and vanishing gradients, GCNs can scale to dozens of layers, leading to significant performance boosts in complex 3D tasks.

Future research directions include optimizing the computational cost of dynamic k-NN (re-calculating neighbors at every layer) and exploring more advanced concepts like deformable graph convolutions to further enhance how these networks handle intricate 3D geometries.

Summary updated

Abstract

Convolutional Neural Networks (CNNs) achieve impressive performance in a wide variety of fields. Their success benefited from a massive boost when very deep CNN models were able to be reliably trained. Despite their merits, CNNs fail to properly address problems with non-Euclidean data. To overcome this challenge, Graph Convolutional Networks (GCNs) build graphs to represent non-Euclidean data, borrow concepts from CNNs, and apply them in training. GCNs show promising results, but they are usually limited to very shallow models due to the vanishing gradient problem. As a result, most state-of-the-art GCN models are no deeper than 3 or 4 layers. In this work, we present new ways to successfully train very deep GCNs. We do this by borrowing concepts from CNNs, specifically residual/dense connections and dilated convolutions, and adapting them to GCN architectures. Extensive experiments show the positive effect of these deep GCN frameworks. Finally, we use these new concepts to build a very deep 56-layer GCN, and show how it significantly boosts performance (+3.7% mIoU over state-of-the-art) in the task of point cloud semantic segmentation. We believe that the community can greatly benefit from this work, as it opens up many opportunities for advancing GCN-based research.
BibTeX
@inproceedings{li2019can,
  author = {Guohao Li and Matthias Müller and Ali K. Thabet and Bernard Ghanem},
  title = {{Can GCNs Go as Deep as CNNs?}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00936},
}
Figure