Abstract
In this paper, we study the problem of learning Graph Convolutional Networks (GCNs) for regression. Current architectures of GCNs are limited to the small receptive field of convolution filters and shared transformation matrix for each node. To address these limitations, we propose Semantic Graph Convolutional Networks (SemGCN), a novel neural network architecture that operates on regression tasks with graph-structured data. SemGCN learns to capture semantic information such as local and global node relationships, which is not explicitly represented in the graph. These semantic relationships can be learned through end-to-end training from the ground truth without additional supervision or hand-crafted rules. We further investigate applying SemGCN to 3D human pose regression. Our formulation is intuitive and sufficient since both 2D and 3D human poses can be represented as a structured graph encoding the relationships between joints in the skeleton of a human body. We carry out comprehensive studies to validate our method. The results prove that SemGCN outperforms state of the art while using 90% fewer parameters.
Introduction
Predicting 3D human joint locations from a single 2D image or 2D pose is a classic "inverse problem" in computer vision. It is inherently ambiguous because multiple 3D configurations can project to the same 2D points. While standard Convolutional Neural Networks (CNNs) excel at processing grid-like data (pixels), human skeletons are better represented as graphs.
However, "vanilla" Graph Convolutional Networks (GCNs) have two major flaws for regression tasks like pose estimation. First, they typically apply a shared transformation matrix to every node, which lacks the expressiveness of position-specific kernels in CNNs. Second, they have limited receptive fields, usually aggregating information only from immediate neighbors (1-hop), making it difficult to learn global body constraints, such as the relationship between a foot and the head.
Methods
The authors propose a two-pronged architectural solution to overcome the limitations of standard GCNs: Semantic Graph Convolutions and interleaved non-local layers.
Semantic Graph Convolution (SemGConv)
Unlike standard GCNs that use a fixed adjacency matrix, SemGConv learns a weighting matrix ($M$) that acts as a "semantic" prior for the graph edges. Crucially, this is extended to channel-wise weights, meaning different feature channels can learn different relationships between joints. This mimics the expressive power of traditional CNN kernels while maintaining the efficiency of a graph structure.
Interleaved Non-Local Layers
To solve the "small receptive field" problem, the authors interleave SemGConv layers with non-local layers. These layers use a self-attention mechanism to compute relationships between all pairs of joints in the skeleton, regardless of whether they are physically connected. This allows the model to capture global symmetry and long-range dependencies.
Perceptual Feature Pooling
While the model can run on 2D coordinates alone, it can also incorporate image data. The authors use a ResNet backbone to extract visual features at the location of each 2D joint. These "perceptual features" are concatenated with the 2D coordinates to provide the GCN with visual context, helping to resolve depth ambiguities or occlusions.
Key Results
The model was primarily benchmarked on the Human3.6M dataset, containing 3.6 million video frames with high-accuracy motion capture ground truth.
- State-of-the-Art Accuracy: SemGCN achieves a Mean Per Joint Position Error (MPJPE) of 43.8mm when using ground truth 2D joints.
- Efficiency: By treating the skeleton as a graph rather than a flat vector, the model achieves high performance with only 0.43M parameters, compared to 4.29M in previous state-of-the-art fully-connected networks—a 90% reduction.
- Speed: Inference time is roughly 1.8ms per frame on a Titan Xp GPU, making it highly suitable for real-time applications.
Conclusion
SemGCN demonstrates that graph inductive biases are powerful for structured regression tasks. By learning semantic edge weights and integrating global non-local layers, the architecture achieves superior accuracy with a fraction of the parameters of previous methods. Future work will focus on incorporating temporal information from video sequences to further resolve movement ambiguities.
Summary updated