Semantic Graph Convolutional Networks for 3D Human Pose Regression

Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, Dimitris N. Metaxas

Abstract

In this paper, we study the problem of learning Graph Convolutional Networks (GCNs) for regression. Current architectures of GCNs are limited to the small receptive field of convolution filters and shared transformation matrix for each node. To address these limitations, we propose Semantic Graph Convolutional Networks (SemGCN), a novel neural network architecture that operates on regression tasks with graph-structured data. SemGCN learns to capture semantic information such as local and global node relationships, which is not explicitly represented in the graph. These semantic relationships can be learned through end-to-end training from the ground truth without additional supervision or hand-crafted rules. We further investigate applying SemGCN to 3D human pose regression. Our formulation is intuitive and sufficient since both 2D and 3D human poses can be represented as a structured graph encoding the relationships between joints in the skeleton of a human body. We carry out comprehensive studies to validate our method. The results prove that SemGCN outperforms state of the art while using 90% fewer parameters.

Introduction

Estimating the 3D position of a human body from a 2D image is a fundamental "inverse problem" in computer vision. Because a 2D projection loses depth information, multiple 3D poses can correspond to the same 2D representation. While deep learning has significantly advanced this field, most models treat the human skeleton as either a flat vector of coordinates or a simple grid.

Researchers from Rutgers and Binghamton University argue that the skeleton should be treated as a graph. Their work introduces Semantic Graph Convolutional Networks (SemGCN), a framework designed to understand the complex relationships between joints more intelligently than standard Graph Convolutional Networks (GCNs).

Limitations of Vanilla GCNs

Standard Convolutional Neural Networks (CNNs) excel at grid-like data (e.g., pixels). However, a human skeleton is naturally represented as a graph of joints and bones. While GCNs were designed for such data, "vanilla" GCNs face two major hurdles in regression tasks like 3D pose estimation:

  1. Shared Transformation Matrices: Standard GCNs typically share a transformation matrix across all nodes. This forces the network to treat the relationship between a "hip and knee" identically to an "elbow and wrist," ignoring the distinct mechanical and semantic roles of different joint connections.
  2. Limited Receptive Field: GCN filters usually only aggregate information from immediate neighbors (1-hop). Capturing long-range dependencies, such as the relationship between a foot and the head, requires deep, often unstable stacks of layers.

The SemGCN Architecture

To overcome these limitations, SemGCN introduces two primary innovations: Semantic Graph Convolutions and Interleaved Non-Local Layers.

Semantic Graph Convolution (SemGConv)

Instead of relying on a fixed adjacency matrix, SemGConv learns a weighting matrix ($M$) applied to the graph edges.

Evolution of convolutions: from standard grid-based (a) to shared graph weights (b), individual weights (c), and finally the proposed semantic channel-wise weights (d).

As shown in Figure 1, SemGConv moves beyond shared weights by learning channel-wise weighting. Unlike previous methods that apply a single weight to an entire edge, SemGConv learns different weights for every feature channel. This allows the model to capture nuanced anatomical priors directly from data. Furthermore, Softmax normalization is used to stabilize the contributions of neighboring nodes during training.

Interleaved Non-Local Layers

To address the receptive field problem, the authors incorporate "Non-Local" operations. These layers calculate relationships between every pair of joints in the skeleton, regardless of physical connectivity. By interleaving these with SemGConv layers within a residual structure, the model captures both local anatomical constraints and global body posture.

The SemGCN architecture featuring interleaved semantic graph convolutions and non-local layers within a residual framework.

Experimental Results

The model was rigorously evaluated on the Human3.6M benchmark and the MPII dataset for "in-the-wild" generalization.

  • State-of-the-Art Accuracy: On Human3.6M, SemGCN achieved a Mean Per Joint Position Error (MPJPE) of 43.8mm using ground truth 2D joints, a 20% improvement over previous GCN-based methods.
  • Extreme Efficiency: The model achieves these results with approximately 0.43M parameters, representing a 90% reduction compared to the ~4.29M parameters required by previous leading fully-connected baselines.
  • Real-Time Performance: With a forward pass of only 1.8ms on a Titan Xp GPU, SemGCN is highly suitable for real-time applications.

Conclusion

SemGCN demonstrates that incorporating semantic awareness into graph convolutions significantly improves 3D pose regression. By learning channel-wise edge weights and utilizing non-local layers, the model captures both local and global dependencies efficiently. Future work includes extending the architecture to exploit temporal information in video sequences and applying it to other graph-based regression tasks like 3D mesh deformation.

Summary updated

Abstract

In this paper, we study the problem of learning Graph Convolutional Networks (GCNs) for regression. Current architectures of GCNs are limited to the small receptive field of convolution filters and shared transformation matrix for each node. To address these limitations, we propose Semantic Graph Convolutional Networks (SemGCN), a novel neural network architecture that operates on regression tasks with graph-structured data. SemGCN learns to capture semantic information such as local and global node relationships, which is not explicitly represented in the graph. These semantic relationships can be learned through end-to-end training from the ground truth without additional supervision or hand-crafted rules. We further investigate applying SemGCN to 3D human pose regression. Our formulation is intuitive and sufficient since both 2D and 3D human poses can be represented as a structured graph encoding the relationships between joints in the skeleton of a human body. We carry out comprehensive studies to validate our method. The results prove that SemGCN outperforms state of the art while using 90% fewer parameters.
BibTeX
@inproceedings{zhao2019semantic,
  author = {Long Zhao and Xi Peng and Yu Tian and Mubbasir Kapadia and Dimitris N. Metaxas},
  title = {{Semantic Graph Convolutional Networks for 3D Human Pose Regression}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=cCRUTDpbXw},
}
Figure