Abstract
This thesis proposes new architectures for deep neural networks with attention enhancement and multilinear algebra methods to increase their performance. We also explore graph convolutions and their particularities. We focus here on the problems related to real-time human pose estimation. We explore different architectures to reduce computational complexity, and, as a result, we propose two novel neural network models for 2D and 3D pose estimation. We also introduce a new architecture for Graph attention networks called Semantic Graph Attention.
Introduction
Human pose estimation—the task of identifying 2D or 3D coordinates of body joints—is a fundamental component of modern computer vision. However, its widespread adoption is hindered by two primary challenges. First, standard Convolutional Neural Networks (CNNs) are often over-parameterized, containing significant redundancy that makes them computationally expensive and slow on consumer-grade hardware. Second, traditional convolutions focus on local pixel neighborhoods, often failing to capture the global structural context of the human body, which is essential for accurate estimation during occlusions or motion blur.
This research addresses these hurdles by introducing a lightweight framework designed for real-time performance on standard CPUs. By combining tensor decompositions for 2D efficiency with a novel Semantic Graph Attention (SGAT) mechanism for 3D lifting, the proposed models achieve state-of-the-art accuracy while drastically reducing parameter counts.
Methodology
The framework decouples the pose estimation task into two distinct stages: efficient 2D detection and structural 3D lifting.
2D Efficiency via Factorized Convolutions
To optimize the 2D pose detector, the authors utilize Tucker Decomposition to factorize convolutional kernels. This process transforms a standard 4D kernel into a set of low-rank approximations, specifically two pointwise convolutions and one regular convolution with reduced spatial dimensions. To avoid manual tuning, Global Analytic Variational Bayesian Matrix Factorization (VBMF) is employed to automatically determine the optimal rank for each layer, ensuring maximum compression without sacrificing accuracy.
3D Lifting via Semantic Graph Attention (SGAT)
For 3D estimation, the human skeleton is modeled as a graph where joints are nodes. Building upon Semantic Graph Convolutions (SGCs), the authors introduce the Semantic Graph Attention (SGAT) mechanism. This novel attention layer projects joint data into a "frequency space" (akin to PCA) to identify global correlations. A sigmoid-activated gating function then recalibrates these features, emphasizing critical structural relationships and suppressing noise. This allows the network to maintain a global "big picture" of human movement, which is vital for resolving ambiguities in 2D inputs.
Evaluation and Results
The models were rigorously tested on industry-standard benchmarks, including Human 3.6M, COCO, and MPI-INF-3DHP.
Quantitative Performance
The proposed 3D model demonstrates exceptional efficiency, utilizing 58% fewer parameters than the original SGC model and up to 95% fewer than other popular baselines. Despite this reduction, accuracy improved, with a 4.9% reduction in Mean Per Joint Position Error (MPJPE) on the Human 3.6M dataset compared to previous graph-based methods.
In 2D tasks, the factorized model achieved a throughput of 13 FPS on an AMD Ryzen 7 CPU. In contrast, the industry-standard OpenPose model achieved only 0.3 FPS on the same hardware, representing a massive leap in accessibility for real-time applications.
Qualitative Robustness
Qualitative evaluations on "in-the-wild" images from the COCO dataset show that the model effectively handles diverse poses and environments, accurately reconstructing 3D structures from single 2D images.
Conclusion
This work presents a modular and highly efficient approach to human pose estimation. By leveraging tensor decompositions and semantic graph attention, the framework overcomes the computational bottlenecks of traditional CNNs while improving structural intelligence. While the current model is susceptible to error propagation from the 2D detector, future work integrating temporal information from video sequences promises to further enhance robustness and accuracy in complex, multi-person scenarios.
Summary updated