Learning Local-Global Contextual Adaptation for Fully End-to-End Bottom-Up Human Pose Estimation

Nan Xue, Tianfu Wu, Zhen Zhang, Gui-Song Xia

TL;DR

LOGO-CAP bridges the performance gap in bottom-up human pose estimation by refining fast, coarse center-offset predictions using a fully end-to-end framework. Inspired by Daniel Kahneman’s "thinking, fast and slow," it lifts initial keypoint estimates into local and global expansion maps, using convolutional message passing and dynamic convolution to achieve state-of-the-art accuracy at near real-time speeds (over 20 FPS).

What Problem Does This Paper Address?

In 2D multi-person pose estimation, bottom-up approaches detect all body joints in an image and group them into individuals. These generally fall into two categories:

  1. Limb/Grouping-based methods (e.g., Part Affinity Fields, Associative Embeddings): Accurate, but computationally expensive and reliant on non-differentiable heuristic grouping algorithms.
  2. Direct center-offset regression (e.g., CenterNet): Conceptually simple and fast, representing poses as star graphs offset from a person center anchor. However, direct offset regression struggles with localization accuracy, particularly for long-range limbs and articulated poses.

Existing solutions often apply complex multi-stage post-processing (such as rescoring networks or greedy matching). LOGO-CAP proposes a fully end-to-end, differentiable framework that fixes offset regression inaccuracies without sacrificing speed.

Methods

The authors structure their pipeline around a "Fast Initializer, Slow Solver" paradigm:

[HRNet Backbone] ──┬──> Center Heatmaps & Offset Fields ──> Fast Initializer (Coarse Poses)
                   │                                                  │
                   ├──> Feature Maps ──> Local KEMs (11x11)           │
                   │                           │                      │
                   │               Conv Message Passing (CMP)         │
                   │                           │                      │
                   │                           ▼                      │
                   │               Keypoint Attraction Maps (KAMs)    │
                   │                           │                      │
                   └──> Heatmaps ──────> Global KEMs (97x97)          │
                                               │ (Dynamic Conv)       ▼
                                               ▼                 Refined Offsets
                                    Refined Heatmaps ─────────> Final Pose Output
  1. Fast Keypointer (Initial Prediction): An HRNet backbone predicts keypoint heatmaps, center heatmaps, and 2D offset displacement fields. A quick peak-selection step generates initial coarse poses.
  2. Keypoint Expansion Maps (KEMs): Instead of committing to isolated coordinates, the model extracts spatial windows around each keypoint:
  3. Local KEMs ($11 \times 11$): Sampled via bilinear interpolation from latent feature maps to represent local uncertainty.
  4. Global KEMs ($97 \times 97$): Sampled across a wider receptive field from predicted heatmaps and modulated by a Gaussian prior.
  5. Convolutional Message Passing (CMP): Concatenated local KEMs pass through convolutional layers with Attentive Normalization to model structural anatomical constraints across body parts. This outputs Keypoint Attraction Maps (KAMs), directly supervised by an Object Keypoint Similarity (OKS) loss.
  6. Local-Global Contextual Adaptation: The learned KAMs act as dynamic, pose-sensitive convolutional kernels applied directly to the enlarged Global KEMs.
  7. Pose Decoding: The final keypoint positions are extracted via a convex combination of the top-2 activations in the adapted heatmaps, weighted by the center confidence score.

Datasets

  • MS COCO (Keypoint): Evaluated on the val-2017 (5k images) and test-dev 2017 (20k images) benchmarks; trained on the 65k training split.
  • OCHuman: A dedicated evaluation dataset for heavily occluded human pose scenarios (4,713 images containing 8,110 heavily overlapping person instances). No new datasets were introduced.

Key Insights and Ideas

  • Untapped Potential in Center-Offset Regression: The authors observed that if you take vanilla center-offset predictions and simply perform an oracle search in an $11 \times 11$ window, accuracy leaps from 60.1 to 88.9 AP. The coarse model gets close, but standard regressors lack spatial sub-pixel precision.
  • Why Plain Local Heatmaps Fail: Simply predicting a local heatmap per joint doesn't work because no new structural context is introduced. To resolve ambiguity, the network must combine inter-joint structural relationships with global contextual grounding.
  • Dynamic Kernels as Adapters: Using the structurally-aware local representations (KAMs) as dynamic convolution filters on global heatmaps enables the network to adaptively "pull" predictions toward the true joint center in a fully differentiable manner.

Key Results

  • COCO test-dev 2017: LOGO-CAP achieves 68.2 AP with HRNet-W32 and 70.8 AP with HRNet-W48, outperforming prior bottom-up methods like DEKR (67.3 / 70.0 AP with rescoring; 66.6 / 69.3 AP without) and HigherHRNet (68.4 AP).
  • OCHuman Benchmark: Demonstrates strong generalization under dense occlusion, reaching 41.2 Val AP / 40.4 Test AP (HRNet-W48), outperforming both top-down and bottom-up competitors (e.g., DEKR at 38.8 / 38.2 AP).
  • Inference Speed: Running on a single TITAN RTX GPU, LOGO-CAP processes frames in 48 ms (~20.7 FPS) with HRNet-W32 and 112 ms (~8.95 FPS) with HRNet-W48—faster than DEKR, CenterNet, and PifPaf.

Limitations and Future Work

  • Performance Gap to the Upper Bound: Although LOGO-CAP achieves 70.0 AP on COCO val, an oracle search within the same local window reaches 88.9 AP, indicating room for improved refinement architectures.
  • Message Passing Capacity: The CMP module relies on lightweight convolutions. The authors note that exploring attention mechanisms or Vision Transformers could improve structural modeling, though likely at the expense of inference frame rate.

Summary updated

Abstract

This paper presents a method of learning LOcal-GlObal Contextual Adaptation for fully end-to-end and fast bottom-up human Pose estimation, dubbed as LOGOCAP. It is built on the conceptually simple center-offset formulation that lacks inaccuracy for pose estimation. When revisiting the bottom-up human pose estimation with the thought of “thinking, fast and slow” by D. Kahneman, we introduce a “slow keypointer” to remedy the lack of sufficient accuracy of the “fast keypointer”. In learning the “slow keypointer”, the proposed LOGO-CAP lifts the initial “fast” keypoints by offset predictions to keypoint expansion maps (KEMs) to counter their uncertainty in two modules. Firstly, the local KEMs (e.g. 11×11) are extracted from a low-dimensional feature map. A proposed convolutional message passing module learns to “re-focus” the local KEMs to the keypoint attraction maps (KAMs) by accounting for the structured output prediction nature of human pose estimation, which is directly supervised by the object keypoint similarity (OKS) loss in training. Secondly, the global KEMs are extracted, with a sufficiently large region-of-interest (e.g., 97 × 97), from the keypoint heatmaps that are computed by a direct map-to-map regression. Then, a local-global contextual adaptation module is proposed to convolve the global KEMs using the learned KAMs as the kernels. This convolution can be understood as the learnable offsets guided deformable and dynamic convolution in a pose-sensitive way. The proposed method is end-to-end trainable with near real-time inference speed, obtaining state-of-the-art performance on the COCO keypoint benchmark for bottom-up human pose estimation. With the COCO trained model, our LOGO-CAP also outperforms prior arts by a large margin on the challenging OCHuman dataset.
BibTeX
@inproceedings{xue2022learning,
  author = {Nan Xue and Tianfu Wu and Zhen Zhang and Gui-Song Xia},
  title = {{Learning Local-Global Contextual Adaptation for Fully End-to-End Bottom-Up Human Pose Estimation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.01272},
}
Figure