Hierarchical Binary CNNs for Landmark Localization with Limited Resources

Adrian Bulat, Georgios Tzimiropoulos

Introduction

Landmark localization—the task of pinpointing specific "fiducial points" on a human body or face—is a cornerstone of modern computer vision. While state-of-the-art models like the "Stacked Hourglass" network achieve remarkable accuracy, they are notoriously resource-hungry, often requiring high-end GPUs and hundreds of megabytes of memory. This hardware requirement creates a significant barrier for deployment on edge devices with limited power and memory.

In their work, Bulat and Tzimiropoulos present a comprehensive study on binarizing networks for fine-grained localization tasks. They demonstrate that through architectural innovation, it is possible to achieve near-state-of-the-art results on human pose and face alignment using models that are up to 39x smaller than their full-precision counterparts.

Rethinking the Residual Block

The core challenge in binarization is the "lossy" nature of constraining weights and activations to just -1 or 1. While this offers a 32x reduction in memory and significant speedups via bitwise operations, it often leads to a catastrophic drop in accuracy for tasks requiring sub-pixel precision.

The authors identify that standard residual blocks, which typically use a "bottleneck" structure with $1 \times 1$ convolutions, are ill-suited for binary networks. In a binary context, $1 \times 1$ filters have almost zero representational power, acting more as "feature blockers" than efficient dimensionality reducers.

To address this, the authors proposed a Hierarchical, Parallel, and Multi-scale block. Key features include:

  • Removal of $1 \times 1$ Convolutions: Replacing them with wider $3 \times 3$ layers improved absolute accuracy by over 8%.
  • Parallel Paths: The input is split into multiple branches to process information at different scales.
  • Hierarchical Structure: Branches are organized to ensure the "shortest path" for gradients is minimized, mitigating the fading gradient problem.
  • Decomposed Filters: Large receptive fields are created by stacking smaller, more efficient $3 \times 3$ binary filters.
Comparison of the original bottleneck residual block (a) and the proposed hierarchical binary block (b).

Training Strategies for Binary Networks

Beyond architecture, the authors found that training dynamics must be adjusted for binary precision. Binary gradients are typically an order of magnitude smaller than real-valued ones, which can hinder convergence.

By switching from the standard L2 (Mean Squared Error) loss to Sigmoid cross-entropy loss for heatmap regression, the authors boosted gradient magnitude by 10-15x. Additionally, they found that maintaining a ReLU activation after binary layers—despite the "Sign" function used for binarization—provided a 2% performance boost by introducing necessary non-linearity.

Experimental Evaluation

The proposed approach was evaluated on human pose estimation (MPII) and various face alignment datasets (AFLW, AFLW2000-3D).

On the MPII dataset, the hierarchical binary block achieved 76% PCKh, a significant improvement over the 67.2% achieved by a standard binarized bottleneck. The model also proved competitive with much larger real-valued models on face alignment tasks. Furthermore, in facial part segmentation, the binary model nearly matched real-valued performance, achieving 97.91% pixel accuracy.

Performance curves (PCKh) on the MPII validation set across different body parts, showing the effectiveness of the proposed binary architecture.

In terms of efficiency, the models achieved a theoretical 39x memory compression. Practical tests showed a 3.5x speedup on GPUs, with projected speedups of up to 58x for optimized CPU implementations.

Conclusion

This work proves that landmark localization can be efficiently deployed on resource-constrained hardware without sacrificing significant accuracy. By moving away from standard architectures and focusing on gradient flow and representational power in binary space, the authors have bridged the gap between high-performance research models and practical edge deployment. Future work may explore hybrid precision or novel loss functions to further close the remaining performance gap with full-precision models.

Reference: Bulat, A., & Tzimiropoulos, G. (2020). Hierarchical Binary CNNs for Landmark Localization with Limited Resources. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(2), 343-356.

Summary updated

Abstract

Our goal is to design architectures that retain the groundbreaking performance of Convolutional Neural Networks (CNNs) for landmark localization and at the same time are lightweight, compact and suitable for applications with limited computational resources. To this end, we make the following contributions: (a) we are the first to study the effect of neural network binarization on localization tasks, namely human pose estimation and face alignment. We exhaustively evaluate various design choices, identify performance bottlenecks, and more importantly propose multiple orthogonal ways to boost performance. (b) Based on our analysis, we propose a novel hierarchical, parallel and multi-scale residual architecture that yields large performance improvement over the standard bottleneck block while having the same number of parameters, thus bridging the gap between the original network and its binarized counterpart. (c) We perform a large number of ablation studies that shed light on the properties and the performance of the proposed block. (d) We present results for experiments on the most challenging datasets for human pose estimation and face alignment, reporting in many cases state-of-the-art performance. (e) We further provide additional results for the problem of facial part segmentation. Code can be downloaded from https://www.adrianbulat.com/binary-cnn-landmarks.
BibTeX
@article{bulat2020hierarchical,
  author = {Adrian Bulat and Georgios Tzimiropoulos},
  title = {{Hierarchical Binary CNNs for Landmark Localization with Limited Resources}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2020},
  doi = {10.1109/tpami.2018.2866051},
}
Figure