Hierarchical binary CNNs for landmark localization with limited resources

Adrian Bulat, Georgios Tzimiropoulos

Abstract

Our goal is to design architectures that retain the groundbreaking performance of Convolutional Neural Networks (CNNs) for landmark localization and at the same time are lightweight, compact and suitable for applications with limited computational resources. We study the effect of neural network binarization on localization tasks, identify performance bottlenecks, and propose a novel hierarchical, parallel and multi-scale residual architecture. This design yields large performance improvements over standard bottleneck blocks while maintaining the same number of parameters, bridging the gap between original networks and their binarized counterparts. We report state-of-the-art performance on challenging datasets for human pose estimation and face alignment.

Introduction: The Efficiency Challenge in Localization

Landmark localization—identifying specific points on a face or body—has been revolutionized by Deep Learning. However, state-of-the-art models, such as the "Stacked Hourglass" network, are computationally intensive, typically requiring high-end GPUs and hundreds of megabytes of memory. This makes them impractical for real-time use on mobile phones or embedded devices.

While Network Binarization (constraining weights and activations to just -1 or +1) offers a path to massive efficiency, it usually results in a significant drop in accuracy, especially for "fine-grained" tasks like localization where precision is critical. This research investigates the causes of this performance degradation and proposes architectural solutions to make binary networks viable for landmark localization.

Methods: Rethinking the Bottleneck

The researchers identified that simply binarizing existing "bottleneck" blocks leads to a massive performance hit. To address this, they introduced a novel hierarchical architecture designed to optimize binary representation:

  1. Hierarchical & Multi-scale Structure: Instead of a single path, the proposed block splits the signal into multiple parallel branches with different receptive fields. This allows the network to process features at different scales simultaneously.
  2. Removing 1x1 Convolutions: In real-valued networks, 1x1 convolutions are used to save parameters. However, in the binary domain, 1x1 filters have almost zero representational power, acting merely as simple gates. Replacing them with 3x3 filters significantly boosts performance.
  3. Optimizing Gradient Flow: Binary networks often suffer from fading gradients. The hierarchical design ensures short paths back to the output for every layer, keeping the learning signal strong.
Comparison between the standard bottleneck block (a) and the proposed hierarchical, parallel, and multi-scale architecture (b).

Key Insights

  • The 1x1 Conv Trap: In binary networks, 1x1 convolutions are a bottleneck rather than an efficiency tool. Because a binary 1x1 filter can only represent two states, it lacks the capacity for complex feature extraction.
  • Receptive Field vs. Parameters: For binary networks, increasing the receptive field is more important than adding layers. The multi-scale approach provides a wider view without exploding the parameter count.
  • Width over Depth: Widening "thin" layers in a bottleneck block provides more tangible gains for binary networks than adding more sequential layers.

Experimental Results

The models were evaluated on gold-standard benchmarks, including MPII Human Pose and AFLW for face alignment.

  • Accuracy: On the MPII dataset, the proposed binary block improved the PCKh (a standard accuracy metric) by ~6% absolute compared to a standard binarized bottleneck, nearly matching full-precision models.
  • Efficiency: The binary model achieved a 39x compression rate in memory and a practical 3.5x speedup over highly optimized libraries like cuBLAS.
  • Face Alignment & Segmentation: The binarized network outperformed several real-valued state-of-the-art methods on the AFLW dataset and achieved a mean Intersection over Union (mIU) of 68.05% in facial part segmentation, approaching the real-valued baseline of 69.29%.
Qualitative comparison showing that the proposed binary network (top) produces pose estimations nearly identical to the real-valued baseline (bottom).

Conclusion

This work demonstrates that with proper architectural adjustments, binary CNNs can achieve high accuracy in landmark localization tasks while being significantly more efficient. While a small performance gap remains in very deep architectures due to activation noise, the proposed hierarchical block successfully bridges the gap for most practical applications, enabling state-of-the-art localization on resource-constrained devices.

Summary updated

Abstract

Our goal is to design architectures that retain the groundbreaking performance of Convolutional Neural Networks (CNNs) for landmark localization and at the same time are lightweight, compact and suitable for applications with limited computational resources. To this end, we make the following contributions: (a) we are the first to study the effect of neural network binarization on localization tasks, namely human pose estimation and face alignment. We exhaustively evaluate various design choices, identify performance bottlenecks, and more importantly propose multiple orthogonal ways to boost performance. (b) Based on our analysis, we propose a novel hierarchical, parallel and multi-scale residual architecture that yields large performance improvement over the standard bottleneck block while having the same number of parameters, thus bridging the gap between the original network and its binarized counterpart. (c) We perform a large number of ablation studies that shed light on the properties and the performance of the proposed block. (d) We present results for experiments on the most challenging datasets for human pose estimation and face alignment, reporting in many cases state-of-the-art performance. (e) We further provide additional results for the problem of facial part segmentation. Code can be downloaded from https://www.adrianbulat.com/binary-cnn-landmark
BibTeX
@article{bulat2020hierarchical,
  author = {Adrian Bulat and Georgios Tzimiropoulos},
  title = {{Hierarchical binary CNNs for landmark localization with limited resources}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2020},
  doi = {10.1109/tpami.2018.2866051},
}
Figure