Recombinator Networks: Learning Coarse-to-Fine Feature Aggregation

Sina Honari, Jason Yosinski, Pascal Vincent, Christopher J. Pal

Introduction

Precise spatial localization in computer vision, such as facial keypoint detection, faces a fundamental conflict between robustness and precision. Deep Convolutional Neural Networks (CNNs) typically employ max-pooling to achieve translation invariance, which is beneficial for classification but detrimental to exact localization as spatial information is discarded. While shallow networks preserve detail, they lack the global context necessary to resolve ambiguities caused by occlusions or complex poses.

Recombinator Networks (RCN) address this by introducing a "coarse-to-fine" architecture. Unlike traditional approaches that aggregate multi-resolution features only at the final output stage, RCN integrates coarse, abstract information into finer layers early in the network hierarchy. This "intermediate conditioning" allows fine-grained detectors to be guided by global context throughout their formation.

The Recombinator Architecture

The core innovation of RCN is the method of feature aggregation. In standard "SumNet" architectures, features from different resolutions are upsampled and summed at the very end. RCN instead uses a branching structure where coarse features from deep layers are upsampled and concatenated with the features of the subsequent finer layer.

Comparison of traditional summation-based aggregation (left) versus the Recombinator Network's concatenation-based early fusion (right).

This early fusion enables the network to learn non-linear combinations of resolutions. By conditioning fine-scale features on coarse-scale "global" information (e.g., the general location of a face), the model can more effectively distinguish relevant local edges (like an eyelid) from noise or shadows. This architectural choice not only improves accuracy but also leads to significantly faster convergence during training.

Denoising and Occlusion Handling

To further refine localization, the system incorporates a secondary Denoising Keypoint Model. This component acts as a learned prior for facial geometry. It takes the initial keypoint predictions from the RCN—represented as one-hot heatmaps—and "denoises" them to ensure the final output conforms to a valid face shape, independent of the original image pixels.

The integrated pipeline showing the Recombinator Network (RCN) providing initial predictions to a Denoising Keypoint Model for final refinement.

Robustness is further enhanced through "Occlusion Training." By randomly occluding parts of the face with black rectangles during training, the network is forced to rely on global structural cues when local features are missing, a common challenge in real-world "in-the-wild" datasets.

Experimental Results

RCN was evaluated on several benchmark datasets, including AFW, AFLW, and the challenging 300W (IBUG subset). Key findings include:

  • State-of-the-Art Accuracy: RCN achieved a 30% reduction in mean error compared to previous leading models like TCDCN on AFW and AFLW. On the IBUG subset, it reached a mean error of 4.81%, outperforming competitors without requiring massive auxiliary datasets.
  • Training Efficiency: Despite its depth, RCN converges much faster than summation-based baselines. In tests, RCN reached a target error rate in 15 epochs (20 minutes), whereas the SumNet baseline required 110 epochs (nearly 2 hours).
  • Inference Speed: The model is highly optimized for real-time applications, with an inference time of approximately 2.5ms per image on a K20 GPU.

Conclusion

Recombinator Networks demonstrate that early, non-linear fusion of multi-scale features is superior to late-stage linear summation for localization tasks. By combining this architecture with a geometry-aware denoising model and occlusion-robust training, RCN provides a powerful framework for precise and efficient facial landmark detection. Future work may involve end-to-end joint training of the RCN and denoising components to further push the boundaries of spatial precision.

Summary updated

BibTeX
@inproceedings{honari2016recombinator,
  author = {Sina Honari and Jason Yosinski and Pascal Vincent and Christopher J. Pal},
  title = {{Recombinator Networks: Learning Coarse-to-Fine Feature Aggregation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2016},
  url = {https://openreview.net/forum?id=H14Kf6Zd-B},
}
Figure