Neural best-buddies: sparse cross-domain correspondence

Kfir Aberman, Jing Liao, Mingyi Shi, Dani Lischinski, Baoquan Chen, Daniel Cohen-Or

Abstract

Finding semantic correspondences between images of different object categories is a fundamental challenge in computer vision. "Neural Best-Buddies" (NBB) addresses this by identifying sparse corresponding points across drastically different domains, such as animals and vehicles. By leveraging the hierarchical features of a pre-trained CNN and a novel local style-normalization technique, NBB enables automated high-quality image morphing and semantic hybridization.

Introduction

Identifying which point in one image matches another is a cornerstone of computer vision, essential for tasks ranging from 3D reconstruction to image editing. While traditional methods like SIFT or SURF excel at matching different views of the same scene, they struggle with cross-domain correspondence—for instance, matching the wing of a plane to the wing of a bird. These objects share semantic meaning but differ significantly in appearance and geometry.

Figure 3: Neural Best-Buddies identifies semantically related landmarks across diverse categories like cats, lions, planes, and birds.

Methodology

The NBB framework utilizes a coarse-to-fine approach based on the feature hierarchy of a pre-trained VGG-19 network.

Neural Best Buddies and Percolation

Two neurons are defined as "Best Buddies" if they are mutual nearest neighbors in the feature space. The algorithm begins at the deepest semantic layer (Level 5) to find initial matches. It then "percolates" these matches down through shallower layers (Level 4 to Level 1), using the receptive fields of higher-level NBBs to constrain the search space at finer scales.

Figure 6: The coarse-to-fine percolation process maps semantic features from deep layers to precise spatial locations in shallower layers.

Local Appearance Normalization

To overcome domain-specific textures (e.g., fur vs. metal), the authors employ a local style-transfer technique. By normalizing the mean and variance of features within local search windows, the network focuses on shared structural information rather than superficial appearance. This "secret sauce" allows the algorithm to ignore domain-specific styles and focus on semantic structure.

Filtering and Distribution

Matches are filtered based on activation values to ensure they represent significant features that the CNN "cares" about, such as eyes or corners. Finally, k-means clustering ensures a spatially diverse set of landmarks, preventing points from clustering in a single region.

Results and Applications

The method was evaluated on the Pascal 3D+ dataset for intra-class matching, achieving a 31.4% Percentage of Correct Keypoints (PCK), outperforming previous state-of-the-art methods like Deep Analogy (28.3%). A user study further confirmed that NBB-generated points align more closely with human intuition for cross-domain tasks than competing methods.

These sparse correspondences enable impressive applications, including fully automated cross-domain morphing (e.g., a cat turning into a human) and "Semantic Hybridization," where distinctive features of two different species are combined into a single coherent image.

Conclusion

Neural Best-Buddies provides a robust mechanism for finding semantic landmarks across disparate domains. While it can occasionally be misled by purely geometric similarities or missing parts, it represents a significant step forward in automated cross-domain image understanding and manipulation.

Summary updated

BibTeX
@article{aberman2018neural,
  author = {Kfir Aberman and Jing Liao and Mingyi Shi and Dani Lischinski and Baoquan Chen and Daniel Cohen-Or},
  title = {{Neural best-buddies: sparse cross-domain correspondence}},
  journal = {ACM Trans. Graphics},
  year = {2018},
  doi = {10.1145/3197517.3201332},
}
Figure