Neural Best-Buddies: Sparse Cross-Domain Correspondence

Kfir Aberman, Jing Liao, Mingyi Shi, Dani Lischinski, Baoquan Chen, Daniel Cohen-Or

Abstract

Finding semantic correspondences between images of different object categories is a fundamental challenge in computer vision. "Neural Best-Buddies" (NBB) addresses this by identifying sparse corresponding points across drastically different domains, such as animals and vehicles. By leveraging the hierarchical features of a pre-trained CNN and a novel local style-normalization technique, NBB enables automated high-quality image morphing and semantic hybridization.

Introduction

Identifying which point in one image matches another is a cornerstone of computer vision, essential for tasks ranging from 3D reconstruction to image editing. While traditional methods like SIFT or SURF excel at matching different views of the same scene, they struggle with cross-domain correspondence—for instance, matching the wing of a plane to the wing of a bird. These objects share semantic meaning but differ significantly in appearance and geometry.

Figure 3: Neural Best-Buddies identifies semantically related landmarks across diverse categories like cats, lions, planes, and birds.

Methodology

The NBB framework utilizes a coarse-to-fine approach based on the feature hierarchy of a pre-trained VGG-19 network.

Neural Best Buddies and Percolation

Two neurons are defined as "Best Buddies" if they are mutual nearest neighbors in the feature space. The algorithm begins at the deepest semantic layer (Level 5) to find initial matches. It then "percolates" these matches down through shallower layers (Level 4 to Level 1), using the receptive fields of higher-level NBBs to constrain the search space at finer scales.

Figure 6: The coarse-to-fine percolation process maps semantic features from deep layers to precise spatial locations in shallower layers.

Local Appearance Normalization

To overcome domain-specific textures (e.g., fur vs. metal), the authors employ a local style-transfer technique. By normalizing the mean and variance of features within local search windows, the network focuses on shared structural information rather than superficial appearance. This "secret sauce" allows the algorithm to ignore domain-specific styles and focus on semantic structure.

Filtering and Distribution

Matches are filtered based on activation values to ensure they represent significant features that the CNN "cares" about, such as eyes or corners. Finally, k-means clustering ensures a spatially diverse set of landmarks, preventing points from clustering in a single region.

Results and Applications

The method was evaluated on the Pascal 3D+ dataset for intra-class matching, achieving a 31.4% Percentage of Correct Keypoints (PCK), outperforming previous state-of-the-art methods like Deep Analogy (28.3%). A user study further confirmed that NBB-generated points align more closely with human intuition for cross-domain tasks than competing methods.

These sparse correspondences enable impressive applications, including fully automated cross-domain morphing (e.g., a cat turning into a human) and "Semantic Hybridization," where distinctive features of two different species are combined into a single coherent image.

Conclusion

Neural Best-Buddies provides a robust mechanism for finding semantic landmarks across disparate domains. While it can occasionally be misled by purely geometric similarities or missing parts, it represents a significant step forward in automated cross-domain image understanding and manipulation.

Summary updated

Abstract

Correspondence between images is a fundamental problem in computer vision, with a variety of graphics applications. This paper presents a novel method for sparse cross-domain correspondence. Our method is designed for pairs of images where the main objects of interest may belong to different semantic categories and differ drastically in shape and appearance, yet still contain semantically related or geometrically similar parts. Our approach operates on hierarchies of deep features, extracted from the input images by a pre-trained CNN. Specifically, starting from the coarsest layer in both hierarchies, we search for Neural Best Buddies (NBB): pairs of neurons that are mutual nearest neighbors. The key idea is then to percolate NBBs through the hierarchy, while narrowing down the search regions at each level and retaining only NBBs with significant activations. Furthermore, in order to overcome differences in appearance, each pair of search regions is transformed into a common appearance. We evaluate our method via a user study, in addition to comparisons with alternative correspondence approaches. The usefulness of our method is demonstrated using a variety of graphics applications, including cross-domain image alignment, creation of hybrid images, automatic image morphing, and more.
BibTeX
@article{aberman2018neural,
  author = {Kfir Aberman and Jing Liao and Mingyi Shi and Dani Lischinski and Baoquan Chen and Daniel Cohen-Or},
  title = {{Neural Best-Buddies: Sparse Cross-Domain Correspondence}},
  journal = {ACM Trans. Graphics},
  year = {2018},
  doi = {10.1145/3197517.3201332},
}
Figure