Pyramidal Semantic Correspondence Networks

Sangryul Jeon, Seungryong Kim, Dongbo Min, Kwanghoon SohnIEEE TPAMI 2022 · OpenReview

Abstract

This paper presents a deep architecture, called pyramidal semantic correspondence networks (PSCNet), that estimates locally-varying affine transformation fields across semantically similar images. To deal with large appearance and shape variations that commonly exist among different instances within the same object category, we leverage a pyramidal model where the affine transformation fields are progressively estimated in a coarse-to-fine manner so that the smoothness constraint is naturally imposed. Different from the previous methods which directly estimate global or local deformations, our method first starts to estimate the transformation from an entire image and then progressively increases the degree of freedom of the transformation by dividing coarse cell into finer ones. To this end, we propose two spatial pyramid models by dividing an image in a form of quad-tree rectangles or into multiple semantic elements of an object. Additionally, to overcome the limitation of insufficient training data, a novel weakly-supervised training scheme is introduced that generates progressively evolving supervisions through the spatial pyramid models by leveraging a correspondence consistency across image pairs. Extensive experimental results on various benchmarks including TSS, Proposal Flow-WILLOW, Proposal Flow-PASCAL, Caltech-101, and SPair-71k demonstrate that the proposed method outperforms the lastest methods for dense semantic correspondence.

Introduction

Establishing dense correspondences is a fundamental task in computer vision, but "semantic" correspondence is significantly harder than standard optical flow or stereo matching. While optical flow deals with the same object moving slightly between frames, semantic correspondence requires matching different instances within the same category—for example, matching the headlight of a red sedan to the headlight of a blue SUV.

The core challenges are intra-class variations: objects differ wildly in appearance, scale, and shape. Previous deep learning methods typically fell into two traps: global models that estimate a single transformation for the whole image (robust but lacking detail), and local models that estimate independent pixel-level matches (detailed but lacking spatial consistency).

Methodology

The authors propose Pyramidal Semantic Correspondence Networks (PSCNet). The key idea is to start with a global view and "zoom in" progressively, increasing the degrees of freedom of the transformation at each level.

1. The Pyramidal Models

The paper introduces two ways to divide the image:

  • PSCNet-UR (Uniform Rectangles): Divides the image into a standard quad-tree (1x1, 2x2, 4x4 grid).
  • PSCNet-SE (Semantic Elements): A more sophisticated approach that uses an unsupervised "element detection network" to find meaningful parts of an object (like wheels or windows) and tracks them across levels.
The PSCNet architecture progressively increases degrees of freedom from global (cell lv. 1) to local (pixel lv.) transformations.

2. Constrained Cost Volume and Differentiable Upsampling

Standard matching often builds a "cost volume" comparing every pixel in image A to every pixel in image B. PSCNet uses a Constrained Cost Volume, where the search space for a pixel at a fine level is restricted to a small window around the location predicted by the previous, coarser level. This reduces noise and computational load. Since the "Semantic Elements" model (SE) uses irregularly distributed points rather than a grid, the authors developed a differentiable upsampling layer inspired by Moving Least Squares (MLS) to create a dense, smooth transformation field from sparse part-based matches.

3. Weakly-Supervised Training

Because pixel-perfect ground truth for semantic matching is nearly impossible to label manually, the authors use a correspondence consistency check. During training, the model matches forward (Image A to B) and backward (B to A); if the cycle doesn't return to the starting point, the match is discarded. This allows the model to learn from raw image pairs without explicit labels.

Experimental Results

PSCNet achieved state-of-the-art (SOTA) results across five major benchmarks: TSS, Proposal Flow (WILLOW & PASCAL), Caltech-101, and SPair-71k.

Visualizing the unsupervised discovery of semantic elements at the third level of the pyramid.
  • SPair-71k: On this highly difficult dataset, PSCNet-SE achieved a PCK (Probability of Correct Keypoint) of 27.0%, outperforming previous SOTA methods like NCNet and SFNet.
  • TSS Benchmark: It reached an average accuracy of 82.3%, a significant jump over the previous best of ~75%.
  • Efficiency: Despite the multi-level approach, the parallelized version of PSCNet-SE runs in about 248ms per pair, making it faster than several non-pyramidal competitors.

Conclusion

PSCNet demonstrates that a pyramidal, coarse-to-fine approach effectively addresses the challenges of semantic correspondence. By enforcing smoothness through architecture rather than manual penalties and leveraging unsupervised part discovery, the model achieves superior performance across diverse object categories. While powerful, the authors note that extremely large geometric differences remain a challenge for future work, potentially requiring more robust global descriptors or better handling of occlusions.

Summary updated

Abstract

This paper presents a deep architecture, called pyramidal semantic correspondence networks (PSCNet), that estimates locally-varying affine transformation fields across semantically similar images. To deal with large appearance and shape variations that commonly exist among different instances within the same object category, we leverage a pyramidal model where the affine transformation fields are progressively estimated in a coarse-to-fine manner so that the smoothness constraint is naturally imposed. Different from the previous methods which directly estimate global or local deformations, our method first starts to estimate the transformation from an entire image and then progressively increases the degree of freedom of the transformation by dividing coarse cell into finer ones. To this end, we propose two spatial pyramid models by dividing an image in a form of quad-tree rectangles or into multiple semantic elements of an object. Additionally, to overcome the limitation of insufficient training data, a novel weakly-supervised training scheme is introduced that generates progressively evolving supervisions through the spatial pyramid models by leveraging a correspondence consistency across image pairs. Extensive experimental results on various benchmarks including TSS, Proposal Flow-WILLOW, Proposal Flow-PASCAL, Caltech-101, and SPair-71k demonstrate that the proposed method outperforms the lastest methods for dense semantic correspondence.
BibTeX
@article{jeon2022pyramidal,
  author = {Sangryul Jeon and Seungryong Kim and Dongbo Min and Kwanghoon Sohn},
  title = {{Pyramidal Semantic Correspondence Networks}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  url = {https://openreview.net/forum?id=LisfuYDHucG},
}
Figure