Pixel-Level Semantic Correspondence Through Layout-Aware Representation Learning and Multi-Scale Matching Integration

Yixuan Sun, Zhangyue Yin, Haibo Wang, Yan Wang, Xipeng Qiu, Weifeng Ge, Wenqiang Zhang

Abstract

Semantic correspondence aims to establish dense matches between images of different objects within the same category. This task is complicated by intra-class variations in appearance, pose, and scale. We present LPMFlow, a novel framework that leverages layout-aware representation learning and a progressive feature upscaling strategy to achieve high-precision, pixel-level matching. LPMFlow effectively mitigates geometric confusion and scale inconsistency, setting new state-of-the-art benchmarks on SPair-71K, PF-PASCAL, and PF-WILLOW.

Introduction

Semantic correspondence is the foundational task of identifying visually and semantically equivalent points across distinct images of the same object class—for instance, matching the wing of one bird species to the wing of another. Unlike traditional optical flow, which tracks identical pixels across temporal frames, semantic matching must overcome significant "intra-class" gaps, including variations in lighting, viewpoint, and object geometry.

The authors identify three critical challenges in current approaches:

  1. Geometric Confusion: Models often mismatch symmetrical or repetitive parts (e.g., left vs. right wheels) due to a lack of global layout understanding.
  2. Scale Inconsistency: Disparities in object size or distance from the camera hinder uniform matching resolution.
  3. Pixel Ambiguity: Local textures of neighboring pixels are often indistinguishable, leading to coarse or "blurry" flow predictions.

Methodology

LPMFlow addresses these issues through a tripartite architecture designed for robust representation and high-resolution output.

The LPMFlow architecture consists of three main stages: Layout-aware Representation Learning, Progressive Feature Super-resolution, and Multiscale Matching Flow Integration.

Layout-aware Representation Learning (LARL)

Utilizing a Transformer backbone, this module introduces a self-supervised task called Referenced Patch Token Correction (RPTC). By replacing source image patches and requiring the model to reconstruct them using the target image as a reference, the network is forced to learn spatial relationships and global geometric layouts rather than relying on local textures.

Progressive Feature Super-Resolution (PFSR)

To avoid the loss of detail inherent in upscaling final correlation maps, LPMFlow progressively upscales the features themselves. It generates 4D similarity tensors at multiple scales, capturing the likelihood of pixel-wise matches while preserving semantic richness.

Multi-scale Matching Flow Integration (MMFI)

This module transforms high-dimensional 4D tensors into 2D matching flows. A coarse-to-fine refinement structure, powered by Swin-attention blocks, fuses these multi-scale flows to produce a final prediction with pixel-level precision, reaching up to 1/2 the original image scale.

Evaluation and Results

The model was rigorously tested on the SPair-71K, PF-PASCAL, and PF-WILLOW benchmarks. LPMFlow achieved a state-of-the-art [email protected] of 65.6% on SPair-71K, surpassing the previous leading method, ACTR, by 3.5%.

Qualitative comparison of LPMFlow against existing methods (SCOT, CATs, ACTR). LPMFlow produces results significantly closer to the ground truth, demonstrating superior handling of geometric warping and scale.

Notably, the performance gap widens under stricter evaluation metrics. At [email protected], LPMFlow outperforms competitors by 4.7%, highlighting its superior accuracy in fine-grained matching. Despite its architectural complexity, the model maintains an efficient inference time of approximately 85ms, outperforming older methods like NC-Net.

Conclusion and Future Work

LPMFlow demonstrates that prioritizing global layout and feature-level upscaling is key to achieving high-precision semantic correspondence. While the model currently relies on global [CLS] tokens and sparse keypoint annotations, it provides a robust foundation for future work in unsupervised pixel-level learning and multi-instance matching in complex scenes.

Summary updated

Abstract

Establishing precise semantic correspondence across object instances in different images is a fundamental and challenging task in computer vision. In this task, difficulty arises often due to three challenges: confusing regions with similar appearance, inconsistent object scale, and indistinguishable nearby pixels. Recognizing these challenges, our paper proposes a novel semantic matching pipeline named LPMFlow toward extracting fine-grained semantics and geometry layouts for building pixel-level semantic correspondences. LPMFlow consists of three modules, each addressing one of the aforementioned challenges. The layout-aware representation learning module uniformly encodes source and target tokens to distinguish pixels or regions with similar appearances but different geometry semantics. The progressive feature superresolution module outputs four sets of 4D correlation tensors to generate accurate semantic flow between objects in different scales. Finally, the matching flow integration and refinement module is exploited to fuse matching flow in different scales to give the final flow predictions. The whole pipeline can be trained end-to-end, with a balance of computational cost and correspondence details. Extensive experiments based on benchmarks such as SPair-71K, PF-PASCAL, and PF-WILLOW have proved that the proposed method can well tackle the three challenges and outperform the previous methods, es-pecially in more stringent settings. Code is available at https://github.com/YXSUNMADMAX/LPMFlow.
BibTeX
@inproceedings{sun2024pixel,
  author = {Yixuan Sun and Zhangyue Yin and Haibo Wang and Yan Wang and Xipeng Qiu and Weifeng Ge and Wenqiang Zhang},
  title = {{Pixel-Level Semantic Correspondence Through Layout-Aware Representation Learning and Multi-Scale Matching Integration}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01613},
}
Figure