Abstract
Semantic correspondence aims to establish dense matches between images of different objects within the same category. This task is complicated by intra-class variations in appearance, pose, and scale. We present LPMFlow, a novel framework that leverages layout-aware representation learning and a progressive feature upscaling strategy to achieve high-precision, pixel-level matching. LPMFlow effectively mitigates geometric confusion and scale inconsistency, setting new state-of-the-art benchmarks on SPair-71K, PF-PASCAL, and PF-WILLOW.
Introduction
Semantic correspondence is the foundational task of identifying visually and semantically equivalent points across distinct images of the same object class—for instance, matching the wing of one bird species to the wing of another. Unlike traditional optical flow, which tracks identical pixels across temporal frames, semantic matching must overcome significant "intra-class" gaps, including variations in lighting, viewpoint, and object geometry.
The authors identify three critical challenges in current approaches:
- Geometric Confusion: Models often mismatch symmetrical or repetitive parts (e.g., left vs. right wheels) due to a lack of global layout understanding.
- Scale Inconsistency: Disparities in object size or distance from the camera hinder uniform matching resolution.
- Pixel Ambiguity: Local textures of neighboring pixels are often indistinguishable, leading to coarse or "blurry" flow predictions.
Methodology
LPMFlow addresses these issues through a tripartite architecture designed for robust representation and high-resolution output.
Layout-aware Representation Learning (LARL)
Utilizing a Transformer backbone, this module introduces a self-supervised task called Referenced Patch Token Correction (RPTC). By replacing source image patches and requiring the model to reconstruct them using the target image as a reference, the network is forced to learn spatial relationships and global geometric layouts rather than relying on local textures.
Progressive Feature Super-Resolution (PFSR)
To avoid the loss of detail inherent in upscaling final correlation maps, LPMFlow progressively upscales the features themselves. It generates 4D similarity tensors at multiple scales, capturing the likelihood of pixel-wise matches while preserving semantic richness.
Multi-scale Matching Flow Integration (MMFI)
This module transforms high-dimensional 4D tensors into 2D matching flows. A coarse-to-fine refinement structure, powered by Swin-attention blocks, fuses these multi-scale flows to produce a final prediction with pixel-level precision, reaching up to 1/2 the original image scale.
Evaluation and Results
The model was rigorously tested on the SPair-71K, PF-PASCAL, and PF-WILLOW benchmarks. LPMFlow achieved a state-of-the-art [email protected] of 65.6% on SPair-71K, surpassing the previous leading method, ACTR, by 3.5%.
Notably, the performance gap widens under stricter evaluation metrics. At [email protected], LPMFlow outperforms competitors by 4.7%, highlighting its superior accuracy in fine-grained matching. Despite its architectural complexity, the model maintains an efficient inference time of approximately 85ms, outperforming older methods like NC-Net.
Conclusion and Future Work
LPMFlow demonstrates that prioritizing global layout and feature-level upscaling is key to achieving high-precision semantic correspondence. While the model currently relies on global [CLS] tokens and sparse keypoint annotations, it provides a robust foundation for future work in unsupervised pixel-level learning and multi-instance matching in complex scenes.
Summary updated