Abstract
Convolutional neural networks (CNNs) based approaches for semantic alignment and object landmark detection have improved their performance significantly. Current efforts for the two tasks focus on addressing the lack of massive training data through weakly- or unsupervised learning frameworks. In this paper, we present a joint learning approach for obtaining dense correspondences and discovering object landmarks from semantically similar images. Based on the key insight that the two tasks can mutually provide supervisions to each other, our networks accomplish this through a joint loss function that alternatively imposes a consistency constraint between the two tasks, thereby boosting the performance and addressing the lack of training data in a principled manner. To the best of our knowledge, this is the first attempt to address the lack of training data for the two tasks through the joint learning. To further improve the robustness of our framework, we introduce a probabilistic learning formulation that allows only reliable matches to be used in the joint learning process. With the proposed method, state-of-the-art performance is attained on several benchmarks for semantic matching and landmark detection.
Introduction
In computer vision, semantic alignment—finding dense correspondences between different instances of the same object class—and landmark detection—identifying key structural points—are fundamental but challenging tasks. The primary bottleneck for both is the scarcity of annotated data. Supervised learning requires thousands of images with pixel-perfect annotations, which are tedious and subjective to produce.
Existing weakly-supervised methods often rely on synthetic data (e.g., artificially warping a single image). However, these models frequently fail when faced with real-world "intra-class" variations, such as different car brands viewed from varying angles. This paper proposes that these two tasks are intrinsically linked and can be leveraged to supervise each other.
Methodology
The core of the proposed approach is a joint learning framework that exploits the synergy between alignment and detection. Landmarks provide structural "anchors" that guide the alignment network toward salient object parts. Conversely, dense correspondences from the alignment network serve as a consistency check for landmarks: a landmark detected on a source image should warp precisely to the corresponding landmark on a target image.
Key Components
- Mutual Supervision: The model uses a joint loss function that imposes a consistency constraint between the two tasks.
- Probabilistic Formulation: To handle unreliable matches caused by occlusions or lack of texture, an uncertainty module predicts a confidence map ($\sigma$). This allows the loss function to discount noisy data during training.
- Network Architecture:
- Feature Extraction: A shared ResNet-101 backbone.
- Landmark Detection: A decoder-style network outputting probability maps for $K$ landmarks, leveraging "self-similarity" scores to identify discriminative regions.
- Semantic Alignment: An encoder-decoder estimating locally-varying transformation fields and uncertainty maps.
- Alternating Optimization: Training alternates between optimizing landmark detection and alignment parameters, iteratively refining both.
Experiments and Results
The framework was evaluated on several benchmarks, including the newly introduced JLAD (Joint Learning of Alignment and Detection) dataset, which contains ~1,000 image pairs for 12 object categories and human faces.
Key Findings
- Ablation Study: On the JLAD dataset, joint learning improved alignment accuracy ([email protected]) from 63.2% (separate learning) to 72.7%.
- Semantic Matching: The model achieved state-of-the-art results on the PF-PASCAL benchmark with a PCK of 79.2% at $\alpha=0.15$.
- Landmark Detection: For facial landmarks (MAFL dataset), the model achieved an Inter-Ocular Distance (IOD) error of 2.98, outperforming previous unsupervised methods.
- Qualitative Performance: The model demonstrates robustness under significant changes in viewpoint and appearance where previous methods like CNNgeo or NCNet struggled.
Conclusion
This work demonstrates that joint learning of semantic alignment and landmark detection effectively addresses the lack of training data. By moving beyond synthetic warps to real image pairs and utilizing mutual geometric constraints, the model achieves superior performance. Future work may extend this principle to related tasks like 3D reconstruction or object part segmentation.
Summary updated