Siamese Regression Networks with Efficient mid-level Feature Extraction for 3D Object Pose Estimation

Andreas Doumanoglou, Vassileios Balntas, Rigas Kouskouridas, Tae-Kyun KimarXiv 2016 · OpenReview

TL;DR

This paper introduces Siamese Regression Networks (SRNs) for direct 3D object pose estimation from images. By enforcing a similarity constraint between feature and pose spaces using a Siamese architecture, SRNs learn more discriminative features and achieve improved pose estimation accuracy, even under severe occlusions.

What Problem Does This Paper Address?

Estimating the 3D pose of objects from images is a notoriously difficult computer vision problem, especially when objects are occluded, have varying scales, or are in cluttered environments. Existing state-of-the-art methods often tackle this indirectly: they first learn discriminative features and then use these features in a separate system (like nearest neighbor matching) to estimate the pose. This indirect approach can be suboptimal. Directly regressing pose angles with deep networks is challenging due to the complex, non-linear nature of angle spaces, which can lead to many local minima during training. This paper aims to bridge this gap by proposing an end-to-end framework that directly regresses 3D pose while simultaneously learning robust and discriminative features.

Methods

The core innovation is the Siamese Regression Network (SRN). Unlike traditional CNNs that process images independently, SRNs process pairs of images. The network has two main components:

  1. Feature Extractor (f): A convolutional neural network that maps an input image to a feature vector.
  2. Pose Regressor (g): A fully connected layer that takes the feature vector and outputs the 3D pose (represented as quaternions to avoid issues with angle periodicity).

The key novelty lies in the Siamese objective function. During training, the network takes pairs of images and their corresponding ground truth poses. The loss function enforces two main objectives:

  • Regression Loss ($l_R$): The standard Mean Squared Error (MSE) between the predicted pose and the ground truth pose.
  • Feature-Guided Pose Regression Loss ($l_F$): This is the crucial part. It enforces a similarity constraint: the Euclidean distance between the feature vectors of two images in the feature space should be similar to the Euclidean distance between their corresponding ground truth poses in the pose space. This is achieved by minimizing $||f(x_{n,1}) - f(x_{n,2})||^2 - ||y_{n,1} - y_{n,2}||^2$.

This Siamese approach encourages the feature space to be structured such that distances between features directly correspond to distances in pose space, leading to more discriminative features optimized for the pose estimation task. The authors also explore feature normalization and batch formation strategies to further improve performance. For handling occlusions, they propose an additional loss term that compares an occluded image with a synthetically rendered clean version of the same object at the same pose.

Datasets

The paper uses two datasets:

  1. LINEMOD [8]: A standard benchmark for 3D object pose estimation. The authors use a variant with RGBD images where objects are centered.
  2. Novel Hand-Object Dataset: A new dataset created by the authors to specifically address severe occlusions. It features an object being manipulated by a human hand, capturing realistic occlusion scenarios. This dataset contains approximately 21,000 training and 5,000 testing images.

Key Insights and Ideas

  • Direct Pose Regression is Viable: Contrary to prior work suggesting indirect methods are necessary, this paper demonstrates that direct end-to-end regression of 3D pose angles is achievable and beneficial.
  • Feature-Pose Space Alignment is Key: The core insight is that aligning distances in the learned feature space with distances in the pose space significantly boosts regression performance. This makes features more "pose-aware."
  • Siamese Networks for Pose: Leveraging Siamese architectures, which are successful in metric learning, proves effective for learning pose-guided features.
  • Occlusion Robustness: The proposed feature-guided learning framework can be naturally extended to handle severe occlusions by comparing occluded and clean image pairs, achieving impressive results.
  • Batching Matters: The way training samples (pairs) are formed within mini-batches significantly impacts performance, with well-formed batches (combining similar and different pose pairs) yielding better results.

Key Results

  • Outperforms State-of-the-Art: The Siamese Regression Network (both end-to-end and feature-based nearest neighbor) outperforms the state-of-the-art method [25] on both the LINEMOD dataset and their novel occlusion dataset.
  • Improved Accuracy: On the LINEMOD dataset, their end-to-end regression achieves an average Euler angle error of 13.2 degrees, and their features with nearest neighbor achieve 11.4 degrees, compared to 14.5 degrees for [25].
  • Occlusion Handling: On their occlusion dataset, the SRN achieves an average error of 11.8 degrees, significantly outperforming the baseline and even approaching the accuracy of non-occluded objects.
  • Feature Discriminativeness: The learned features are shown to be more discriminative for nearest neighbor matching than those learned by [25].
  • Efficiency: Training one epoch on a Nvidia Titan X takes about 15 minutes for their dataset and 20 seconds for LINEMOD. Inference for one image is very fast (4ms for regression, 6ms for NN).

Limitations and Future Work

  • Overfitting on Small Datasets: The paper notes that on smaller datasets like LINEMOD, the end-to-end regression can be prone to overfitting, with the nearest neighbor approach using learned features performing slightly better.
  • Feature Size Trade-off: While larger feature layers generally improve performance, there's a point of diminishing returns (around 32 units), suggesting a trade-off between feature size and performance.
  • Future Work: The authors suggest extending the network to simultaneously handle object localization and classification, in addition to pose estimation.

Summary updated

Abstract

In this paper we tackle the problem of estimating the 3D pose of object instances, using convolutional neural networks. State of the art methods usually solve the challenging problem of regression in angle space indirectly, focusing on learning discriminative features that are later fed into a separate architecture for 3D pose estimation. In contrast, we propose an end-to-end learning framework for directly regressing object poses by exploiting Siamese Networks. For a given image pair, we enforce a similarity measure between the representation of the sample images in the feature and pose space respectively, that is shown to boost regression performance. Furthermore, we argue that our pose-guided feature learning using our Siamese Regression Network generates more discriminative features that outperform the state of the art. Last, our feature learning formulation provides the ability of learning features that can perform under severe occlusions, demonstrating high performance on our novel hand-object dataset.
BibTeX
@misc{doumanoglou2016siamese,
  author = {Andreas Doumanoglou and Vassileios Balntas and Rigas Kouskouridas and Tae-Kyun Kim},
  title = {{Siamese Regression Networks with Efficient mid-level Feature Extraction for 3D Object Pose Estimation}},
  year = {2016},
  eprint = {1607.02257},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1607.02257},
}
Figure