3DPoseLite: A Compact 3D Pose Estimation Using Node Embeddings

Meghal Dani, Karan Narain, Ramya HebbalaguppeWACV 2021

TL;DR

This paper introduces 3DPoseLite, a novel approach for efficient 3D object pose estimation that leverages graph representations of 3D shapes and node embeddings. By representing complex 3D models as compact graphs, 3DPoseLite achieves significant reductions in computational time and memory footprint compared to state-of-the-art methods, making it suitable for on-device AR applications.

What Problem Does This Paper Address?

Accurate 3D pose estimation is crucial for many computer vision applications, particularly in Augmented Reality (AR), autonomous navigation, and robotics. These applications often require real-time performance and efficient resource utilization, especially when running on mobile or embedded devices. Existing methods for 3D pose estimation, while effective, can be computationally expensive and memory-intensive. They often rely on complex 3D representations like point clouds or multi-view images, which are cumbersome to process. The challenge is to develop a pose estimation method that is both accurate and highly efficient, enabling its deployment on resource-constrained platforms without sacrificing performance.

Methods

The core innovation of 3DPoseLite lies in its novel representation of 3D shapes and its subsequent processing. Instead of using dense point clouds or multi-view images, the authors represent 3D models as graphs. This is achieved by first voxelizing the 3D meshes and then converting them into skeletons using thinning algorithms. These skeletons are then transformed into graphs where nodes represent key points and edges represent their connectivity.

To make these graphs amenable to deep learning, the authors employ node2vec, a technique inspired by word embeddings, to learn low-dimensional node embeddings. This unsupervised learning process captures the structural and semantic information of the graph. These node embeddings are then converted into 2D image-like structures (histograms) which can be processed by a standard 2D Convolutional Neural Network (CNN).

The overall architecture of 3DPoseLite involves two parallel pipelines: one for processing the graph representation of the 3D shape and another for processing the input RGB image using a ResNet-18 model. The features extracted from both pipelines are fused using an early fusion strategy and then fed into a fully connected network for final pose estimation. The pose is outputted as azimuth, elevation, and in-plane rotation angles, with a loss function that combines cross-entropy for classification and Smooth L1 loss for regression.

Datasets

The primary dataset used for training and evaluation is Pascal3D+ [48]. This dataset comprises images with annotations for 12 common object categories and includes corresponding 3D CAD models. The authors also evaluate their model on the Pix3D dataset [41] to demonstrate its performance on unseen data.

Key Insights and Ideas

  • Graphs as Compact Shape Representations: The paper highlights that complex 3D shapes can be effectively represented by graphs derived from skeletons, offering a more concise and computationally efficient alternative to point clouds or multi-view representations.
  • Node Embeddings for Graph Understanding: Leveraging node2vec for learning node embeddings is a key insight, allowing the model to capture rich structural information from graphs in a way that is easily digestible by CNNs.
  • Category-Agnostic Approach: By using graph representations and node embeddings, the method aims to be category-agnostic, meaning it can potentially estimate poses for objects it hasn't been explicitly trained on, a significant advantage for real-world applications.
  • Early Fusion for Multimodal Information: Combining features from the 3D shape representation (graph-based) and the 2D image early in the network architecture proves effective for pose estimation.

Key Results

3DPoseLite achieves comparable accuracy to the state-of-the-art method PoseFromShape [50] on the Pascal3D+ dataset. Crucially, it offers significant improvements in efficiency:

  • Computational Time: Reduced by approximately 3x compared to the baseline.
  • Memory Footprint: Reduced by a factor of 11x.
  • Number of Parameters: The graph feature extraction module requires 738,112 parameters, which is about 15x less than the baseline's requirement of 11,307,840 parameters.

The paper also demonstrates the model's robustness on unseen data (Armadillo object) and its ability to generalize to different object categories.

Limitations and Future Work

The authors acknowledge that while their approach is competitive, there are situations where it might struggle:

  • Multiple Objects of Similar Category: Distinguishing between multiple instances of the same object category in an image.
  • Occlusion and Camouflage: Difficulty in detecting objects that are heavily occluded or blend into the background.
  • Invisible Geometry: Scenarios where significant parts of the object's geometry are not visible in the image.

For future work, the authors plan to explore further compression techniques for neural networks, such as pruning graphs and optimizing feature extractor modules. They also aim to develop real-time applications using frameworks like ONNX and TensorFlowLite to enhance user experience in AR.

Summary updated

BibTeX
@inproceedings{dani2021dposelite,
  author = {Meghal Dani and Karan Narain and Ramya Hebbalaguppe},
  title = {{3DPoseLite: A Compact 3D Pose Estimation Using Node Embeddings}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure