Integration of quantitative and qualitative techniques for deformable model fitting from orthographic, perspective, and stereo projections

Dimitris N. Metaxas, Sven J. DickinsonICCV 1993 · OpenReview

TL;DR

This paper introduces a novel two-stage approach to 3D object shape recovery by combining qualitative shape recognition with quantitative physics-based deformable model fitting. This integration allows for more robust and accurate shape estimation, even from challenging image data like occluded contours and various projection types (orthographic, perspective, stereo).

What Problem Does This Paper Address?

Recovering the 3D shape of objects from 2D images is a fundamental challenge in computer vision. Traditional methods often fall into two camps: those that focus on qualitative shape descriptions (like Biederman's geons) for recognition but lack precise geometric detail, and those that use physics-based deformable models for accurate shape fitting but are highly sensitive to initial segmentation and model placement, especially with occlusions. This paper bridges this gap by leveraging the strengths of both approaches. It aims to overcome the limitations of purely data-driven deformable models by using qualitative shape analysis to provide strong, informed constraints for the fitting process, thereby improving robustness and accuracy across different imaging conditions.

Methods

The core of the proposed method is a two-step process:

  1. Qualitative Shape Recovery: This initial stage uses techniques to identify and represent the object's coarse shape in terms of its constituent volumetric primitives (geons). This involves:
  2. Face Recovery: Segmenting the image into regions and classifying their bounding contours to identify potential faces.
  3. Aspect Recovery: Matching these faces to a pre-defined "aspect hierarchy" which represents different views of primitives. This hierarchy is crucial for handling occlusions.
  4. Primitive Recovery: Inferring the underlying 3D volumetric primitives (like blocks, cylinders, cones) that form the object based on the recovered aspects. This stage provides a qualitative understanding of the object's structure and the visible parts.
  5. Quantitative Shape Recovery (Deformable Model Fitting): Once the qualitative shape is understood, a physics-based deformable model (specifically, deformable superquadrics) is fitted to the image data. The key innovation here is how the qualitative information is used:
  6. Strong Constraints: The identified primitives and their qualitative orientations (derived from aspects) provide strong initial constraints for the deformable model. This significantly reduces the search space and improves robustness against poor initial guesses.
  7. Occlusion Handling: The aspect hierarchy's ability to represent occluded parts allows the system to selectively use only the visible contour data for fitting, avoiding errors caused by occluding boundaries.
  8. Projection Generalization: The paper extends previous work to handle orthographic, perspective, and stereo projections, making the approach more broadly applicable. This involves deriving specific Jacobian matrices and force calculations for each projection type.

The quantitative fitting itself uses a physics-based approach, treating the deformable model as a dynamic system with mass, damping, and strain energy, driven by forces derived from image data.

Datasets

The paper uses real images for demonstration, specifically:

  • A real image of a toy table lamp.
  • Images of blocks on a table.
  • Stereo pair images of a cylinder.

The paper does not introduce new datasets but rather demonstrates its approach on existing types of imagery.

Key Insights and Ideas

  • Decoupling Recognition and Localization: The paper champions the idea that object recognition (what is it?) can be decoupled from precise localization and shape recovery (where is it and what are its exact dimensions?). Qualitative shape provides sufficient information for recognition, while quantitative methods can then refine the shape.
  • Qualitative as a "Smart Prior": Using qualitative shape information (like identified primitives and their likely orientations) acts as a powerful "prior" for physics-based deformable models. This significantly reduces ambiguity and improves the robustness of the fitting process, especially in the presence of noise or partial views.
  • Aspect Hierarchy for Occlusion: The hierarchical representation of aspects is a clever way to manage uncertainty and occlusion. It allows the system to infer partial information about primitives even when they are not fully visible.
  • Generalizing Projection Models: Extending the deformable model fitting to handle orthographic, perspective, and stereo projections makes the approach more practical for real-world scenarios where camera parameters and viewing conditions vary.

Key Results

The paper demonstrates the effectiveness of its approach through qualitative results shown in figures:

  • Lamp Shade Fitting: The method successfully fits a truncated cone model to a lamp shade, showing how forces are applied to deform a cylinder into the correct shape based on image contours.
  • Block Fitting: The approach recovers and fits models to three blocks from an image, illustrating the primitive recovery and fitting process.
  • Cylinder Stereo Fitting: The method successfully unifies left and right stereo images to determine the scale and depth of a cylinder, demonstrating its capability with stereo data.

While quantitative metrics like accuracy percentages are not explicitly detailed, the visual results showcase the method's ability to recover plausible 3D shapes from challenging 2D images.

Limitations and Future Work

  • Manual Segmentation Dependency (Initial Stage): While the paper aims to reduce reliance on manual segmentation, the initial qualitative shape recovery still relies on image segmentation and edge detection, which can be sensitive to image quality and noise.
  • Correspondence Problem in Stereo: The paper mentions that matching qualitative primitives from multiple images (especially for stereo) is a complex problem and is "beyond the scope of this paper." This suggests that robust stereo correspondence remains an area for future work.
  • Limited Primitive Set: The approach is based on a predefined set of volumetric primitives. Recovering objects composed of more complex or novel shapes might require extending this primitive vocabulary.
  • Computational Cost: Physics-based deformable model fitting can be computationally intensive, although the paper mentions using a first-order Euler method for faster interactive response.

Future work could focus on improving the robustness of the initial segmentation, developing more sophisticated methods for handling the correspondence problem in multi-view scenarios, and expanding the library of recoverable primitives.

Summary updated

Abstract

The authors synthesize a new approach to 3-D object shape recovery by integrating qualitative shape recovery techniques and quantitative physics-based shape estimation techniques. They first use qualitative shape recovery and recognition techniques to provide strong fitting constraints on physics-based deformable model recovery techniques. Previously developed techniques of fitting deformable models to occluding image contours are then extended to the case of image data captured under general orthographic, perspective, and stereo projections. Experimental results are presented to illustrate the shape recovery approach.<
BibTeX
@inproceedings{metaxas1993integration,
  author = {Dimitris N. Metaxas and Sven J. Dickinson},
  title = {{Integration of quantitative and qualitative techniques for deformable model fitting from orthographic, perspective, and stereo projections}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {1993},
  url = {https://openreview.net/forum?id=SkVsEWG_br},
}
Figure