Adaptive colour classification for structured light systems

Philipp Fechteler, Peter EisertIET 2009 · DOI

TL;DR

This paper introduces an adaptive color classification method for structured light 3D scanning systems, enabling high-quality, single-shot reconstructions of faces even under challenging lighting conditions and with varying skin reflectances. By adapting the classification to image statistics rather than relying on explicit modeling, the system achieves improved robustness and accuracy.

What Problem Does This Paper Address?

Reconstructing 3D shapes from 2D images is a long-standing goal in computer vision. Structured light systems project patterns onto a scene and use cameras to capture the distortions, which are then used to infer depth. However, existing methods often struggle with variations in ambient lighting, the inherent color properties of the scanned object (like skin tone), and color cross-talk between the projector and camera. These issues can lead to inaccurate reconstructions, especially in non-ideal environments. The paper specifically targets the challenge of achieving robust and high-quality 3D face reconstruction using a single-shot structured light system, which is crucial for capturing dynamic subjects without the need for controlled, dark laboratory settings.

Methods

The core innovation lies in an adaptive color classification approach. Instead of relying on pre-defined color models or explicit calibration for lighting and material properties, the system dynamically adapts its color classification to the statistical characteristics of the captured image. This is achieved using a modified KMeans algorithm (KMeansLineFit).

Here's a breakdown of the key methodological steps:

  • Specialized Low-Level Image Preprocessing: The authors move beyond standard image processing pipelines. They utilize raw photo sensor data (12-bit instead of 8-bit) and apply custom filters (directional hourglass low-pass filter and a 1D band-pass filter) to enhance stripe visibility and reduce noise, specifically tailored for structured light patterns.
  • Sub-Pixel Stripe Localization: To achieve high accuracy, stripes are detected with sub-pixel resolution. This involves fitting parabolas to pixel intensities to pinpoint stripe centers, even when dealing with color cross-talk from different sensor types (RGB).
  • Adaptive Color Classification (KMeansLineFit):
    • The system identifies clusters of pixel colors corresponding to the projected pattern colors.
  • Instead of fitting simple cluster centers, it fits straight 3D lines (prototypes) through these color clusters in RGB space.
  • The KMeans algorithm is adapted to iteratively refine these prototype lines based on the actual data, effectively learning the color statistics of the scene, including distortions caused by lighting and reflectance.
  • A key aspect is the introduction of an artificial constraint that all prototype lines must pass through a common point near black, which is also adapted. This helps to manage the complexity of fitting multiple lines.
  • Robustness is further enhanced by using weighted covariance matrices and Mahalanobis distance for cluster analysis, making the classification less sensitive to outliers and considering correlations between color channels.
  • Probabilistic Color Assignments: The classification produces soft probabilities for each pixel belonging to a particular projected color, rather than hard assignments. This is achieved using a softmax-like function applied to the distances between pixel colors and prototype lines.
  • Stripe Matching: A dynamic programming approach is used to establish correspondences between projected and detected stripes. This process leverages the likelihood of a stripe being valid, its assigned color probability, and its sequential position within the projected pattern to find the most probable matches.

Datasets

The paper does not explicitly mention the use of pre-existing public datasets for training or evaluation. Instead, it relies on experimental data collected using their own custom-built structured light system. The results are demonstrated on scanned faces under various conditions, including "ideal conditions" and "strong environment illumination," as shown in Figure 6.

Key Insights and Ideas

  • Adaptation is Key: The most significant insight is that explicitly modeling complex phenomena like ambient illumination, color cross-talk, and varying skin reflectance is often unnecessary and difficult. Instead, adapting the color classification directly to the observed image statistics provides a more robust and simpler solution.
  • Lines as Prototypes: Using straight lines in RGB space as prototypes for color clusters, rather than simple points, is a novel approach that better captures the elongated nature of color distributions under varying conditions.
  • Single-Shot Advantage: The ability to achieve high-quality 3D reconstructions from a single captured image is a major advantage, enabling the capture of dynamic subjects and simplifying the overall system setup.
  • Raw Data Power: Leveraging raw sensor data (12-bit) and specialized low-level preprocessing unlocks finer details and improves the accuracy of stripe detection and classification.

Key Results

  • The proposed system achieves high-quality 3D reconstructions of faces.
  • It demonstrates robustness to perturbing light conditions and varying skin reflectance characteristics.
  • The adaptive color classifier significantly improves the number of correctly labeled stripe pixels compared to a ratio-based color classification method (Figure 10). Specifically, the KMeansLineFit classifier achieved 98.55% correct classifications, while the ratio-based method achieved 82.57%. Conversely, incorrect classifications were significantly reduced from 17.43% to 1.45%.
  • The system is capable of generating high-speed scans due to its single-shot nature.
  • The KMeansLineFit algorithm converges quickly, typically within less than ten iterations.
  • The resulting 3D scans capture rich detail, as illustrated by the example scan with 126,544 triangles and 63,644 vertices.

Limitations and Future Work

  • Local Adaptation: The authors suggest that adapting the color classifier to local regions of the image, rather than the entire image, could further enhance accuracy.
  • Global Optimization: Considering the whole input image for correspondence calculation, instead of processing each scan column separately, is proposed as a way to improve accuracy.
  • Computational Cost: While the system is described as efficient, the paper notes that the C++ running time for generating a 3D model is around a minute on a 3 GHz Pentium-4 computer, suggesting potential for further optimization through parallel processing.
  • Explicit Modeling: While the adaptive approach avoids explicit modeling, the paper acknowledges that standard color calibration methods would account for prototype line displacements caused by inherent system characteristics. The adaptive method implicitly handles these.

Summary updated

Abstract

The authors present an adaptive colour classification method as well as specialised low-level image processing algorithms. With this approach the authors achieve high-quality 3D reconstructions with a single-shot structured light system without the need of dark laboratory environments. The main focus of the presented work lies in the enhancement of the robustness with respect to environment illumination, colour cross-talk, reflectance characteristics of the scanned face etc. For this purpose the colour classification is made adaptive to the characteristics of the captured image to compensate for such distortions. Further improvements are concerned with enhancing the quality of the resulting 3D models. Therefore the authors replace the typical general-purpose image preprocessing with specialised low-level algorithms performing on raw photo sensor data. The presented system is suitable for generating high-speed scans of moving objects because it relies only on one captured image. Furthermore, due to the adaptive nature of the used colour classifier, it generates high-quality 3D models even under perturbing light conditions.
BibTeX
@inproceedings{fechteler2009adaptive,
  author = {Philipp Fechteler and Peter Eisert},
  title = {{Adaptive colour classification for structured light systems}},
  booktitle = {IET},
  year = {2009},
  doi = {10.1049/iet-cvi.2008.0058},
}
Figure