Registration of 3D and Multispectral Data for the Study of Cultural Heritage Surfaces

Camille Simon Chane, Rainer Schütze, Frank Boochs, Franck Marzani

TL;DR

This paper introduces a novel photogrammetry-based tracking system for precisely registering 3D and multispectral data from featureless cultural heritage objects. By tracking acquisition systems with multiple cameras and a calibrated target frame, they achieve sub-millimeter accuracy, enabling seamless integration of detailed 3D models with rich spectral information for enhanced analysis and preservation.

What Problem Does This Paper Address?

Studying cultural heritage objects often requires combining different types of data, such as 3D shape information and multispectral reflectance properties. While 3D models offer geometric representation and multispectral images provide insights into surface materials (like pigments), integrating these datasets accurately is challenging, especially for objects with smooth or featureless surfaces where traditional point-based registration methods struggle. Existing integrated systems lack flexibility, and separate acquisition systems require robust registration techniques that minimize disturbance to delicate artifacts. This paper aims to bridge this gap by providing a flexible, non-invasive, and accurate method for registering 3D and multispectral data from separate acquisition systems.

Methods

The core of the proposed method is a photogrammetric tracking system. Here's a breakdown:

  • Tracking System: Four "tracking cameras" are used to monitor the position and orientation of the acquisition systems (a 3D scanner and a multispectral camera).
  • Target Frame: A cubic target frame (500 mm side length) equipped with numerous targets is used. The acquisition systems are mounted within this frame.
  • Calibration: Extensive calibration is crucial. This includes:
  • Camera Calibration: Determining the intrinsic parameters (focal length, distortion, etc.) of the tracking cameras.
  • Target Frame Calibration: Precisely mapping the positions of targets on the frame.
  • Acquisition System Calibration: Calibrating the intrinsic parameters of the 3D scanner and multispectral camera.
  • Inter-System Calibration: Determining the transformation between the target frame and each acquisition system.
  • Tracking: The tracking cameras capture images of the target frame and the acquisition systems. Using photogrammetry and bundle adjustment software, the position and orientation of the target frame (and thus the acquisition system) in a world coordinate system are calculated for each acquisition.
  • Registration: The tracked positions and orientations, along with the calibrated system parameters, are used to project all acquired data into a single, unified coordinate system. This allows for the seamless registration of 3D meshes and the projection of multispectral images onto the 3D model.

The novelty lies in using photogrammetry to track the acquisition systems themselves rather than relying on feature matching on the object, making it suitable for featureless surfaces.

Datasets

The paper does not introduce new datasets for training or evaluation. Instead, it uses a specific experimental setup to test its registration method. The "dataset" in this context refers to the acquired 3D scans and multispectral images of a framed cross-stitch canvas. This object was chosen because it presents a mix of smooth (frame) and textured (canvas) surfaces, along with variations in reflectance.

Key Insights and Ideas

  • Featureless Registration is Achievable: The primary insight is that accurate registration of smooth or featureless surfaces is possible by tracking the acquisition system's pose rather than relying on object features. This opens up new possibilities for analyzing objects that were previously difficult to register.
  • Photogrammetry as a Flexible Tracker: The study demonstrates the power and flexibility of photogrammetry for tracking acquisition systems in situ, offering a transportable and potentially cost-effective alternative to more rigid industrial solutions.
  • Accuracy is Key: The paper emphasizes the critical role of meticulous calibration at multiple levels (cameras, target frame, acquisition systems, and their interrelationships) for achieving high registration accuracy.
  • Multimodal Integration is Crucial for Cultural Heritage: The work underscores the value of combining 3D geometry with spectral information for a more comprehensive understanding and documentation of cultural heritage artifacts.

Key Results

The system achieved impressive accuracy:

  • Tracking Accuracy: Better than 0.03 mm spatially and 0.150 mrad angularly.
  • 3D Registration Accuracy: Theoretical spatial accuracy of 0.045 mm, well below the target of 0.09 mm.
  • Multispectral Registration Accuracy: A horizontal inaccuracy of approximately 0.8 mm was observed between the multispectral data and the 3D model. The theoretical spatial accuracy for this step was calculated at 0.950 mm, largely attributed to the orientation accuracy between the target frame and the multispectral camera.

Limitations and Future Work

  • Multispectral Registration Accuracy: The paper acknowledges that the registration accuracy of multispectral data onto the 3D model could be significantly improved. This is primarily due to inaccuracies in the orientation between the target frame and the multispectral camera, and potentially issues with the camera's interior orientation calibration (e.g., focal length and principle point offset).
  • Setup Cumbersomeness: The current setup, while flexible, is described as "cumbersome," requiring multiple calibrations and a well-organized team for efficient operation, especially for highly 3D objects where more tracking cameras might be needed.
  • Software Integration: The authors suggest the need for an independent and integrated software tool for data processing and display to broaden the technique's applicability.
  • Future Work:
    • Improving the calibration between the multispectral camera and the target frame.
  • Investigating the impact of camera interior orientation accuracy on multispectral registration.
  • Developing more integrated software solutions.
  • Exploring the use of lighter materials for the target frame and camera supports to enhance transportability.
  • Potentially using thermal sensors or other optical sensors for complementary data.

Summary updated

Abstract

We present a technique for the multi-sensor registration of featureless datasets based on the photogrammetric tracking of the acquisition systems in use. This method is developed for the in situ study of cultural heritage objects and is tested by digitizing a small canvas successively with a 3D digitization system and a multispectral camera while simultaneously tracking the acquisition systems with four cameras and using a cubic target frame with a side length of 500 mm. The achieved tracking accuracy is better than 0.03 mm spatially and 0.150 mrad angularly. This allows us to seamlessly register the 3D acquisitions and to project the multispectral acquisitions on the 3D model.
BibTeX
@inproceedings{chane2013registration,
  author = {Camille Simon Chane and Rainer Schütze and Frank Boochs and Franck Marzani},
  title = {{Registration of 3D and Multispectral Data for the Study of Cultural Heritage Surfaces}},
  booktitle = {Sensors},
  year = {2013},
  url = {https://openreview.net/forum?id=YuHkE8fK0Z},
}

Loading related content...

Figure