Appearance Compression and Synthesis based on 3D Model for Mixed Reality

Ko Nishino, Yoichi Sato, Katsushi Ikeuchi

Abstract

This paper presents a sophisticated approach to creating realistic digital twins for mixed reality by combining image-based rendering (IBR) with 3D geometric models. The authors introduce an improved Eigen-Texture method that addresses the alignment challenges between 3D scanners and 2D cameras. By utilizing Mutual Information (MI) for robust alignment and Principal Component Analysis (PCA) for cell-based texture compression, the system enables the high-quality synthesis of complex, non-Lambertian surfaces with significant storage efficiency.

Introduction

In the late 1990s, the field of computer vision faced a dichotomy between Image-Based Rendering (IBR) and Model-Based Rendering (MBR). While IBR excelled at capturing complex visual details, it lacked the geometric structure necessary for realistic interactions like shadow casting. Conversely, MBR provided structural integrity but often relied on oversimplified reflectance models (e.g., Lambertian) that failed to capture the nuanced appearance of real-world materials.

The authors' Eigen-Texture method bridges this gap by projecting 2D image sequences onto 3D meshes. However, the effectiveness of this approach hinges on the precise alignment of geometry and texture. This paper specifically targets the alignment problem, making the system viable for practical applications in cultural heritage and mixed reality.

Methodology: The Eigen-Texture Pipeline

The proposed workflow integrates geometric modeling with appearance compression through four primary stages:

  1. Robust Alignment via Mutual Information: To synchronize a 3D model with 2D images, the authors leverage the "reflectance power" (laser return intensity) from the 3D scanner. By treating this reflectance data as a 2D image and maximizing the Mutual Information (MI) between it and the CCD camera image, the system achieves precise pose estimation even with disparate sensor locations.
  2. Cell-Based Decomposition: The 3D mesh is partitioned into small triangular cells. The appearance of each cell is tracked across a sequence of images as the object rotates, capturing view-dependent variations.
  3. Eigenspace Compression (PCA): Principal Component Analysis is applied to the appearance sequence of each cell. This process generates "Eigen-Textures"—the fundamental visual components that describe how a specific surface patch changes with the viewing angle.
  4. Synthesis and Interpolation: New views are generated by reconstructing cells using stored eigenvectors and interpolating between known poses within the eigenspace.
The complete Eigen-Texture workflow, illustrating the transition from raw 3D models and color images to compressed eigenspace representations and final synthesized views.

Evaluation and Datasets

The method was evaluated using three objects with distinct surface properties:

  • Ceramic Cat: A "shiny" (non-Lambertian) surface with complex specular highlights.
  • Stuffed Raccoon: A "rough" or fuzzy surface that challenges standard reflectance models.
  • Great Buddha of Kamakura: A 13-meter cultural heritage site, demonstrating the method's scalability for large-scale outdoor objects.

Key Results

The system demonstrated high visual fidelity, with synthesized images often indistinguishable from the original inputs. The cell-adaptive dimensionality allowed the system to allocate more data to complex regions (like the cat's highlights) while maintaining high compression for simpler areas.

Synthesized images of the ceramic cat, showing the preservation of complex specular highlights and surface texture.
  • Compression Efficiency: The ceramic cat achieved a compression ratio of 15.3:1. The stuffed raccoon, due to its surface complexity, required more data, resulting in a 5.54:1 ratio.
  • Interpolation Potential: By interpolating between sparse views (e.g., every 12 degrees), the authors suggest that compression ratios could reach up to 80:1 without sacrificing photorealism.

Conclusion and Future Directions

The Eigen-Texture method successfully combines the strengths of MBR and IBR. Future work aims to optimize data storage by moving from float to unsigned char through vector quantization and implementing adaptive cell resolutions to further enhance compression ratios.

Summary updated

Abstract

Rendering photorealistic virtual objects from their real images is one of the main research issues in mixed reality systems. We previously proposed the Eigen-Texture method (K. Nishino et al., 1999), a new rendering method for generating virtual images of objects from their real images to deal with the problems posed by past work in image based methods and model based methods. Eigen-Texture method samples appearances of a real object under various illumination and viewing conditions, and compresses them in the 2D coordinate system defined on the 3D model surface. However, we had a serious limitation in our system, due to the alignment problem of the 3D model and color images. We deal with this limitation by solving the alignment problem; we do this by using the method originally designed by P. Viola (1995). The paper describes the method and reports on how we implement it.
BibTeX
@inproceedings{nishino1999appearance,
  author = {Ko Nishino and Yoichi Sato and Katsushi Ikeuchi},
  title = {{Appearance Compression and Synthesis based on 3D Model for Mixed Reality}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {1999},
  doi = {10.1109/iccv.1999.791195},
}
Figure