Modeling from Reality

Katsushi Ikeuchi, Yoichi SatoOpenalex Unknown 2001 · DOI · OpenAlex

Abstract: This book summarizes the results of our modeling-from-reality (MFR) project which took place over the last decade or so. The goal of this project is to develop techniques for modeling real objects and/or environments into geometric and photometric models through computer vision techniques. By developing such techniques, time consuming modeling process, currently undertaken by human programmers, can be (semi-)automatically performed, and, as a result, we can drastically shorten the developing time of such virtual reality systems, reduce their developing cost, and widen their application areas. Originally, we began to develop geometric modeling techniques that acquire shape information of objects/environments for object recognition. Soon, this effort evolved into an independent modeling project, virtual-reality modeling, with the inclusion of photometric modeling aspects that acquire appearance information, such as color, texture, and smoothness. Over the course of this development, it became apparent that environmental modeling techniques were necessary when applying our techniques to mixed realities that seamlessly combine generated virtual models with other real/virtual images. The material in his book covers these aspects of development.

Introduction

In the early 2000s, the creation of assets for virtual reality (VR) and mixed reality (MR) environments faced a significant bottleneck. Human programmers were required to manually craft 3D models, a process that was not only slow and expensive but often failed to capture the subtle nuances and "realness" of physical objects. The Modeling from Reality (MFR) project was established to automate this pipeline, leveraging computer vision to "scan" physical objects and their environments to create high-fidelity digital twins.

To achieve a perfect digital representation, the MFR framework addresses three distinct challenges: geometric modeling (shape), photometric modeling (appearance), and environmental modeling (lighting context).

The Three Pillars of MFR

1. Geometric Modeling

The geometric pipeline transforms raw sensor data into usable 3D meshes through a three-step process: Mesh Generation, Alignment, and Merging. For objects with planar faces, the project utilizes a weighted least squares approach to align and merge multiple views simultaneously, minimizing error accumulation.

For complex, free-form objects—such as a human hand or a toy—the framework employs Spherical Attribute Images (SAI). This technique maps surface attributes like curvature and color onto a unit sphere, allowing the system to find correspondences between different viewing angles. To handle the inherent noise in sensor data, the Consensus Surface algorithm was developed, using an implicit-surface representation to find agreement among multiple observations.

2. Photometric Modeling

Capturing how a surface reflects light is critical for realism. The MFR project uses two primary strategies:

  • Model-based Rendering: This applies the Torrance-Sparrow reflection model to analytically separate "body reflection" (diffuse color) from "surface reflection" (specular highlights).
  • Eigen-Texture Rendering: For complex materials like fur or textiles that defy simple physics models, a data-driven approach is used. By capturing a sequence of images and applying Principal Component Analysis (PCA), the system compresses the object's appearance into a small set of "eigen-textures," allowing for efficient and realistic rendering from any angle.

3. Environmental Modeling

To ensure virtual objects do not look "disconnected" when placed in new scenes, the system must capture the original lighting environment. The Direct Method uses specialized fish-eye camera rigs to build a 3D radiance map of a room. Conversely, the Indirect Method estimates light distribution by analyzing the shadows cast by a known object within a single image, effectively turning shadows into a data source for light intensity and location.

Case Study: The Great Buddha of Kamakura

The primary "grand challenge" for the MFR project was the digitization of the Great Buddha of Kamakura. This massive cultural heritage site served as the ultimate test for scaling these techniques from small laboratory objects to large-scale monuments. The project successfully created a high-resolution digital archive of the statue and virtually reconstructed the 12th-century temple hall that originally housed it, demonstrating the potential for a "Digital Museum" concept.

Key Insights and Results

The MFR project demonstrated that geometry alone is insufficient for high-fidelity VR; a unified approach including reflectance and environmental context is mandatory. Key technical achievements include:

  • Efficiency: The Eigen-texture method can synthesize 360 different views using only 8 compressed images.
  • Robustness: The Consensus Surface algorithm proved significantly more resilient to sensor noise than standard marching-cubes approaches.
  • Scalability: The transition from modeling small toys (like a rubber duck) to massive cultural monuments proved the framework's versatility.

Conclusion and Future Directions

While the MFR project made significant strides, challenges remain regarding highly transparent or refractive objects and the hardware requirements for direct environmental modeling. Future work points toward a massive initiative to digitize cultural heritage on a national scale, ensuring that physical history is preserved in the digital realm with absolute fidelity.

Summary updated

Abstract

This book summarizes the results of our modeling-from-reality (MFR) project which took place over the last decade or so. The goal of this project is to develop techniques for modeling real objects and/or environments into geometric and photometric models through computer vision techniques. By developing such techniques, time consuming modeling process, currently un­ dertaken by human programmers, can be (semi-)automatically performed, and, as a result, we can drastically shorten the developing time of such virtual reality systems, reduce their developing cost, and widen their application areas. Originally, we began to develop geometric modeling techniques that acquire shape information of objects/environments for object recognition. Soon, this effort evolved into an independent modeling project, virtual-reality modeling, with the inclusion of photometric modeling aspects that acquire appearance information, such as color, texture, and smoothness. Over the course of this development, it became apparent that environmental modeling techniques were necessary when applying our techniques to mixed realities that seamlessly combine generated virtual models with other real/virtual images. The material in his book covers these aspects of development
BibTeX
@inproceedings{ikeuchi2001modeling,
  author = {Katsushi Ikeuchi and Yoichi Sato},
  title = {{Modeling from Reality}},
  booktitle = {Openalex Unknown},
  year = {2001},
  doi = {10.1007/978-1-4615-0797-0},
}
Figure