Abstract: Presents a new approach to rendering arbitrary views of real-world 3D objects of complex shapes. We propose to represent an object by a sparse set of corresponding 2D views, and to construct any other view as a combination of these reference views. We show that this combination can be linear, assuming proximity of the views, and we suggest how the visibility of constructed points can be determined. Our approach makes it possible to avoid difficult 3D reconstruction, assuming only rendering is required. Moreover, almost no calibration of views is needed. We present preliminary results on real objects, indicating that the approach is feasible.
Introduction
In computer graphics, rendering a real-world object from a novel viewpoint traditionally requires the construction of a complete 3D model. This process, known as 3D reconstruction, is notoriously difficult, often failing when faced with complex geometries, specular surfaces, or intricate textures. This paper, originally presented at ICCV 1995, proposes a "shortcut" that bypasses the 3D modeling bottleneck.
The authors suggest that if the primary goal is visualization rather than geometric analysis, an object can be represented as a sparse collection of 2D reference images. By solving the "correspondence problem"—identifying matching pixels across these views—new viewpoints can be synthesized through linear interpolation. This image-based approach offers a robust alternative to traditional geometry-based rendering, maintaining high visual fidelity with significantly lower computational overhead.
Methodology
The proposed framework for view interpolation relies on three core pillars: parametrization, linear interpolation, and visibility logic.
Each reference image is associated with a parameter $p$, such as a rotation angle on a turntable. When a new view is requested at parameter $p'$, the system calculates the positions of pixels in the new view as a linear combination of their positions in the nearest reference views. This linear approximation holds true when the reference views are sufficiently close.
A critical challenge in view synthesis is handling occlusions—determining which point is visible when multiple points from the reference views project to the same location in the new view. The authors utilize epipolar geometry to establish a depth-ordering heuristic. As illustrated in Figure 1, the geometric relationship between camera centers and object points allows the system to resolve visibility without a full 3D coordinate system. For correspondence acquisition, a modified binocular stereo matcher using dynamic programming is employed to find matching pixel intensities across scanlines.
Experimental Setup
The method was evaluated using two distinct real-world objects captured via a camera and a calibrated turntable:
- Geometric Composition: A scene consisting of a book, styrofoam, a box, and a matchbox, providing clear edges and surfaces.
- Linen Towel: A challenging subject chosen for its non-rigid shape and complex, fine-grained texture, which would be extremely difficult to model using traditional 3D meshes.
Results and Discussion
The authors successfully demonstrated the feasibility of their approach by rendering intermediate views for objects rotated by 5 and 10 degrees.
A key insight from the results is the robustness of view interpolation to matching errors. While errors in 3D reconstruction often manifest as jarring geometric "spikes" or holes, errors in view interpolation are often visually benign. In low-texture regions, a slight mismatch in pixel correspondence still results in a pixel of the correct intensity, making the final image appear acceptable to the human eye. As seen in Figure 4, even with minor artifacts (visible as small stripes in the towel images), the overall appearance of the synthesized views remains high-quality.
Furthermore, the method is "calibration-lite." Unlike 3D modeling, which requires precise camera calibration, this approach only needs enough calibration to assist the stereo matcher. The rendering itself is driven by the arbitrary parameter $p$, simplifying the pipeline.
Conclusion
This research demonstrates that view interpolation is a viable and efficient alternative to 3D reconstruction for object rendering. By treating view synthesis as a linear problem and leveraging image-based correspondences, the authors laid the groundwork for real-time interactive rendering of complex real-world objects.
Future work includes determining the optimal number of reference views needed for complex scenes and expanding the framework from 1D turntable rotations to full 6-degree-of-freedom camera movements. The authors also suggest that implementing these algorithms on multiprocessor systems could achieve the speeds necessary for interactive applications.
Summary updated