<title>Compression of computer graphics images with image-based rendering</title>

Ilmi Yoon, Ulrich NeumannProceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE 1998 · DOI

TL;DR

This paper introduces a novel compression algorithm for synthetic graphics images that leverages image-based rendering (IBR) to achieve significantly higher compression ratios (4-10x over MPEG) than traditional methods. By intelligently predicting motion between frames and only transmitting sparse residual data, it drastically reduces bandwidth requirements, making remote rendering of complex 3D scenes more feasible for low-end clients.

What Problem Does This Paper Address?

The increasing complexity of 3D models (millions of polygons, gigabytes of data) poses a significant challenge for applications requiring remote visualization or transmission over networks, especially for internet-based applications or systems with limited client-side resources. Traditional compression techniques like MPEG, designed for video, struggle with the distinct characteristics of synthetic graphics (e.g., sharp edges, lack of noise) and don't fully exploit the temporal coherence inherent in rendering sequences. The goal is to enable efficient transmission and rendering of high-quality 3D graphics without overwhelming network bandwidth or client processing power.

Methods

The core of the proposed method is an integration of image-based rendering (IBR) with compression. Here's a breakdown:

  • Reference Frames (I-frames): High-quality images of the 3D scene are rendered using a traditional model renderer (RM). These I-frames contain color, depth, and object ID information.
  • Image-Based Rendering (IBR) for Prediction: An IBR system (R₁) takes an I-frame and attempts to generate a new frame by "warping" or re-projecting pixels from the I-frame. This process inherently exploits temporal coherence and provides motion prediction.
  • Confidence-Based Residuals: The IBR system assesses the confidence of its pixel motion estimates.
  • High Confidence: Pixels with high-confidence motion estimates are directly interpolated from the I-frame.
  • Low Confidence: For pixels where motion estimation is uncertain (e.g., occlusions, complex geometry), the model renderer (RM) is invoked to render these specific pixels. The difference between the RM's output and the IBR's prediction forms a sparse residual image.
  • Compression: Both the I-frames and the sparse residual images are compressed. The key insight is that the residual images are extremely sparse, allowing for very high compression ratios using standard entropy coding (like gzip). The paper highlights that the residual data doesn't need explicit address or index data due to the ordered processing.
  • Client-Side Decoding: The remote client receives the compressed I-frames and residual images. It uses its own IBR system to reconstruct the final image by merging the extracted (high-confidence) pixels from the I-frame with the decoded residual pixels.

This approach effectively merges rendering and compression, using IBR to accelerate both the prediction of new frames and the generation of compressed data.

Datasets

The paper evaluates its method on polygon data of varying complexity and volume data. Specific examples include:

  • Brain data (volume)
  • Polygon models (e.g., 897 triangles, 33264 triangles)

The authors generated animations by moving the camera around these objects to test rapid viewpoint changes.

Key Insights and Ideas

  • IBR as a Compression Tool: The paper demonstrates that the motion prediction and coherence extraction capabilities of IBR, typically used for rendering acceleration, can be repurposed for highly effective image compression.
  • Sparse Residuals are Key: The core of the compression gain comes from the sparsity of the residual images. By only transmitting the "difficult" pixels that the IBR couldn't confidently predict, the amount of data to transmit is drastically reduced.
  • Eliminating Address/Index Data: The image-order, pixel-by-pixel processing of residuals allows for their compression without explicit positional information, further shrinking their size.
  • Server-Client Synergy: The approach is ideal for client-server architectures where the server holds the complex 3D model and rendering capabilities, while the client is lightweight, requiring only the IBR for decoding.
  • Lossless Compression of Residuals: While the overall system is lossy due to IBR interpolation, the residual data itself can be compressed losslessly (e.g., with gzip), leading to very high overall compression ratios.

Key Results

  • Compression Ratio: The proposed method achieves compression ratios of 4-10 times better than MPEG-1 for synthetic image sequences. Residual images alone were compressed 10x better than GIF and 100x better than lossless JPG.
  • Rendering Speedup: The integrated image-based rendering and compression approach provides significant speedups. The IBRAC method shows a speedup of 4-7 times in encoding and 7+ times in decoding compared to traditional model rendering.
  • Image Quality: Subjectively, the images showed no visible degradation. Objectively, the images exhibited a better Signal-to-Noise Ratio (SNR) than MPEG-1 encodings.

Limitations and Future Work

Limitations: * Lossy Compression: The overall system is a lossy compression method due to the interpolation and potential inaccuracies in IBR. Diffuse surfaces are handled better than specular ones. * Moving Objects: The current implementation primarily handles moving cameras. Moving objects within the scene are not directly addressed, though the authors suggest they could be handled via compositing separate scenes. * Aliasing: Potential aliasing issues can arise from using sampled geometry and color, though object IDs and depth thresholds are used to mitigate this.

Future Work: * Multiple I-frames: Developing efficient methods to use multiple I-frames, potentially by subdividing images into blocks and selecting the best I-frame for prediction. * Automatic I-frame Optimization: Creating mechanisms to automatically determine the optimal frequency and sequence of I-frames for a given animation script. * P-frame Concept: Exploring the integration of P-frame concepts (similar to MPEG) for further compression gains. * Ray-Tracing Acceleration: Employing ray-tracing acceleration techniques (e.g., spatial subdivision) to further enhance rendering speeds.

Summary updated

Abstract

We present a new compression algorithm for synthetic images that produces high compression rates by utilizing depth and color information from previously rendered images. Images predicted from prior images are combined with a residual image that may be transmitting from a remote location, to generate new images. The image-based rendering technique provides accurate motion prediction and accelerates rendering at the same time by exploiting temporal coherence. The motion prediction is computed and evaluated in image- order, pixel by pixel, producing residual images that are sparse and do not require address or index data. The system yields a compression ratio improvement of a factor of 4 - 10 over MPEG, in many cases. This approach is attractive for remote rendering applications where a client system may be a relatively low-performance machine and limited network bandwidth makes transmission of large 3D data impractical. The efficiency of the server generally increases with scene complexity or data size since the rendering time is predominantly a function of image size. This technique is also applicable to archiving animation.
BibTeX
@inproceedings{yoon1998lt,
  author = {Ilmi Yoon and Ulrich Neumann},
  title = {{\<title\>Compression of computer graphics images with image-based rendering\</title\>}},
  booktitle = {Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE},
  year = {1998},
  doi = {10.1117/12.333824},
}
Figure