A Flexible Client-Driven 3DTV System for Real-Time Acquisition, Transmission, and Display of Dynamic Scenes

Xun Cao, Yebin Liu, Qionghai DaiEURASIP J. Adv. Signal Process 2009 · DOI · OpenReview

Abstract

This paper presents a novel client-driven 3DTV system capable of capturing, transmitting, and rendering dynamic 3D scenes in real time over standard IP networks. By integrating a "lengthened-B-field" coding scheme with a "Clarity Degree" rendering metric, the system enables seamless free-viewpoint navigation with bandwidth requirements under 1.2 Mbps per user. The architecture effectively addresses the challenges of massive data volume, view-switching latency, and rendering artifacts common in multi-view video systems.

Introduction

While high-definition 2D streaming is ubiquitous, interactive 3DTV—allowing viewers to change perspectives freely—remains technically challenging. Current systems struggle with three primary hurdles: the "data deluge" from dozens of cameras, the switching lag inherent in standard predictive video codecs, and visual artifacts like ghosting during view synthesis. This research proposes an end-to-end architecture that optimizes the entire pipeline based on the client's active viewpoint, significantly reducing both computational and network overhead.

System Architecture and Methods

The system utilizes an 8 × 8 camera array (64 cameras total) to capture the scene from multiple angles. These cameras are managed by 16 producer PCs that handle initial processing. The core of the system is a streaming server cluster that mediates between the producers and various client displays, including 2D, autostereoscopic, and multi-projector 3D displays.

The end-to-end client-driven 3DTV system architecture, from camera array to various display types.

Novel Coding and Transmission

To eliminate the lag typically associated with switching between camera views, the authors introduced a lengthened-B-field coding structure. Traditional codecs require waiting for a new "I-frame" to switch views. In this proposed setup, "I-fields" (spatial prediction across cameras) and "P-fields" (temporal refresh) provide the backbone, while optional "B-fields" (bi-predictive) allow the server to switch views instantaneously without waiting for a specific sequence completion.

The proposed coding structure featuring (a) I-field spatial prediction and (b) the lengthened-B-field temporal sequence.

Furthermore, the system employs Region of Interest (ROI) streaming. Instead of broadcasting full frames from all 64 cameras, the system identifies the specific pixel blocks required to render the user's current virtual perspective and transmits only that data, drastically lowering bandwidth consumption.

All-Focus Rendering via Clarity Degree

To avoid the computational cost of full 3D reconstruction, the system uses a plane-sweeping algorithm combined with a new metric called Clarity Degree (CD). Based on Mean Change Energy (MCE), the CD metric identifies the sharpest regions across different focal planes. By stitching these clear regions together, the system creates an all-focus 3D image that bypasses the need for perfect depth maps, effectively eliminating "ghosting" effects.

Experimental Results

The system was evaluated using synthetic scenes (teapot, grid) and real-world sequences captured by the 64-camera array. Key performance metrics include:

  • Real-Time Performance: Achieved a rendering rate of 22 fps on standard commodity hardware.
  • Bandwidth Efficiency: Transmission bitrates were maintained below 1.2 Mbps per user, suitable for broadband IP networks.
  • Computational Savings: The proposed rendering scheme reduced computation time by approximately 32% compared to traditional all-focus methods by utilizing motion vectors to approximate depth changes.

Conclusion

The client-driven approach successfully demonstrates that high-quality, interactive 3DTV is viable over existing network infrastructures. By prioritizing data transmission based on the viewer's perspective and using sharpness as a proxy for depth, the system achieves a balance between visual quality and real-time performance. Future work will focus on adapting the system for non-grid, unstructured camera arrays and improving performance for moving camera setups.

Summary updated

Abstract

3D experience and free-viewpoint navigation are expected to be two essential features of next generation television. In this paper, we present a flexible 3DTV system in which multiview video streams are captured, compressed, transmitted, and finally converted to high-quality 3D video in real time. Our system consists of an camera array, 16 producer PCs, a streaming server, multiple clients, and several autostereoscopic displays. The whole system is implemented over IP network to provide multiple users with interactive 2D/3D switching, viewpoint control, and synthesis for dynamic scenes. In our approach, multiple video streams are first captured by a synchronized camera array. Then, we adopt a lengthened-B-field and region of interest- (ROI-) based coding scheme to guarantee a seamless view switching for each user as well as saving per-user transmission bandwidth. Finally, a convenient rendering algorithm is used to synthesize a visually pleasing result by introducing a new metric called Clarity Degree (CD). Experiments on both synthetic and real-world data have verified the feasibility, flexibility, and good performance of our system.
BibTeX
@article{cao2009flexible,
  author = {Xun Cao and Yebin Liu and Qionghai Dai},
  title = {{A Flexible Client-Driven 3DTV System for Real-Time Acquisition, Transmission, and Display of Dynamic Scenes}},
  journal = {EURASIP J. Adv. Signal Process.},
  year = {2009},
  doi = {10.1155/2009/351452},
}
Figure