In-Network View Synthesis for Interactive Multiview Video Systems

Laura Toni, Gene Cheung, Pascal Frossard

Abstract

To enable Interactive multiview video systems with a minimum view-switching delay, multiple camera views are sent to the users, which are used as reference images to synthesize additional virtual views via depth-image-based rendering. In practice, bandwidth constraints may however restrict the number of reference views sent to clients per time unit, which may in turn limit the quality of the synthesized viewpoints. We argue that the reference view selection should ideally be performed close to the users, and we study the problem of in-network reference view synthesis such that the navigation quality is maximized at the clients. We consider a distributed cloud network architecture where data stored in a main cloud is delivered to end users with the help of cloudlets, i.e., resource-rich proxies close to the users. In order to satisfy last-hop bandwidth constraints from the cloudlet to the users, a cloudlet re-samples viewpoints of the 3D scene into a discrete set of views (combination of received camera views and virtual views synthesized) to be used as reference for the synthesis of additional virtual views at the client. This in-network synthesis leads to better viewpoint sampling given a bandwidth constraint compared to simple selection of camera views, but it may however carry a distortion penalty in the cloudlet-synthesized reference views. We therefore cast a new reference view selection problem where the best subset of views is defined as the one minimizing the distortion over a view navigation window defined by the user under some transmission bandwidth constraints. We show that the view selection problem is NP-hard, and propose an effective polynomial time algorithm using dynamic programming to solve the optimization problem. Simulation results finally confirm the performance gain offered by virtual view synthesis in the network.

Introduction: The Last-Mile Bottleneck

Interactive Multiview Video Systems (IMVS) allow users to navigate through a 3D scene by switching between different camera perspectives. To ensure a smooth, lag-free experience, the system must provide enough reference views so that the user's device can fill in the gaps using Depth-Image-Based Rendering (DIBR).

The primary challenge is the "last-mile" bottleneck. While core networks often have high capacity, the connection between local service providers and end-users is frequently limited. If only a few original camera views are sent, the distance between them becomes too large, leading to significant visual artifacts and distortion when the user attempts to view an intermediate perspective. Traditionally, providers sub-sample available camera views, but this paper proposes a more intelligent approach: creating better "starting points" for the user right at the edge of the network.

Proposed System Architecture

The authors introduce a hierarchical cloud architecture designed to optimize data delivery. In this model, a Main Cloud stores all original camera data. This data is transmitted through the core network to cloudlets—resource-rich proxies located close to the end-users.

Figure 1: The proposed hierarchical cloud architecture, showing the flow from the Main Cloud through the Core Network to Cloudlets and finally to end users.

The core innovation is In-Network View Synthesis. Instead of merely selecting which original camera views to forward, the cloudlet uses its computing power to synthesize new virtual views. These virtual views are then sent to the user as references. While synthesizing a view at the cloudlet introduces a small amount of "synthesis noise," sending this slightly noisy intermediate view is often far superior to sending a "perfect" original camera view that is geographically further from the user's current perspective.

Optimization and Algorithm Design

To implement this effectively, the authors formulated a "View Selection Problem" to find the best subset of views (a mix of real and synthesized) that minimizes total distortion for a user's expected navigation window, given a strict bandwidth limit.

Mathematically, the authors used a Gauss-Markov model to demonstrate the quality trade-offs. They proved that the general selection problem is NP-hard. However, by identifying two logical properties—shared optimality (a good reference for one view is likely good for its neighbor) and independence of reference optimality—they discovered that the optimal selection of views follows a predictable "staircase" pattern.

Figure 5: Visualization of the view selection problem, illustrating the staircase pattern that allows for an efficient Dynamic Programming solution.

Leveraging this structure, the researchers developed a highly efficient Dynamic Programming (DP) algorithm. This algorithm solves the selection problem in polynomial time, matching the performance of exhaustive searches at a fraction of the computational cost.

Key Results and Performance

The proposed system was evaluated using two well-known 3D video datasets: "Statue" (51 cameras, 5.33 mm spacing) and "Mansion" (51 cameras, 10 mm spacing). The results demonstrated significant improvements over traditional methods:

  • Quality Gains: The system achieved a peak signal-to-noise ratio (PSNR) improvement of up to 2.1 dB compared to methods that only transmit original camera views.
  • Bandwidth Efficiency: In-network synthesis maintained high quality even when the number of views sent to the user was severely restricted (e.g., only 2 or 3 views).
  • Robustness: The approach successfully compensated for poorly positioned cameras or lower total camera counts in the original setup, making it viable for real-world deployments.

Conclusion

By utilizing edge computing for in-network view synthesis, IMVS can overcome the limitations of last-mile bandwidth. While the approach requires significant computational resources at the cloudlet and depends on depth map accuracy, the resulting gains in navigation quality and bandwidth efficiency are substantial. Future work will explore more complex user behavior models to further refine view selection in real-time.

Summary updated

Abstract

To enable Interactive multiview video systems with a minimum view-switching delay, multiple camera views are sent to the users, which are used as reference images to synthesize additional virtual views via depth-image-based rendering. In practice, bandwidth constraints may however restrict the number of reference views sent to clients per time unit, which may in turn limit the quality of the synthesized viewpoints. We argue that the reference view selection should ideally be performed close to the users, and we study the problem of in-network reference view synthesis such that the navigation quality is maximized at the clients. We consider a distributed cloud network architecture where data stored in a main cloud is delivered to end users with the help of cloudlets, i.e., resource-rich proxies close to the users. In order to satisfy last-hop bandwidth constraints from the cloudlet to the users, a cloudlet re-samples viewpoints of the 3D scene into a discrete set of views (combination of received camera views and virtual views synthesized) to be used as reference for the synthesis of additional virtual views at the client. This in-network synthesis leads to better viewpoint sampling given a bandwidth constraint compared to simple selection of camera views, but it may however carry a distortion penalty in the cloudlet-synthesized reference views. We therefore cast a new reference view selection problem where the best subset of views is defined as the one minimizing the distortion over a view navigation window defined by the user under some transmission bandwidth constraints. We show that the view selection problem is NP-hard, and propose an effective polynomial time algorithm using dynamic programming to solve the optimization problem. Simulation results finally confirm the performance gain offered by virtual view synthesis in the network.
BibTeX
@article{toni2016network,
  author = {Laura Toni and Gene Cheung and Pascal Frossard},
  title = {{In-Network View Synthesis for Interactive Multiview Video Systems}},
  journal = {IEEE Trans. Multimedia},
  year = {2016},
  doi = {10.1109/tmm.2016.2537207},
}
Figure