In-Network View Synthesis for Interactive Multiview Video Systems

Laura Toni, Gene Cheung, Pascal Frossard

TL;DR

This paper proposes a "smart middleman" approach to interactive 3D video streaming: using edge computing nodes (cloudlets) to synthesize new virtual reference views before they reach the user. By strategically creating these intermediate views, the system can bypass last-mile bandwidth bottlenecks and provide a much smoother, higher-quality navigation experience for users exploring 3D scenes.

What Problem Does This Paper Address?

Interactive Multiview Video Systems (IMVS) allow users to "walk through" a 3D scene by switching between different camera angles. To make this smooth and lag-free, the system must provide enough reference views so the user's device can fill in the gaps using Depth-Image-Based Rendering (DIBR).

The challenge is the "last-mile" bottleneck. While the core network might have plenty of capacity, the connection between the local service provider and the end-user is often limited. If you only send a few original camera views, the distance between them becomes too large, leading to ugly visual artifacts (distortion) when the user tries to view a perspective in between. Traditionally, providers just sub-sample the available camera views, but this paper asks: What if we could create better "starting points" for the user right at the edge of the network?

Methods

The authors introduce a hierarchical cloud architecture. A main cloud stores all original camera data, which is then sent to cloudlets—resource-rich proxies located close to the users.

The core innovation is In-Network View Synthesis. Instead of just picking which original camera views to forward, the cloudlet can use its computing power to synthesize new virtual views. These virtual views are then sent to the user as references.

To make this work, the authors:

  1. Model Distortion: They use a Gauss-Markov model to show mathematically why synthesizing an intermediate view at the cloudlet can improve the final image quality at the client.
  2. Optimize Selection: They formulate a "View Selection Problem" to find the best subset of views (a mix of real and synthesized) that minimizes total distortion for a user's expected navigation window, given a strict bandwidth limit.
  3. Algorithm Design: They prove the general problem is NP-hard (similar to the "Set Cover" problem). However, by introducing two logical assumptions—shared optimality (a good reference for one view is likely good for its neighbor) and independence of reference optimality—they develop a highly efficient Dynamic Programming (DP) algorithm that solves the problem in polynomial time.

Datasets

The researchers evaluated their approach using two well-known 3D video datasets:

  • "Statue": A sequence with 51 cameras and a 5.33 mm spacing.
  • "Mansion": A sequence with 51 cameras and a 10 mm spacing. These datasets provide the high-density camera views and depth maps necessary to simulate realistic interactive navigation.

Key Insights and Ideas

  • The Reference Tradeoff: Synthesizing a view at the cloudlet introduces a small amount of "synthesis noise." However, sending this slightly noisy intermediate view to the user is often much better than sending a "perfect" original camera view that is geographically further away from where the user is looking.
  • Staircase Solutions: The optimal selection of views tends to follow a "staircase" pattern in the navigation window. This structure is what allows the complex optimization problem to be broken down into a manageable Dynamic Programming solution.
  • Compensating for Poor Hardware: The authors found that in-network synthesis can actually compensate for poorly positioned cameras or a lower total number of cameras in the original setup, making the system more robust for real-world deployments.

Key Results

  • Quality Gains: The proposed system achieved a peak signal-to-noise ratio (PSNR) improvement of up to 2.1 dB compared to traditional methods that only transmit original camera views.
  • Bandwidth Efficiency: In-network synthesis allows the system to maintain high quality even when the number of views that can be sent to the user is severely restricted (e.g., only 2 or 3 views).
  • Robustness: The DP algorithm consistently matched the performance of an "exhaustive search" (checking every possible combination) but at a fraction of the computational cost.

Limitations and Future Work

  • Computational Overhead: While the DP algorithm is efficient, performing DIBR synthesis for multiple users simultaneously at a cloudlet still requires significant GPU/CPU resources.
  • Depth Map Dependency: Like all DIBR-based systems, the quality is heavily dependent on the accuracy of the depth maps. Errors in depth estimation at the source will propagate through the cloudlet to the user.
  • Future Directions: The authors suggest that future work could involve more complex user behavior models (predicting exactly where a user will move next) to further refine which views the cloudlet chooses to synthesize.

Summary updated

Abstract

In multiview applications, camera views can be used as reference views to synthesize additional virtual viewpoints, allowing users to freely navigate within a 3D scene. However, bandwidth constraints may restrict the number of reference views sent to clients, limiting the quality of the synthesized viewpoints. In this work, we study the problem of in-network reference view synthesis aimed at improving the navigation quality at the clients. We consider a distributed cloud network architecture, where data stored in a main cloud is delivered to end users with the help of cloudlets, i.e., resource-rich proxies close to the users. We argue that, in case of limited bandwidth from the cloudlet to the users, re-sampling at the couldlet the viewpoints of the 3D scene (i.e., synthesizing novel virtual views in the cloudlets to be used as new references to the decoder) is beneficial compared to mere subsampling of the original set of camera views. We therefore cast a new reference view selection problem that seeks the subset of views minimizing the distortion over a view navigation window defined by the user under bandwidth constraints. We prove that the problem is NP-hard, and we propose an effective polynomial time algorithm using dynamic programming to solve the optimization problem under general assumptions that cover most of the multiview scenarios in practice. Simulation results confirm the performance gain offered by virtual view synthesis in the network.
BibTeX
@article{toni2016network,
  author = {Laura Toni and Gene Cheung and Pascal Frossard},
  title = {{In-Network View Synthesis for Interactive Multiview Video Systems}},
  journal = {IEEE Trans. Multimedia},
  year = {2016},
  doi = {10.1109/tmm.2016.2537207},
}

Loading related content...

Figure