A 3d data intensive tele-immersive grid

Benjamin Petit, Thomas Dupeux, Benoît Bossavit, Joeffrey Legaux, Bruno Raffin, Emmanuel Melin, Jean-Sébastien Franco, Ingo Assenmacher, Edmond BoyerACM Multimedia 2010 · DOI

TL;DR

This paper presents a novel tele-immersive system that combines multi-camera 3D modeling with a data grid to enable rich, high-presence virtual collaboration across geographically dispersed locations. The system allows users to interact within a shared virtual environment as realistic 3D avatars, overcoming the limitations of traditional avatar-based interactions.

What Problem Does This Paper Address?

Current networked virtual environments, while enabling remote meetings, often fall short in providing a true sense of presence and facilitating rich human interaction. Users are typically represented by generic avatars controlled by basic input devices, severely limiting their ability to convey body language and engage in nuanced communication. Furthermore, these environments often struggle to handle complex, data-intensive scenarios, which are crucial for applications like distributed scientific research or disaster response coordination. This paper tackles the challenge of creating a tele-immersive system that not only offers a high degree of user presence through realistic 3D avatars but also supports the integration and interaction with large datasets in a distributed manner.

Methods

The core of the system is a multi-camera 3D modeling approach that captures and reconstructs users in real-time, generating textured 3D meshes. This allows for a much higher sense of presence compared to traditional avatars. These 3D models are then integrated into a rich virtual environment rendered by a parallel terrain rendering engine. The system leverages the FlowVR middleware, a hierarchical component model designed for large, distributed interactive applications. This modular approach simplifies development, debugging, and deployment across heterogeneous computing resources and networks. For data handling, the system utilizes a data grid architecture, enabling the sharing and aggregation of large datasets, such as terrain data from LIDAR scans. Interactions are managed through various input devices, including head-mounted displays (HMDs) for immersive visualization and touchpads for intuitive control.

Datasets

The paper mentions the use of real land data from a LIDAR airborne scan for terrain rendering. This dataset is described as consisting of hundreds of tiles, each with approximately 2 million triangles, representing a significant amount of data. No new benchmarks or datasets were introduced in this paper.

Key Insights and Ideas

  • Realistic 3D Avatars Enhance Presence: The use of multi-camera 3D modeling to create realistic, textured 3D avatars significantly improves the sense of presence and enables more natural, body-language-based communication. Users instinctively use their bodies to convey information, leading to more intuitive interactions.
  • Hierarchical Component Model for Scalability: The FlowVR middleware's hierarchical component approach proved effective in managing the complexity of a distributed application involving multiple sites, numerous processes, and heterogeneous networks. This modularity is crucial for collaborative development and deployment.
  • Data Grid for Complex Scenarios: Integrating a data grid allows the system to handle and share large, complex datasets, opening up possibilities for data-intensive collaborative tasks that were previously difficult to achieve in virtual environments.
  • HMDs are Key for Immersion: The head-mounted display (HMD) was identified as a critical component for achieving a strong sense of immersion, making users feel as if remote collaborators are physically present.

Key Results

The system was successfully experimented with across three geographically dispersed cities, demonstrating the feasibility of a rich 3D multimedia environment for remote collaboration. Users reported a high satisfaction with the feel of presence and discussion capabilities. The Grenoble site, with its high-performance network and processing nodes, handled the most data and achieved the lowest latencies, resulting in the best user experience. Latencies were generally acceptable for interactions, though higher latencies were observed at Bordeaux and Orléans due to network and computing limitations. The paper highlights that the system can handle data streams of up to 80 Mbit/s and 240 Mbit/s for 3D modeling, and the terrain server requires bandwidths of around 96 Mbit/s for multiple viewpoints.

Limitations and Future Work

The authors acknowledge several limitations. The 3D modeling of users can sometimes occlude parts of their faces, making eye contact difficult. The current approach to controlling data transfer relies on simple adaptive mechanisms, and more sophisticated optimizations are needed for scaling to a larger number of participants. Future work is suggested towards a service-oriented architecture where the application can dynamically build itself as new services register. Further research into optimizing data transfer strategies and exploring more advanced interaction paradigms is also recommended. The paper also notes that while HMDs pair well with multi-camera 3D modeling, further work is needed to address issues like face occlusion.

Summary updated

BibTeX
@inproceedings{petit2010d,
  author = {Benjamin Petit and Thomas Dupeux and Benoît Bossavit and Joeffrey Legaux and Bruno Raffin and Emmanuel Melin and Jean-Sébastien Franco and Ingo Assenmacher and Edmond Boyer},
  title = {{A 3d data intensive tele-immersive grid}},
  booktitle = {ACM Multimedia},
  year = {2010},
  doi = {10.1145/1873951.1874210},
}
Figure