In Situ 3D Scene Synthesis for Ubiquitous Embodied Interfaces

Haiyan Jiang, Song Leiyu, dongdong weng, Zhe Sun, Li Huiying, Xiaonuo Dongye, Zhenliang Zhang

TL;DR

This paper introduces a novel "scene agent" that synthesizes personalized 3D virtual environments in real-time, directly within a user's physical space. By understanding both the user's spoken desires and the physical surroundings, it creates immersive VR experiences that respect real-world safety constraints and user preferences, bridging the gap between physical and virtual realities.

What Problem Does This Paper Address?

Virtual Reality (VR) offers immersive experiences, but a significant challenge remains: users are still physically present in their real-world environments. This creates a safety risk, as users might collide with physical objects, and it limits immersion, as the virtual world doesn't acknowledge or integrate with the real one. Existing methods often generate entirely separate virtual scenes or simply treat physical objects as impassable obstacles. This paper addresses the need for VR environments that are dynamically generated in situ, meaning they are aware of and adapt to the user's immediate physical surroundings, while also catering to user-specified desires for the virtual scene's appearance and theme.

Methods

The core of the proposed solution is a "scene agent" that leverages Large Language Models (LLMs) to achieve in-situ 3D scene synthesis. The process involves several key steps:

  1. User Demand Inference: An LLM parses user input (e.g., "I want to go to a beach in summer") to extract desired place types, seasons, and specific objects.
  2. Physics-based Inference: The system analyzes the user's physical environment, identifying physical objects and inferring their affordances (e.g., a table can be sat on, a wall is an obstruction). This is crucial for safety and interaction.
  3. Virtual Object-based Inference: For potential virtual objects, the system predicts their likelihood of appearing in specific places and seasons, and their affordances, using LLM-based predictors.
  4. Scene Synthesis: Finally, the scene agent combines the inferred user demands, physical environment constraints, and virtual object properties. It selects a base scene and virtual objects that best match the user's request and the physical context, ensuring affordance consistency and desired style. A key aspect is the use of a size similarity module and an adjusted Intersection over Union (IoU) module to ensure virtual objects are appropriately scaled and positioned relative to physical objects.

Datasets

The authors utilized two datasets for evaluation:

  • SceneNN [27]: A dataset of 18 indoor scenes.
  • ProcTHOR [13]: A dataset of 12 indoor scenes.

These datasets were used to provide the physical room inputs for the scene synthesis process. The experiments involved synthesizing scenes based on user demands for each of these physical environments.

Key Insights and Ideas

  • Bridging Physical and Virtual: The most significant insight is the successful integration of physical environment awareness into virtual scene generation. This moves beyond purely virtual or obstacle-mapped environments towards a more seamless and safer VR experience.
  • LLMs as Contextual Interpreters: The paper demonstrates the power of LLMs not just for text generation, but for complex reasoning tasks like inferring affordances from physical objects and understanding nuanced user preferences for virtual scenes.
  • Affordance as a Safety and Immersion Pillar: By prioritizing the maintenance of physical affordances in synthesized virtual scenes, the system directly addresses safety concerns (e.g., not placing virtual objects where a user might trip) and enhances immersion by allowing users to interact with virtual objects in ways that mirror real-world possibilities.
  • Personalization Beyond Simple Placement: The system goes beyond just placing objects; it aims to match the style and feel of the virtual scene to user requests, leading to a more engaging and personalized experience.

Key Results

The proposed method demonstrated superior performance across several metrics compared to baselines (LLM, Semantics, VRoamer):

  • Affordance Maintenance: The synthesized scenes showed better alignment with the affordances of physical objects.
  • Scene Diversity: The method achieved a more realistic scene diversity, with a similar number of object types to the physical environments.
  • Style Similarity: The synthesized scenes better matched the user's requested style (place, season, objects).
  • 3D IoU: The virtual objects in the synthesized scenes showed a higher degree of overlap with physical objects, indicating better spatial integration.

Quantitative results in Tables 1 and 2 show the proposed method consistently outperforming baselines in metrics like Style Similarity and 3D IoU. A perceptual study also indicated that users found the synthesized scenes to be significantly better aligned with user demands and better at maintaining affordance and style.

Limitations and Future Work

  • LLM Reliance: The performance of the scene agent is inherently tied to the inference capabilities of the LLM. Future work could explore more advanced LLMs for improved accuracy.
  • Text-Only Prompts: Currently, user input is limited to text. The authors suggest that incorporating multimodal prompts (e.g., images) could enhance prediction accuracy.
  • Virtual Object Dataset Size: While 350 virtual objects were used, a larger and more diverse dataset could further improve scene synthesis quality and affordance matching.
  • Virtual Objects for Physical Walls: The paper notes that directly mapping virtual objects to physical walls can be problematic, creating unnecessary obstacles. Future work aims to develop strategies for creating virtual scenes with broader views, especially in limited physical spaces.
  • Extensibility: The authors envision extending the method by incorporating more factors, such as user preferences, to further refine scene synthesis.

Summary updated

Abstract

Virtual reality (VR) provides an interface to access virtual environments anytime and anywhere, allowing us to experience and interact with an immersive virtual world. It has been widely used in various fields, such as entertainment, training, and education. However, the user's body cannot be separated from the physical world. When users are immersed in virtual scenes, they encounter safety and immersion issues caused by physical objects in the surrounding environment. Although virtual scene synthesis has attracted widespread attention, many popular methods are limited to generating purely virtual scenes independent of the physical environment or simply mapping all physical objects as obstacles. To this end, we propose a scene agent that synthesizes situated 3D virtual scenes as a kind of ubiquitous embodied interface in VR for users. The scene agent synthesizes scenes by perceiving the user's physical environment as well as inferring the user's demands. The synthesized scenes maintain the affordances of the physical environment, enabling immersive users to interact with the physical environment and improving the user's sense of security. Meanwhile, the synthesized scenes maintain the style described by the user, improving the user's immersion. The comparison results show that the proposed scene agent can synthesize virtual scenes with better affordance maintenance, scene diversity, style maintenance, and 3D intersection over union (3D IoU) compared to state-of-the-art baseline methods. To the best of our knowledge, this is the first work that achieves in situ scene synthesis with virtual-real affordance consistency and user demand.
BibTeX
@inproceedings{jiang2024situ,
  author = {Haiyan Jiang and Song Leiyu and dongdong weng and Zhe Sun and Li Huiying and Xiaonuo Dongye and Zhenliang Zhang},
  title = {{In Situ 3D Scene Synthesis for Ubiquitous Embodied Interfaces}},
  booktitle = {ACM Multimedia},
  year = {2024},
  url = {https://openreview.net/forum?id=hwieLHhkRr},
}
Figure