Abstract
MS2Mesh-XR is a novel pipeline designed to facilitate the creation of high-fidelity 3D assets within Extended Reality (XR) environments. By integrating mid-air sketching with verbal commands, the system leverages generative AI—specifically ControlNet and the Convolutional Reconstruction Model (CRM)—to transform coarse user inputs into detailed, textured 3D meshes. The entire process completes in under 20 seconds, significantly reducing the technical expertise and time required for 3D modeling in virtual and mixed reality.
Introduction
Creating 3D content for XR is traditionally a complex task requiring specialized modeling skills and substantial time. While existing VR sketching tools allow for spatial input, they often suffer from a "skill gap," where users struggle to draw complex shapes, and "precision issues" caused by the jittery nature of mid-air hand movements. Conversely, text-to-3D AI solutions often lack the fine-grained spatial control needed to specify exact dimensions or curves.
MS2Mesh-XR addresses these challenges by combining two complementary modalities: sketches for spatial intent and voice for semantic detail. This approach allows users to define the basic structure of an object through a simple drawing while using natural language to describe textures, styles, and specific features.
Methods
The MS2Mesh-XR pipeline is divided into four distinct stages that bridge the gap between rough user input and a final 3D asset.
- Multi-modal Input: Users interact with a virtual canvas using hand-tracking (via MRTK3) to create a 2D sketch. Simultaneously, they provide verbal descriptions transcribed through the Meta Voice SDK.
- Image Inference: To generate high-quality 3D geometry, the system first creates a 2D reference image. It employs ControlNet 1.1 (using Scribble, Canny, and IP2P models) to interpret the sketch's outlines while applying the textures and lighting suggested by the text prompt.
- Mesh Reconstruction: The generated 2D image is processed by the Convolutional Reconstruction Model (CRM). This model produces orthographic views and canonical coordinate maps (CCMs), which are then decoded into Signed Distance Function (SDF) values. A dual marching cubes algorithm finally extracts the textured 3D mesh.
- XR Integration: The final mesh is sent from a high-performance backend server to the Meta Quest 3 headset via HTTP, where it is imported at runtime for immediate user interaction.
Key Insights and Ideas
A primary innovation of MS2Mesh-XR is the use of 2D as a latent space for 3D. Because 2D diffusion models like Stable Diffusion are currently more mature and capable of higher photorealism than direct 3D generation models, using an intermediate 2D image allows the system to inherit superior texture and detail. Furthermore, the researchers found that sketching and voice are complementary modalities: sketches provide the necessary structure that text lacks, while voice provides the stylistic nuance that rough sketches cannot convey.
Key Results
The system demonstrates both efficiency and versatility in 3D asset generation.
- Performance: The average end-to-end generation time is approximately 16.22 seconds, with image inference taking ~3.8s and mesh reconstruction taking ~12.4s.
- Qualitative Success: The pipeline successfully generates a wide variety of objects, such as "Two-story houses," "Flower pattern chairs," and "British Shorthair" cats, while strictly adhering to the user's initial spatial silhouette.
- Application Scenarios: The authors showcased practical utility in VR asset creation (e.g., generating interactive environment items) and MR interior design (e.g., placing virtual furniture based on physical room sketches).
Limitations and Future Work
Despite its strengths, MS2Mesh-XR has limitations. Currently, the color information in the user's sketch is often overridden by the text prompt due to ControlNet constraints. Additionally, the system requires a powerful external GPU (e.g., RTX 4090) for inference, as standalone mobile XR hardware cannot yet handle the computational load. Future work will focus on improving color mapping accuracy and optimizing the pipeline for potential local execution on mobile headsets.
Summary updated