TL;DR
LucidLyra is a feed-forward framework that transforms a single image into a "queryable" 3D scene using 3D Gaussian Splatting (3DGS). By distilling knowledge from video diffusion and 2D vision foundation models, it jointly reconstructs geometry and understands semantics and object instances in a single pass, enabling immediate use in robotics, simulation, and 3D editing.
What Problem Does This Paper Address?
Recent generative models can "imagine" a 3D environment from a single photo, but these reconstructions are typically just unstructured clouds of colored primitives. While they look realistic, they are "dumb"—a robot cannot distinguish a chair from the floor, and a designer cannot easily select and move an object.
Existing methods to add "understanding" (semantics and instances) to 3D scenes usually require multiple captured views of a real scene or rely on slow, per-scene optimization processes that can take hours. LucidLyra tackles the challenge of creating structured, queryable 3D worlds from a single image in a fast, feed-forward manner, bridging the gap between generative realism and functional scene understanding.
Methods
LucidLyra builds upon the Lyra architecture, which maps camera-controlled video diffusion latents directly to 3D Gaussians. The authors introduce a multi-task distillation strategy to teach the model about objects and meanings without requiring manual 3D labels.
- The Teacher Branch: The model generates a multi-view video from the input image. It then applies "frozen" 2D foundation models—like SAM/SAM2 for object tracking and SigLIP for open-vocabulary semantics—to these generated frames. This creates a rich set of 2D "labels" across different viewpoints.
- The Student Branch: A 3DGS decoder takes the video latents, input image features (extracted via C-RADIO), and ray encodings to predict Gaussian primitives. Crucially, each Gaussian is assigned not just color and shape, but also a semantic descriptor and instance embeddings.
- Joint Learning: The student is trained to match the teacher's 2D predictions. By consolidating these signals into a persistent 3D representation, the model learns to produce view-consistent 3D segments.
- Dynamic Extension: The framework extends to monocular video, using a time-conditioned decoder to reconstruct moving 3D scenes while maintaining object identities over time.
Datasets
The authors utilize and augment several datasets to train and evaluate LucidLyra:
- Lyra-SDG: A large-scale synthetic dataset of ~59,000 static and ~7,000 dynamic scenes generated from text prompts.
- LucidLyra-SDG & SDG Dynamic: New, manually curated benchmarks (25 scenes each) with human-refined annotations for rigorous evaluation.
- LucidLyra-ADE20K: A benchmark using real-world images from the ADE20K dataset to test how well the model generalizes to real-image conditioning.
- LAION-2B: Used to train the vision-language feature projector.
Key Insights and Ideas
- Video as a Scalable Supervisor: The core "aha moment" is that pre-trained video generators can act as a massive, automated source of multi-view training data. If a model can generate a consistent video of a scene, that video contains all the information needed to distill 3D structure and semantics.
- Feature Compression is Key: Storing raw 1152-dimensional SigLIP features on millions of Gaussians is memory-prohibitive. The authors found that compressing these to 128 dimensions provides the best balance between accuracy and GPU memory limits.
- Joint Training Improves Quality: Interestingly, adding semantic and instance tasks doesn't degrade the visual reconstruction. In fact, it slightly improves it (a 0.53 dB PSNR gain), suggesting that learning "what" an object is helps the model better define "where" it is.
- Open-Vocabulary Flexibility: Because the semantic features are aligned with SigLIP’s text-image space, users can query the generated 3D scene with arbitrary text (e.g., "find the panda," "select the blue towel").
Key Results
- Superior Understanding: On the static SDG benchmark, LucidLyra improved semantic mIoU by 31.7 percentage points and instance tracking (mTrackPR) by 9.8 points over the strongest comparable baselines.
- Efficiency: LucidLyra performs inference in approximately 2.1 minutes on a single A100 GPU. For context, optimization-based methods like LangSplatV2 require nearly 12 hours (697 minutes) for the same task.
- Real-World Transfer: Even when conditioned on real ADE20K images, the model maintains high performance, achieving 49.8 mIoU, outperforming two-stage feed-forward baselines.
- Downstream Utility: The authors successfully demonstrated the model's output in NVIDIA Isaac Sim for robot navigation and manipulation, as well as in interactive 3D scene editing.
Limitations and Future Work
- Clustering Artifacts: While the feature fields are continuous, the final step of grouping Gaussians into discrete objects (using HDBSCAN) can still suffer from occasional errors, especially in very crowded scenes.
- Preprocessing Cost: Generating the "teacher" labels for the massive training corpus is computationally expensive, requiring over 1,500 GPU-days.
- Future Directions: The authors suggest that future work could involve scaling to even larger video foundation models and exploring tighter integration where the generation and reconstruction processes are trained simultaneously from the start.
Summary updated