Abstract
Remarkable progress has been made in 3D reconstruction of rigid structures from a video or a collection of images. However, it is still challenging to reconstruct nonrigid structures from RGB inputs, due to the under-constrained nature of this problem. While template-based approaches, such as parametric shape models, have achieved great success in terms of modeling the "closed world" of known object categories, their ability to handle the "open-world" of novel object categories and outlier shapes is still limited. In this work, we introduce a template-free approach for 3D shape learning from a single video. It adopts an analysis-by-synthesis strategy that forward-renders object silhouette, optical flow, and pixels intensities to compare against video observations, which generates gradients signals to adjust the camera, shape and motion parameters. Without relying on a category-specific shape template, our method faithfully reconstructs nonrigid 3D structures from videos of human, animals, and objects of unknown classes in the wild.
Introduction
Reconstructing 3D shapes from 2D images is a classic "inverse problem" in computer vision. It is fundamentally under-constrained because many different 3D structures can produce the same 2D projection. To navigate this ambiguity, most state-of-the-art methods rely on category-specific templates. For instance, reconstructing a human typically requires the SMPL model, while a dog requires SMAL.
While effective, this "closed-world" approach fails when encountering novel categories—such as a camel, a bear, or a fictional creature—or individuals with unique features like a dancer wearing flowing garments. LASR (Learning Articulated Shape Reconstruction) aims to move toward an "open-world" reconstruction where the system learns the 3D structure and its articulation directly from video observations without needing a pre-built skeleton or mesh for every new object type.
Methodology
LASR utilizes an analysis-by-synthesis strategy. Instead of directly predicting 3D coordinates, it optimizes a 3D model so that its 2D renderings match the input video as closely as possible. The system processes video inputs through pre-trained networks to extract essential supervision signals.
Technical Components
- Differentiable Rendering: The system forward-renders silhouettes, optical flow, and pixel values. Because the renderer is differentiable, the error between the rendering and the actual video frame can be backpropagated to update the 3D model's parameters.
- Articulation via Linear Blend Skinning (LBS): To handle movement, LASR models the object as a set of rigid "bones." It uses a Parametric Skinning Model based on a mixture of Gaussians to define vertex movement, ensuring smooth, natural deformations.
- Neural Basis for Optimization: Rather than optimizing parameters directly for every frame, LASR uses a ResNet-18 "neural basis" to predict camera and pose parameters, acting as a powerful regularizer.
- Coarse-to-Fine Refinement: The process starts with a simple sphere and iteratively increases mesh complexity to capture fine-grained details.
Key Insights
Unlike many methods that rely on sparse 2D keypoints, LASR treats dense optical flow as a primary supervision signal. Flow provides rich information about surface movement, which is crucial for discovering "bones" without a template. Furthermore, the paper demonstrates that templates can act as "blinders"; for example, a horse template might ignore the unique humps of a camel. LASR’s template-free approach allows it to recover these instance-specific details. To handle the "hidden" side of objects, LASR incorporates a soft-symmetry constraint that encourages the rest shape to be similar to its reflection.
Results and Evaluation
The authors evaluated LASR on challenging datasets including BADJA (real-world animals), AMA (human dancers), and DAVIS (near-rigid objects).
Performance Highlights
- Superior Animal Reconstruction: On the BADJA dataset, LASR significantly outperformed category-specific baselines. For camel sequences, LASR achieved 81.9% accuracy compared to 60.2% for A-CSM (which uses a template).
- Handling the "Unknown": LASR successfully reconstructed a "stone golem" from a video—a task where human-specific models like VIBE and PIFuHD failed because the golem's proportions did not match the human template.
- High Fidelity: Qualitatively, LASR recovers fine details like the silk ribbon of a dancer, which traditional human models often mistake for a limb or ignore entirely.
Limitations
Despite its success, LASR faces several challenges. It struggles to estimate surfaces that are never visible in the video (e.g., the underbelly of an animal). It is also sensitive to heavy occlusions, as it relies on accurate foreground masks and optical flow. Finally, the optimization process is computationally intensive, currently requiring several hours per video on a single GPU.
Conclusion
LASR represents a significant step toward general-purpose 3D reconstruction. By ditching category-specific 3D models in favor of generic motion and shape priors, it achieves high-fidelity reconstructions of diverse objects that traditional template-based methods cannot handle.
Summary updated