Abstract
We present a method for reconstructing articulated 3D models from videos in real-time, without test-time optimization or manual 3D supervision at training time. Prior work often relies on pre-built deformable models (e.g. SMAL/SMPL), or slow per-scene optimization through differentiable rendering (e.g. dynamic NeRFs). Such methods fail to support arbitrary object categories, or are unsuitable for real-time applications. To address the challenge of collecting large-scale 3D training data for arbitrary deformable object categories, our key insight is to use off-the-shelf video-based dynamic NeRFs as 3D supervision to train a fast feed-forward network, turning 3D shape and motion prediction into a supervised distillation task. Our temporal-aware network uses articulated bones and blend skinning to represent arbitrary deformations, and is self-supervised on video datasets without requiring 3D shapes or viewpoints as input. Through distillation, our network learns to 3D-reconstruct unseen articulated objects at interactive frame rates. Our method yields higher-fidelity 3D reconstructions than prior real-time methods for animals, with the ability to render realistic images at novel viewpoints and poses.
Introduction
Reconstructing the 3D shape and motion of articulated objects—such as a dancing human or a running dog—from a single casual video is a long-standing challenge in computer vision. Current approaches typically fall into two camps: template-based methods and optimization-based neural fields. Template-based models (like SMPL for humans or SMAL for animals) are fast but restricted to specific categories and struggle with "in-the-wild" variations like fur or loose clothing. Conversely, dynamic Neural Radiance Fields (NeRFs) offer category-agnostic flexibility but are computationally expensive, requiring hours of test-time optimization for a single video.
DASR (Distilled Articulated Shape Reconstruction) bridges this gap. By distilling knowledge from a high-quality but slow optimization-based teacher into a fast feed-forward student network, DASR achieves real-time 3D reconstruction of diverse articulated objects at nearly 1,000x the speed of previous state-of-the-art optimization methods.
Methodology
Supervised Distillation
The core innovation of DASR is the use of supervised distillation to bypass the need for manual 3D labels. Instead of training on ground-truth 3D data, which is notoriously difficult to acquire for arbitrary animals, the authors utilize BANMo, a dynamic NeRF that fits 3D models to videos offline. While BANMo is slow (0.016 FPS), it produces high-fidelity 3D shapes, articulations, and textures that serve as "pseudo-ground truth."
Network Architecture
The student network is a feed-forward model designed for speed and temporal consistency. It consists of several specialized branches:
- Pose and Articulation Branch: This branch uses a stacked hourglass network and ResNet-18 to extract features from input video frames (RGB, segmentation, and CSE features). It predicts viewpoint hypotheses and articulation codes.
- Appearance Branch: A separate ResNet-18 predicts texture codes, which are decoded into vertex colors.
- Frozen Decoders: A critical design choice is the use of frozen decoders from the teacher model. By forcing the student to predict latent codes that are compatible with the teacher's pre-trained decoders, the learning task is simplified from generating 3D geometry to indexing into a learned 3D space.
- Temporal Encoding: To ensure smooth motion and prevent frame-to-frame jitter, a 1D convolutional temporal encoder processes a window of frames, providing the network with temporal context.
Experiments and Results
Datasets and Evaluation
The method was evaluated across three main categories: humans (using AMA, MonoPerfCap, and DAVIS datasets), cats, and dogs. For animals, the authors curated a dataset of 77 cat and 88 dog videos. To provide quantitative depth comparisons, they captured unique RGB-D pet videos using an iPad Pro, offering a rare source of ground-truth depth for in-the-wild animal reconstruction.
Performance and Accuracy
DASR achieves a massive performance leap, running at 15 FPS on an NVIDIA RTX-3090 GPU. This represents a ~937x speedup over the BANMo teacher. Despite this acceleration, the reconstruction quality remains competitive. On human datasets, DASR achieved a 3D Chamfer distance of approximately 10.6cm, nearly matching the teacher's 10.3cm performance. Qualitatively, the inclusion of the temporal encoder significantly improved stability, removing the flickering artifacts common in previous frame-by-frame reconstruction methods.
Discussion and Limitations
DASR demonstrates that optimization-based neural fields can be effectively distilled into real-time models. By leveraging frozen knowledge and temporal context, the student network learns to reconstruct unseen articulated objects with high fidelity.
However, the student is fundamentally capped by the teacher's performance; if the teacher fails to reconstruct a complex or occluded pose, the student cannot learn it. Additionally, while global shape and pose are well-captured, some fine-grained geometric details, such as specific clothing folds, may be lost compared to the full optimization process. Future work may explore hybrid approaches where the feed-forward network provides a high-quality initialization for a shortened optimization loop, further refining the results while maintaining interactive speeds.
Summary updated