Abstract
Semantic Flow is a novel neural representation designed to learn the semantic field of dynamic scenes from monocular videos. By shifting the focus from individual 3D points to continuous motion trajectories (flows), the model captures rich temporal context. This approach enables superior performance in scene editing, tracking, and generalization to novel environments compared to traditional point-based methods.
Introduction
Reconstructing dynamic scenes—where objects like people, vehicles, or balloons are in motion—from a single camera remains a significant challenge in computer vision. The primary hurdle is 2D-to-3D ambiguity, where it is difficult to determine the precise 3D location of an object from a single viewpoint.
Existing Dynamic Neural Radiance Fields (NeRFs) typically predict semantic labels for individual points in space. However, point-based representations lack temporal context, often leading to overfitting on training frames and poor generalization to unseen viewpoints or future frames.
Methodology: From Points to Flows
The core innovation of this work is Semantic Flow, which transitions from point-based learning to flow-based learning. Instead of classifying static points, the model learns to identify moving trajectories.
The architecture incorporates several key components:
- Implicit Flow Field: An MLP predicts the 3D trajectory (flow) of points across timestamps.
- Flow Feature Aggregation: The model samples local image features along these 3D trajectories, providing a multi-view, multi-temporal perspective of objects.
- Flow Attention Module: A multi-head attention mechanism reasons about motion to extract a robust semantic signature from aggregated features.
- Volume Density Priors: The model utilizes the scene's physical density to resolve 2D-to-3D ambiguity, guiding the application of semantic labels along camera rays.
The Semantic Dynamic Scene Dataset
To evaluate the model, the authors introduced the Semantic Dynamic Scene Dataset. Built upon existing dynamic scene data, it includes 7 forward-facing scenes such as Balloon, Jumping, and Umbrella. The researchers provided manual pixel-wise semantic annotations, creating a benchmark for semantic understanding in dynamic 3D reconstructions.
Key Results and Applications
Semantic Flow demonstrates significant improvements over baselines like DynNeRF and MonoNeRF.
- Accuracy: The model achieves higher mIoU scores across multiple scenes. For the Balloon1 scene, it reached an mIoU of 0.844, far exceeding DynNeRF's 0.515.
- Temporal Consistency: By ensuring labels are consistent across a flow's trajectory, the model maintains sharp boundaries and temporal stability.
- Scene Editing: The semantic field allows for precise instance-level editing. For example, a specific person can be removed from a scene by zeroing out the density of flows associated with their semantic label.
Conclusion and Future Work
Semantic Flow effectively leverages motion as a semantic cue, providing a more robust representation for dynamic scenes. While the model currently depends on the quality of initial flow estimation, it opens new avenues for self-supervised learning and 360-degree dynamic scene reconstruction.
Summary updated