Semantic Flow: Learning Semantic Field of Dynamic Scenes from Monocular Videos

Fengrui Tian, Yueqi Duan, Angtian Wang, Jianfei Guo, Shaoyi DuarXiv 2024 · DOI · arXiv

Abstract

Semantic Flow is a novel neural representation designed to learn the semantic field of dynamic scenes from monocular videos. By shifting the focus from individual 3D points to continuous motion trajectories (flows), the model captures rich temporal context. This approach enables superior performance in scene editing, tracking, and generalization to novel environments compared to traditional point-based methods.

Introduction

Reconstructing dynamic scenes—where objects like people, vehicles, or balloons are in motion—from a single camera remains a significant challenge in computer vision. The primary hurdle is 2D-to-3D ambiguity, where it is difficult to determine the precise 3D location of an object from a single viewpoint.

Existing Dynamic Neural Radiance Fields (NeRFs) typically predict semantic labels for individual points in space. However, point-based representations lack temporal context, often leading to overfitting on training frames and poor generalization to unseen viewpoints or future frames.

Methodology: From Points to Flows

The core innovation of this work is Semantic Flow, which transitions from point-based learning to flow-based learning. Instead of classifying static points, the model learns to identify moving trajectories.

Overview of the Semantic Flow pipeline, illustrating dynamic scene tracking, semantic view synthesis, completion, and adaptation.

The architecture incorporates several key components:

  1. Implicit Flow Field: An MLP predicts the 3D trajectory (flow) of points across timestamps.
  2. Flow Feature Aggregation: The model samples local image features along these 3D trajectories, providing a multi-view, multi-temporal perspective of objects.
  3. Flow Attention Module: A multi-head attention mechanism reasons about motion to extract a robust semantic signature from aggregated features.
  4. Volume Density Priors: The model utilizes the scene's physical density to resolve 2D-to-3D ambiguity, guiding the application of semantic labels along camera rays.

The Semantic Dynamic Scene Dataset

To evaluate the model, the authors introduced the Semantic Dynamic Scene Dataset. Built upon existing dynamic scene data, it includes 7 forward-facing scenes such as Balloon, Jumping, and Umbrella. The researchers provided manual pixel-wise semantic annotations, creating a benchmark for semantic understanding in dynamic 3D reconstructions.

Key Results and Applications

Semantic Flow demonstrates significant improvements over baselines like DynNeRF and MonoNeRF.

  • Accuracy: The model achieves higher mIoU scores across multiple scenes. For the Balloon1 scene, it reached an mIoU of 0.844, far exceeding DynNeRF's 0.515.
  • Temporal Consistency: By ensuring labels are consistent across a flow's trajectory, the model maintains sharp boundaries and temporal stability.
  • Scene Editing: The semantic field allows for precise instance-level editing. For example, a specific person can be removed from a scene by zeroing out the density of flows associated with their semantic label.
Example of instance-level scene editing where a specific person is removed from the dynamic scene.

Conclusion and Future Work

Semantic Flow effectively leverages motion as a semantic cue, providing a more robust representation for dynamic scenes. While the model currently depends on the quality of initial flow estimation, it opens new avenues for self-supervised learning and 360-degree dynamic scene reconstruction.

Summary updated

Abstract

In this work, we pioneer Semantic Flow, a neural semantic representation of dynamic scenes from monocular videos. In contrast to previous NeRF methods that reconstruct dynamic scenes from the colors and volume densities of individual points, Semantic Flow learns semantics from continuous flows that contain rich 3D motion information. As there is 2D-to-3D ambiguity problem in the viewing direction when extracting 3D flow features from 2D video frames, we consider the volume densities as opacity priors that describe the contributions of flow features to the semantics on the frames. More specifically, we first learn a flow network to predict flows in the dynamic scene, and propose a flow feature aggregation module to extract flow features from video frames. Then, we propose a flow attention module to extract motion information from flow features, which is followed by a semantic network to output semantic logits of flows. We integrate the logits with volume densities in the viewing direction to supervise the flow features with semantic labels on video frames. Experimental results show that our model is able to learn from multiple dynamic scenes and supports a series of new tasks such as instance-level scene editing, semantic completions, dynamic scene tracking and semantic adaption on novel scenes. Codes are available at https://github.com/tianfr/Semantic-Flow/.
BibTeX
@misc{tian2024semantic,
  author = {Fengrui Tian and Yueqi Duan and Angtian Wang and Jianfei Guo and Shaoyi Du},
  title = {{Semantic Flow: Learning Semantic Field of Dynamic Scenes from Monocular Videos}},
  year = {2024},
  eprint = {2404.05163},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.05163},
}
Figure