DIV-FF: Dynamic Image-Video Feature Fields For Environment Understanding in Egocentric Videos

Lorenzo Mur-Labadia, Josechu Guerrero, Ruben Martinez-CantinCVPR 2025 · DOI

Abstract

DIV-FF (Dynamic Image-Video Feature Fields) is a framework designed to master the inherent "messiness" of first-person (egocentric) video. By decomposing scenes into three distinct neural streams—the static background, moving objects, and the actor—DIV-FF creates a robust 3D representation. By distilling both image-language (CLIP) and video-language (EgoVideo) features, the system understands not just object identity, but action-oriented affordances, maintaining consistency even when objects are moved or occluded by the wearer's hands.

Introduction: The Challenge of Egocentric Vision

Egocentric vision—video captured from a wearer's perspective—is a cornerstone for the future of robotics and augmented reality. However, processing this data is notoriously difficult. Unlike standard computer vision datasets, egocentric video is characterized by constant camera motion, frequent occlusions by the wearer’s hands (the "actor"), and objects that are dynamically manipulated.

Traditional 3D reconstruction methods, such as Neural Radiance Fields (NeRFs), typically assume a static environment. When objects move, these models often fail. Furthermore, existing language-embedded fields like LERF utilize CLIP, which excels at identifying static objects (e.g., a "spoon") but struggles with temporal actions (e.g., "stirring"). DIV-FF addresses these challenges by creating a unified 3D semantic representation that handles dynamics and understands action-oriented context, or affordances.

Methodology: Triple-Stream Architecture

The core of DIV-FF is a triple-stream architecture that separates the visual world into three specialized neural networks:

  1. Persistent Env Network: Models the static background, such as walls and counters.
  2. Dynamic Env Network: Models objects that change position, like pots, knives, and food.
  3. Actor Network: Models the wearer, specifically their hands and arms.

To imbue these streams with semantic intelligence, the authors employ Feature Distillation from pre-trained "teacher" models:

  • Image-Language (CLIP) + SAM: To resolve the blurry boundaries common in CLIP, the Segment Anything Model (SAM) is used to generate sharp masks. High-quality CLIP descriptors are assigned to these pixel-aligned masks, ensuring the 3D model learns precise object boundaries.
  • Video-Language (EgoVideo): This novel addition distills features from EgoVideo, a model trained on egocentric video-text pairs. This allows the system to respond to complex, action-based queries like "wash a kitchen utensil" or "cut the onion," which require temporal context beyond a single frame.

Datasets

The researchers evaluated DIV-FF using the EPIC-Diff subset of the EPIC-Kitchens dataset. This subset is particularly demanding, featuring long sequences of kitchen activities with multiple viewpoints, complex interactions, and significant object movement. Evaluations were conducted using approximately 900 calibrated frames per sequence.

Egocentric video sequences from the EPIC-Kitchens dataset showing various kitchen tasks and viewpoints.

Key Insights: Amodal Understanding and Affordances

DIV-FF offers several unique capabilities through its scene decomposition:

  • Amodal Understanding: By separating the actor stream, the model can perform amodal synthesis. Users can effectively "turn off" the actor stream to visualize objects hidden behind the wearer's hands, or remove dynamic objects to see the underlying static environment.
  • Affordances over Labels: Unlike models that simply label objects, DIV-FF identifies affordances—the potential for action. By leveraging video-language features, the model learns that a sink is not just a "sink" but a location where "washing" occurs.
  • SAM as a Geometric Prior: Using SAM masks during training acts as a powerful regularizer, eliminating the need for complex additional methods like DINO and resulting in significantly sharper 3D object boundaries.
DIV-FF capabilities including affordance segmentation (e.g., 'cut the onion'), amodal scene decomposition, and consistent tracking of dynamic objects.

Results and Performance

DIV-FF significantly outperforms state-of-the-art baselines:

  • Dynamic Object Segmentation: Achieved a 30.5 mIoU, a +40.5% relative improvement over the best baseline (NeuralDiff + OWL-ViT + SAM).
  • Affordance Segmentation: Achieved a 20.7 mIoU, representing a +69.7% relative improvement. This underscores the efficacy of distilling video-language features for action-oriented tasks.
  • Temporal Consistency: The model maintains a stable understanding of objects as they move, avoiding the "ghosting" artifacts or loss of tracking common in previous methods.

Conclusion and Future Work

While DIV-FF represents a major advancement, challenges remain. The computational intensity of NeRF-based training currently precludes real-time application on edge devices like AR glasses. Additionally, the system depends on accurate camera poses, which can be difficult to extract from fast-moving egocentric footage. Future research may leverage this static-dynamic-actor decomposition to build long-term spatial memory for robots, enabling them to track objects even after they leave the immediate field of view.

Summary updated

BibTeX
@inproceedings{murlabadia2025div,
  author = {Lorenzo Mur-Labadia and Josechu Guerrero and Ruben Martinez-Cantin},
  title = {{DIV-FF: Dynamic Image-Video Feature Fields For Environment Understanding in Egocentric Videos}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00329},
}
Figure