Scene-robust Natural Language Video Localization via Learning Domain-invariant Representations

Zehan Wang, Yang Zhao, Haifeng Huang, Yan Xia, Zhou Zhao

Introduction

Natural Language Video Localization (NLVL) is a critical task in computer vision that involves identifying the specific temporal segment within a long video that corresponds to a given natural language query (e.g., "find the part where the person starts washing her hands"). While current state-of-the-art models perform well on familiar data, they often suffer from a lack of robustness when encountering novel environments. This "homebody" problem manifests as a significant drop in performance when lighting, backgrounds, or settings change from those seen during training.

Examples of NLVL queries across different scenes (bathroom, bedroom, kitchen) with ground truth temporal segments.

Researchers from Zhejiang University, in a paper presented at ACL 2023, address this challenge by introducing a framework for Scene-robust NLVL. Their approach focuses on learning domain-invariant representations by decoupling the core action from scene-specific "decor," thereby enhancing generalization to unseen environments.

The Challenge of Domain Generalization in NLVL

Domain Generalization (DG) aims to train models that perform reliably on "out-of-distribution" data. In NLVL, a model might erroneously associate an action like "washing hands" with specific bathroom tiles encountered during training. When tested in a kitchen or outdoor setting, this reliance on scene-specific features leads to failure.

Achieving DG in NLVL is particularly complex due to:

  1. Multi-modality: The need to align high-dimensional video features with dense textual information.
  2. Information Density Disparity: Sentences are semantically dense, whereas videos contain significant redundant background information.
  3. Temporal Reasoning: Localization requires complex multi-step reasoning over time, not just simple pattern matching.

Methodology: Alignment and Decomposition

The proposed framework employs a dual strategy: Multi-modal Domain Alignment and Feature Refinement by Decomposition.

Multi-modal Domain Alignment

To ensure features are consistent across different scenes, the authors utilize:

  • Intra- and Inter-sample Metrics: Alignment is performed not just on average features but by measuring distributions within individual videos and across batches to capture temporal and cross-domain variance.
  • Asymmetric Alignment: Recognizing the density gap between text and video, the model aligns raw video features and fused (video+text) features separately. This prevents the loss of textual nuances that might occur with overly aggressive global alignment.
  • Dirichlet-Mixup: This data augmentation technique blends features from multiple source domains to create synthetic intermediate domains, encouraging the model to learn more flexible, scene-agnostic representations.

Feature Refinement by Decomposition

The architecture incorporates a novel decomposition mechanism using two types of predictors:

  • Domain-Agnostic Predictor: Focuses on features common across all training scenes to identify the target segment.
  • Domain-Specific Predictors: Dedicated predictors for each known scene (e.g., Kitchen, Bedroom) act as a "safety valve." If a sample is highly dependent on its specific environment, the domain-specific predictor handles it, preventing scene-specific noise from polluting the domain-invariant learner.

Experimental Evaluation

The researchers restructured three standard NLVL benchmarks - Charades-STA (split by room), ActivityNet-Caption (split by activity category), and YouCook2 (split by regional cuisine) - to test scene-robustness. The models were trained on all domains except one, which served as the unseen target for testing generalization.

Key Results

The proposed method consistently outperformed standard baselines and existing DG techniques (such as DANN or MMD):

  • Charades-STA: Average accuracy (R@1, IoU=0.5) improved from 31.20% to 37.58%.
  • ActivityNet-Caption: Performance increased from 34.60% to 38.79%.
  • Qualitative Performance: The robust model demonstrated a superior ability to ignore scene-specific distractors (e.g., a child's smile in a "Social" scene) and focus on the semantic content of the query (e.g., "toy on the clothes").

Conclusion and Future Work

This work represents a significant step toward making NLVL models practical for real-world, diverse environments. By explicitly separating scene-specific noise from invariant action representations, the framework achieves superior generalization. Future research will explore scaling these techniques to massive, uncurated datasets and addressing the challenges of fine-grained action recognition and vocabulary gaps in open-world scenarios.

Summary updated

Abstract

Natural language video localization(NLVL) task involves the semantic matching of a text query with a moment from an untrimmed video. Previous methods primarily focus on improving performance with the assumption of independently identical data distribution while ignoring the out-of-distribution data. Therefore, these approaches often fail when handling the videos and queries in novel scenes, which is inevitable in real-world scenarios. In this paper, we, for the first time, formulate the scene-robust NLVL problem and propose a novel generalizable NLVL framework utilizing data in multiple available scenes to learn a robust model. Specifically, our model learns a group of generalizable domain-invariant representations by alignment and decomposition. First, we propose a comprehensive intra- and inter-sample distance metric for complex multi-modal feature space, and an asymmetric multi-modal alignment loss for different information densities of text and vision. Further, to alleviate the conflict between domain-invariant features for generalization and domain-specific information for reasoning, we introduce domain-specific and domain-agnostic predictors to decompose and refine the learned features by dynamically adjusting the weights of samples. Based on the original video tags, we conduct extensive experiments on three NLVL datasets with different-grained scene shifts to show the effectiveness of our proposed methods.
BibTeX
@inproceedings{wang2023scene,
  author = {Zehan Wang and Yang Zhao and Haifeng Huang and Yan Xia and Zhou Zhao},
  title = {{Scene-robust Natural Language Video Localization via Learning Domain-invariant Representations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.11},
}
Figure