Reverse Distribution Based Video Moment Retrieval for Effective Bias Elimination

Lingdu Kong, Xiaochun Yang, Tieying Li, Bin Wang, Xiangmin ZhouAAAI 2025 · DOI

TL;DR

This paper tackles the critical issue of bias in Video Moment Retrieval (VMR), where models can achieve high performance by exploiting dataset artifacts rather than understanding video content. They introduce a novel evaluation framework, ReDis-VMR, that dynamically generates datasets to eliminate data and model biases, and a Dynamic Expandable Adjustment (DEA) pipeline to improve model robustness and focus on true video-text understanding.

What Problem Does This Paper Address?

Video Moment Retrieval (VMR) aims to pinpoint a specific temporal segment within a video that best matches a given textual query. While VMR models have seen significant advancements, a pervasive problem is "bias." This bias manifests in two forms:

  • Data Bias: The training and testing datasets have similar distributions of temporal segments, allowing models to achieve good results by simply predicting common segments without truly analyzing the video content.
  • Model Bias: The model's own prediction distribution (its "candidate set") is too similar to the test set distribution. This means the model might have a tendency to predict certain segments regardless of the input video, leading to high scores even without proper learning.

Existing methods for addressing bias, like dataset resplitting, are insufficient. They primarily tackle data bias and fail to quantify or eliminate model bias, leading to an inaccurate assessment of a VMR model's true capabilities. The authors argue for a more comprehensive definition and evaluation of bias.

Methods

The paper proposes two main contributions:

  1. Reverse Distribution based VMR (ReDis-VMR): This is primarily an evaluation framework. It dynamically generates datasets with "inverse distributions" tailored to specific model designs.
  2. It first trains a model on the original dataset and uses its predictions to estimate a "concentrated" temporal segment distribution.
  3. Then, it generates a "reverse distribution" dataset by sampling temporal segments that are less likely under the original concentrated distribution.
  4. Combining the original and reverse datasets creates a more uniformly distributed dataset, which is then used for evaluation. This process aims to eliminate both data and model biases by forcing the model to perform well on data that doesn't mirror its learned biases.
  5. Dynamic Expandable Adjustment (DEA) Pipeline: This is a method for improving VMR models themselves, built upon the ReDis-VMR evaluation.
  6. Dynamic Reverse Data Generation: This module is integrated into the training process. It uses the ReDis-VMR concept to generate reverse distributed datasets on-the-fly.
  7. Dynamic Expandable Adjustment: The model architecture is dynamically adjusted. It deconstructs existing models into representation-focused and distribution-focused parts. During training, the representation-focused part is expanded (new model structures are added and trained), while the distribution-focused part is frozen. This encourages the model to rely more on video and text features rather than learned distribution patterns.
  8. Fair Loss: A novel loss function is introduced to make the model insensitive to concentrated data distributions. It encourages the predicted confidence distribution of the model to be uniform, thereby reducing predictive tendencies. This is achieved by minimizing the KL divergence between the model's predicted distribution and a uniform distribution.

Datasets

The paper uses the following datasets for training and evaluation:

  • Charades-STA: A dataset of daily indoor activity videos with temporal annotations.
  • ActivityNet Captions: A dataset containing videos with temporally localized sentences.

The authors do not introduce a new benchmark dataset but rather a novel evaluation setting (ReDis-VMR) that dynamically generates data for evaluation.

Key Insights and Ideas

  • Bias is a Two-Headed Monster: The paper clearly distinguishes between data bias and model bias, highlighting that both are crucial and often overlooked. This nuanced definition is a significant contribution.
  • Existing Debias Methods Fall Short: The resplitting method, while a step, is shown to be ineffective and can even exacerbate model bias. This underscores the need for a more robust approach.
  • Dynamic Data Generation is Key: The core idea of ReDis-VMR – creating "inverse" or "reverse" distributions – is powerful. It forces models to generalize beyond their learned biases by exposing them to data that actively counteracts those biases.
  • Architecture Matters for Bias Mitigation: The DEA pipeline's strategy of dynamically expanding the representation-focused part of the model while freezing the distribution-focused part is an elegant way to steer the model towards true understanding.
  • Fair Loss for Uniformity: The fair loss function is a clever mechanism to prevent models from developing strong predictive tendencies based on dataset artifacts.

Key Results

  • State-of-the-Art Bias Elimination: The ReDis-VMR evaluation method significantly reduces both data and model bias ratios compared to traditional methods and resplitting, as demonstrated in Figure 2 and Table 1. For instance, ReDis-VMR achieves much lower bias ratios across various models and datasets.
  • DEA Improves Performance: The DEA pipeline, when used with ReDis-VMR, consistently improves VMR performance (measured by [email protected] and mIoU) across different baselines and datasets. The improvements are more pronounced when bias is lower, indicating DEA effectively leverages video input.
  • Ablation Studies Confirm Component Effectiveness: Ablation studies (Table 2) show that each component of DEA (dynamic structure, replay mechanism, fair loss) contributes to performance improvements, with the dynamic model structure showing significant gains.
  • Fair Loss Effectiveness: Figure 5 shows that the fair loss function effectively improves mIoU and [email protected] metrics, especially when the hyperparameter λ is tuned appropriately, indicating its role in mitigating bias.

Limitations and Future Work

  • Limitations:
    • The paper focuses on bias elimination for evaluation and model improvement. While effective, the inherent complexity of VMR tasks might still present challenges.
  • The paper acknowledges that the definition of P(s,e) and U(s,e) in the fair loss needs to be tailored for different model designs (regression vs. discrete candidate sets), suggesting ongoing research in this area.
  • Future Work:
    • The authors aim to explore more effective methods for bias elimination in VMR to achieve even more accurate assessments of model capabilities.
  • The goal is to develop models that can better understand video content without relying on superficial cues.

Summary updated

BibTeX
@inproceedings{kong2025reverse,
  author = {Lingdu Kong and Xiaochun Yang and Tieying Li and Bin Wang and Xiangmin Zhou},
  title = {{Reverse Distribution Based Video Moment Retrieval for Effective Bias Elimination}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i11.33302},
}
Figure