FedVMR: A New Federated Learning Method for Video Moment Retrieval

Yan Wang, Xin Luo, Zhen-Duo Chen, Peng-Fei Zhang, Meng Liu, Xin-Shun Xu

TL;DR

FedVMR is the first framework to apply Federated Learning (FL) to the task of Video Moment Retrieval (VMR), allowing models to learn from distributed, private video data without centralizing it. By introducing a "grouped sequential" training strategy and a novel temporal-distribution loss, the authors achieve performance that matches or even exceeds traditional centralized training while maintaining data privacy.

What Problem Does This Paper Address?

Video Moment Retrieval (VMR) is the task of pinpointing a specific segment within a long video based on a natural language query (e.g., "the person puts on their shoes"). Traditionally, training these models requires massive, centralized datasets. However, real-world video data—from CCTV systems to personal smartphones—is often siloed across different devices due to massive file sizes and strict privacy concerns.

Standard Federated Learning (FL) offers a solution by training models locally on client devices and only sharing parameter updates. However, standard FL struggles with VMR for two reasons:

  1. Non-IID Data: Different users have very different video habits, leading to "Non-Identically and Independently Distributed" data that confuses global models.
  2. Parallel Inefficiency: In traditional FL, clients train in isolation. This lack of interaction often leads to slower convergence and lower accuracy compared to seeing all the data at once.

Methods

The authors propose FedVMR, which introduces three key technical innovations:

  • Grouped Sequential Federated Learning: Instead of training all clients in parallel (the standard FL approach), FedVMR divides clients into groups. Within each group, clients train sequentially—passing the model from one to the next—allowing the model to "accumulate" knowledge from peers before reaching the server. Groups themselves still run in parallel to maintain efficiency.
  • Selective Model Aggregation (C-Validation): The server maintains a tiny, voluntarily uploaded "c-validation" set. It uses this to score the quality of updates from different clients. Better-performing local models are given higher "attention" weights during the global aggregation process.
  • Temporal-Distribution-Gap Loss: To handle the non-IID problem, the authors treat the start and end times of video segments as "temporal classes." They calculate the distribution of these classes across the entire population and use a KL-divergence loss to encourage local models to bridge the gap between their local data distribution and the global one.

Datasets

The researchers evaluated FedVMR on two standard VMR benchmarks:

  • Charades-STA: 9,848 videos of indoor activities with over 12,000 annotated moment-sentence pairs.
  • ActivityNet Captions: A larger dataset containing 19,209 videos and over 70,000 moment-sentence pairs.

To simulate a federated environment, they partitioned these datasets among 16 clients (for Charades) and 200 clients (for ActivityNet) using a Dirichlet distribution to create realistic, non-IID data splits.

Key Insights and Ideas

  • Sequential > Parallel: The most striking insight is that a bit of sequential "peer learning" within groups significantly boosts performance and convergence speed compared to the purely parallel approach used in algorithms like FedAvg.
  • Temporal Labels as Classes: Since VMR doesn't have discrete "object classes" like image recognition, the authors cleverly discretized the video timeline into quarters to create "temporal classes." This allows them to apply well-known statistical tools (like KL-divergence) to measure and fix data skew.
  • Surpassing the "Gold Standard": Surprisingly, in some tests, FedVMR actually outperformed centralized training. This suggests that the grouped sequential approach might act as a form of regularization or more effective curriculum learning.

Key Results

  • Accuracy: On the Charades-STA dataset (IoU > 0.7), FedVMR achieved an accuracy of 31.60%, surpassing the centralized "gold standard" of 30.15% and significantly beating standard FedAvg (25.54%).
  • Convergence: FedVMR reached peak accuracy in significantly fewer communication rounds than other federated baselines.
  • Efficiency Trade-off: The "Grouped Sequential" method allows for a tunable balance. Using 2 groups of 8 clients was nearly 3x faster than a single sequential chain while still providing much better accuracy than 16 parallel clients.

Limitations and Future Work

  • The C-Validation Set: The method relies on a small set of voluntarily uploaded data. In extremely high-privacy scenarios, even this 1% of data might be a bottleneck or a privacy risk.
  • Communication Overhead: While the paper focuses on accuracy and convergence rounds, the sequential passing of models within groups adds a different type of latency compared to pure parallel broadcasts.
  • Model Specificity: The authors used a specific VMR model (SSMN) for their tests. While they claim the framework is general, future work could explore how it interacts with the latest Transformer-based VMR architectures.

Summary updated

Abstract

Despite the great success achieved, existing video moment retrieval (VMR) methods are developed under the assumption that data are centralizedly stored. However, in real-world applications, due to the inherent nature of data generation and privacy concerns, data are often distributed on different silos, bringing huge challenges to effective large-scale training. In this work, we try to overcome above limitation by leveraging the recent success of federated learning. As the first that is explored in VMR field, the new task is defined as video moment retrieval with distributed data. Then, a novel federated learning method named FedVMR is proposed to facilitate large-scale and secure training of VMR models in decentralized environment. Experiments on benchmark datasets demonstrate its effectiveness. This work is the very first attempt to enable safe and efficient VMR training in decentralized scene, which is hoped to pave the way for further study in the related research field.
BibTeX
@inproceedings{wang2023fedvmr,
  author = {Yan Wang and Xin Luo and Zhen-Duo Chen and Peng-Fei Zhang and Meng Liu and Xin-Shun Xu},
  title = {{FedVMR: A New Federated Learning Method for Video Moment Retrieval}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=zxEGtYNcFu},
}
Figure