FedMIL: Federated-Multiple Instance Learning for Video Analysis with Optimized DPP Scheduling

Ashish Bastola, Hao Wang, Xiwen Chen, Abolfazl Razi

TL;DR

FedMIL is a new framework that combines Federated Learning (FL) with Multiple Instance Learning (MIL) to detect traffic accidents across decentralized CCTV networks without compromising privacy. By introducing a novel client selection strategy called DPPQ, the system intelligently picks the most diverse and informative data sources, achieving higher accuracy and faster convergence even when data is messy and computational resources are limited.

What Problem Does This Paper Address?

Training AI for real-world video analysis, such as detecting car crashes across a city's camera network, faces three massive hurdles:

  1. Privacy and Bandwidth: Sending raw high-resolution video to a central server is a privacy nightmare and a massive drain on network bandwidth.
  2. Labeling Fatigue: Supervised learning usually requires humans to label every single frame. In video, where most frames are redundant, this is incredibly labor-intensive.
  3. The "Non-IID" Mess: In the real world, data isn't distributed equally. One camera might be in a snowy rural area and never see a crash, while another is at a busy urban intersection. This "Non-Independently and Identically Distributed" (non-IID) data often causes standard decentralized learning models to fail or converge very slowly.

Methods

The authors propose FedMIL, which merges two powerful concepts:

  • Federated Learning (FL): Instead of sharing raw video, cameras (clients) train models locally and only share their "learnings" (model weights) with a central server.
  • Multiple Instance Learning (MIL): Instead of labeling every frame, MIL treats a whole video clip as a "bag" and the frames as "instances." If a crash happens anywhere in the clip, the whole bag is labeled "positive." This drastically reduces human labeling effort.

To make this work efficiently, they use an AttentionMIL model, which learns to "pay attention" to the specific frames in a video that actually contain the accident.

The paper's secret sauce is DPPQ (Determinantal Point Process with Quality). Standard client selection often picks cameras at random or just looks for diversity. DPPQ uses a specialized kernel to select a subset of clients that are both diverse (representing different weather, lighting, and locations) and high-quality (having data that the current model finds difficult, indicated by higher loss gradients).

Figure 1: The architecture of FedMIL. Compared to classical central learning, the training process is broken down to the client level to prevent massive data exchange. In advance, with a pre-trained CNN model to extract the video feature and a lightweight MIL model for bag-level classification, the computational cost for edge devices is greatly reduced.

Datasets

The researchers evaluated their framework using two primary datasets:

  • Car Crash Dataset (CCD): A specialized collection of 1,500 dashcam accident videos and 3,000 normal videos. Features were extracted using VGG-16 and ResNeXt-101 architectures.
  • MNIST: The classic handwritten digit dataset was used as a benchmark to validate the general effectiveness of the DPPQ selection method against standard baselines.

To simulate the messy real world, they intentionally imbalanced these datasets using power-law distributions (for label imbalance) and Dirichlet distributions (for feature imbalance, like different environments).

Simulating real-world data imbalance. (a) shows label skew, (b) shows how different clients get different "types" of video clusters, and (c) visualizes these clusters (e.g., snow vs. urban).

Key Insights and Ideas

  • Quality Matters as Much as Diversity: Most diversity-based selection methods (like standard DPP) ignore how much a client can actually contribute to the model's learning. By including a "quality" factor based on local loss, DPPQ ensures the model learns from the most "challenging" and representative data first.
  • Efficiency through Selection: You don't need every camera to participate in every round. In fact, picking a smart subset of clients often leads to better performance than using the entire, noisy dataset.
  • MIL is a Natural Fit for Video: Treating video as a bag of instances isn't just a labeling shortcut; it aligns with how events actually unfold in time, allowing the model to focus on the most informative snippets of a recording.

Key Results

  • Superior Accuracy: On the MNIST benchmark, DPPQ achieved 90.84% accuracy, a significant jump over random selection (78.69%) and standard DPP (83.70%).
  • Faster Convergence: In traffic accident detection (CCD), DPPQ reached higher accuracy levels much faster than other methods, especially in extreme non-IID scenarios.
  • Robustness to Low Data: Even when only 50% of the available data was utilized, the DPPQ-enabled FedMIL framework maintained high performance, making it ideal for edge devices with limited power or intermittent connectivity.

Limitations and Future Work

While the results are promising, the authors note a few areas for growth:

  • Task Complexity: The current work focuses on binary classification (accident vs. no accident). Future iterations could tackle multi-class problems, such as identifying the type of crash or the severity.
  • Broader Applications: The FedMIL framework is general-purpose. The authors suggest it could be easily extended to other critical areas like deepfake detection, human action recognition, and general video anomaly detection.
  • Scalability: While tested on hundreds of nodes, the dynamics of scaling to tens of thousands of sensors in a "mega-city" environment remain an open research question.

Summary updated

Abstract

Many AI platforms, including traffic monitoring systems, use Federated Learning (FL) for decentralized sensor data processing for learning-based applications while preserving privacy and ensuring secured information transfer. On the other hand, applying supervised learning to large data samples, like high-resolution images requires intensive human labor to label different parts of a data sample. Multiple Instance Learning (MIL) alleviates this challenge by operating over labels assigned to the 'bag' of instances. In this paper, we introduce Federated Multiple-Instance Learning (FedMIL). This framework applies federated learning to boost the training performance in video-based MIL tasks such as vehicle accident detection using distributed CCTV networks. However, data sources in decentralized settings are not typically Independently and Identically Distributed (IID), making client selection imperative to collectively represent the entire dataset with minimal clients. To address this challenge, we propose DPPQ, a framework based on the Determinantal Point Process (DPP) with a quality-based kernel to select clients with the most diverse datasets that achieve better performance compared to both random selection and current DPP-based client selection methods even with less data utilization in the majority of non-IID cases. This offers a significant advantage for deployment on edge devices with limited computational resources, providing a reliable solution for training AI models in massive smart sensor networks.
BibTeX
@inproceedings{bastola2024fedmil,
  author = {Ashish Bastola and Hao Wang and Xiwen Chen and Abolfazl Razi},
  title = {{FedMIL: Federated-Multiple Instance Learning for Video Analysis with Optimized DPP Scheduling}},
  booktitle = {DCOSS-IoT},
  year = {2024},
  doi = {10.1109/dcoss-iot61029.2024.00025},
}
Figure