Sense4FL: Vehicular Crowdsensing Enhanced Federated Learning for Object Detection in Autonomous Driving

Yanan Ma, Senkang Hu, Zhengru Fang, Yun Ji, Yiqin Deng, Yuguang Fang

TL;DR

Sense4FL is a novel federated learning (FL) framework designed for autonomous driving that dynamically optimizes vehicle selection and data collection based on driving trajectories. By mathematically minimizing the statistical distance between locally harvested data and the true regional distribution, it significantly improves object detection accuracy and accelerates model adaptation across different urban environments.

What Problem Does This Paper Address?

While autonomous driving (AD) vision models have made massive strides, they still struggle to generalize under localized domain shifts, such as unique street layouts, regional vehicle types, or extreme weather conditions. To improve these models on the fly without compromising user privacy or overloading cellular networks with massive raw video uploads, researchers turn to Federated Learning (FL). In FL, vehicles train models locally on their onboard computers and only upload model weights to a central server.

However, existing vehicular FL schemes suffer from a major blind spot: they assume that vehicles have static, pre-determined datasets. In reality, autonomous vehicles continuously collect data as they drive. Because different routes (trajectories) expose vehicles to vastly different environments—some streets are packed with pedestrians, while others are dominated by trucks—ignoring a vehicle's path leads to highly biased local datasets. This statistical imbalance (non-IID data) severely degrades the performance of the aggregated global model.

Methods

The authors introduce Sense4FL, a framework that treats vehicles simultaneously as mobile crowdsensors and FL clients.

Fig. 2: Illustration of Sense4FL framework. Each autonomous vehicle acts as a mobile data collector and an FL client. As a vehicle traverses through a region, it collects data about street blocks and then leverages the collected data to train local models for FL. The selection of vehicles participating in the training process and the timing of starting their training are carefully designed by considering the impact of vehicles’ uncertain trajectories.

The core technical contributions include:

  1. Convergence Analysis via Earth Mover’s Distance (EMD): The authors derive a new convergence upper bound for FL that accounts for both uncertain vehicle trajectories and wireless upload probabilities. They prove that minimizing training loss is mathematically equivalent to minimizing a weighted sum of local and global EMD between the vehicles' collected data distributions and the target regional distribution.
  2. Problem Formulation: They formulate a joint vehicle selection and data collection problem aimed at minimizing this EMD-based loss. The problem is structured as a non-linear multiple-choice knapsack problem (MCKP), which is NP-hard due to its non-convex, non-separable objective function.
  3. Two-Step Approximation Algorithm: To solve this efficiently on the server side, they develop a low-complexity algorithm:
  4. Step 1 (Bisection): Solves a fractional programming problem using a bisection method to minimize client-level data divergence.
  5. Step 2 (Local Search): Refines the vehicle selection by iteratively swapping out the least effective vehicles to optimize global distribution alignment.
Fig. 3: The workflow of Sense4FL framework.

Datasets

The framework was evaluated using the nuImages dataset (an extension of the popular nuScenes dataset designed for 2D object detection).

  • Training/Evaluation Locations: Data from Singapore (One-North) was used to train models from scratch, and data from Boston (Seaport) was used to evaluate cross-city model adaptation.
  • Traffic Simulation: Real-world vehicle mobility and trajectories were simulated using the SUMO (Simulation of Urban MObility) simulator across 36 distinct geographic blocks.

Key Insights and Ideas

  • Trajectories Dictate Data Quality: A random selection of vehicles might over-sample common objects (like cars) and completely miss rare but critical classes (like traffic cones or strollers). Selecting vehicles based on where they are going ensures a statistically representative dataset for the entire region.
  • The EMD Breakthrough: Linking FL convergence bounds directly to Earth Mover's Distance provides a clean, geometric way to optimize client selection based on semantic data distributions rather than just raw network bandwidth or compute capacity.
  • Controlled Data Collection: Instead of telling vehicles to collect data indefinitely, Sense4FL optimizes when a vehicle should stop collecting data and start training, balancing the trade-off between data diversity and the strict time limits of wireless communication windows.

Key Results

The authors compared Sense4FL against several standard benchmarks, including random selection, gradient-based selection, and uploading-centric selection.

Fig. 1: The data distribution of four neighboring street blocks and the final test accuracy (after convergence) via different vehicle selection strategies in the nuImages dataset. {hx, hy} means that we select one vehicle from trajectory x and one vehicle from y in each round; “Random” means we randomly choose two vehicles in each round.
  • Higher Accuracy: Sense4FL consistently outperformed all benchmarks in mean Average Precision ([email protected] and [email protected]:.95). For instance, trajectory-aware selection yielded a 4.4% accuracy boost over trajectory-agnostic random selection in baseline tests.
  • Superior Adaptation: When adapting a model pre-trained in Singapore to the new environment of Boston, Sense4FL converged much faster and achieved significantly higher precision and recall than other methods, proving its capability to handle domain shifts effectively.
  • Robustness: The framework maintained its performance edge across varying network conditions, including low upload data rates and limited vehicle availability.

Limitations and Future Work

While Sense4FL provides a robust framework for object detection, the authors note a few areas for expansion:

  • Task Scope: The current work focuses strictly on 2D object detection. Future iterations could extend the framework to other critical AD tasks like semantic segmentation or direct control command generation (e.g., acceleration/braking).
  • Trajectory Complexity: The mobility model assumes relatively simple trajectory prediction parameters ($M_v = 2$ potential paths per vehicle). Incorporating more advanced, deep-learning-based intent and trajectory prediction models could improve selection accuracy in highly chaotic environments.

Summary updated

Abstract

To accommodate constantly changing road conditions, real-time vision model training is essential for autonomous driving (AD). Federated learning (FL) serves as a promising paradigm to enable autonomous vehicles to train models collaboratively with their onboard computing resources. However, existing vehicle selection schemes for FL all assume predetermined and location-independent vehicles' datasets, neglecting the fact that vehicles collect training data along their routes, thereby resulting in suboptimal vehicle selection. In this paper, we focus on the fundamental perception problem and propose Sense4FL, a vehicular crowdsensing-enhanced FL framework featuring \textit{trajectory-dependent} vehicular \textit{training data collection} to \rev{improve the object detection quality} in AD for a region. To this end, we first derive the convergence bound of FL by considering the impact of both vehicles' uncertain trajectories and uploading probabilities, from which we discover that minimizing the training loss is equivalent to minimizing a weighted sum of local and global earth mover's distance (EMD) between vehicles' collected data distribution and global data distribution. Based on this observation, we formulate the trajectory-dependent vehicle selection and data collection problem for FL in AD. Given that the problem is NP-hard, we develop an efficient algorithm to find the solution with an approximation guarantee. Extensive simulation results have demonstrated the effectiveness of our approach in improving object detection performance compared with existing benchmarks.
BibTeX
@article{ma2026sense,
  author = {Yanan Ma and Senkang Hu and Zhengru Fang and Yun Ji and Yiqin Deng and Yuguang Fang},
  title = {{Sense4FL: Vehicular Crowdsensing Enhanced Federated Learning for Object Detection in Autonomous Driving}},
  journal = {IEEE Transactions on Mobile Computing},
  year = {2026},
  doi = {10.1109/tmc.2026.3674333},
}
Figure