Abstract
Fleets of networked autonomous vehicles (AVs) collect terabytes of sensory data, which is often transmitted to central servers (the "cloud") for training machine learning (ML) models. Ideally, these fleets should upload all their data, especially from rare operating contexts, in order to train robust ML models. However, this is infeasible due to prohibitive network bandwidth and data labeling costs. Instead, we propose a cooperative data sampling strategy where geo-distributed AVs collaborate to collect a diverse ML training dataset in the cloud. Since the AVs have a shared objective but minimal information about each other's local data distribution and perception model, we can naturally cast cooperative data collection as an $N$-player mathematical game. We show that our cooperative sampling strategy uses minimal information to converge to a centralized oracle policy with complete information about all AVs. Moreover, we theoretically characterize the performance benefits of our game-theoretic strategy compared to greedy sampling. Finally, we experimentally demonstrate that our method outperforms standard benchmarks by up to $21.9\%$ on 4 perception datasets, including for autonomous driving in adverse weather conditions. Crucially, our experimental results on real-world datasets closely align with our theoretical guarantees.
Introduction
Autonomous vehicles (AVs) are inherently data-intensive. To develop robust perception models, these systems must be trained on diverse datasets encompassing both common scenarios and rare "out-of-distribution" (OoD) events, such as extreme weather or unusual road configurations. However, a single AV can generate 20-30 GB of data per second, while typical 5G infrastructure provides only about 10 Gbps for all local users.
This bandwidth bottleneck makes uploading all collected data physically impossible. When robots act independently or "greedily," they tend to upload redundant data that they individually perceive as valuable, leading to a biased and inefficient global dataset. The challenge lies in coordinating a decentralized fleet to ensure the cloud receives a balanced dataset without requiring a central authority to process all raw data first.
Methodology: The INTERACTIVE Algorithm
This research, presented at CoRL 2022, treats swarm data collection as an $N$-player Potential Game. In this framework, all robots share a common potential function: minimizing the distance between the aggregated cloud dataset and a target balanced distribution.
The proposed INTERACTIVE algorithm enables coordination through several key mechanisms:
- Perceptual Uncertainty Modeling: Using a Confusion Matrix, the system accounts for the likelihood of robots misclassifying rare events, ensuring the coordination logic is robust to real-world sensor errors.
- Decentralized Action Sharing: Instead of transmitting raw data, robots share summaries of their "feasible actions"—the distributions of data they are capable of providing.
- Nash Equilibrium Convergence: Robots iteratively refine their sampling strategies. The authors prove that this decentralized process converges to a Nash Equilibrium equivalent to the solution of a "Centralized Oracle" with perfect global information.
- Communication Efficiency: Theoretical guarantees show that in many practical settings, the fleet can reach an optimal strategy with just a single round of message passing.
Experimental Evaluation
The method was rigorously tested on four datasets: MNIST (handwritten digits), CIFAR-10 (object classification), Adverse-Weather (containing rain, fog, and snow), and Berkeley DeepDrive (BDD100K).
Key results include:
- Significant Accuracy Gains: The INTERACTIVE policy achieved up to a 21.9% improvement in final model accuracy over greedy and uniform sampling benchmarks.
- Optimal Dataset Balancing: On the MNIST and CIFAR-10 benchmarks, the algorithm successfully produced a nearly uniform distribution of classes in the cloud, whereas greedy methods left significant gaps in rare classes.
- Real-World Performance: In the BDD100K driving dataset tests, the algorithm outperformed the greedy benchmark by 12.4% in accuracy while uploading only a small fraction of the total observed data.
- Specialization through Cooperation: The game-theoretic approach incentivizes robots to specialize. For example, if only a subset of the fleet encounters snowy conditions, those robots are prioritized to use their upload budget for snow images, while others focus on different data gaps.
Conclusion and Future Work
The "Data Games" framework demonstrates that decentralized fleets can achieve global optimality in data collection with minimal information exchange. By sharing intended distributions rather than raw data, the system preserves bandwidth and privacy while maximizing model robustness.
Future research directions include optimizing the algorithm for extremely large fleets (thousands of robots) through sub-clustering, addressing the discrete nature of image selection (integer programming), and enhancing resilience against adversarial nodes or sensor failures.
Summary updated