FedE2E: One-Shot Federated Learning for End-to-End Autonomous Driving with Closed-Loop Evaluation

Tao Lian, Antonio M. López

TL;DR

FedE2E is a novel "one-shot" federated learning framework that builds a high-performance autonomous driving model by distilling knowledge from multiple pre-trained client models in a single communication round. By aligning disparate feature spaces and using a token-wise consensus mechanism, it achieves state-of-the-art closed-loop driving performance while drastically reducing the communication overhead typical of collaborative training.

What Problem Does This Paper Address?

End-to-end (E2E) autonomous driving—where a neural network maps raw sensor data directly to control commands—requires massive amounts of diverse data. However, privacy concerns and data regulations often prevent organizations from sharing their local datasets.

Federated Learning (FL) offers a solution by allowing models to be trained collaboratively without moving raw data. But standard FL has two major drawbacks for real-world driving:

  1. Communication Overhead: It usually requires hundreds of rounds of back-and-forth communication between a central server and clients, which is bandwidth-intensive and impractical if clients can only share their models once.
  2. Evaluation Gap: Most FL research relies on "open-loop" metrics (like how well a model predicts a recorded trajectory). In reality, driving is "closed-loop"—errors accumulate over time as the car interacts with the environment. A model that looks good on paper might crash in a simulator.

Furthermore, models trained in different cities (Non-IID data) develop different internal "languages" (feature representations), making them difficult to merge effectively.

Methods

The authors propose FedE2E, a framework that treats uploaded client models as "teachers" to train a single "student" (global) model on the server. The process happens in three main stages:

  1. Multi-client Calibration (MCC): Client models often have Batch Normalization (BN) layers tuned to their specific local cities. FedE2E uses a small, unlabeled server-side dataset to "re-calibrate" these BN statistics, ensuring all teacher models are speaking the same statistical language before distillation.
  2. Anchor-based Alignment (ABA): Even with calibration, different models might represent the same concept (e.g., a "pedestrian") using different numerical patterns. ABA uses the initial student model as an "anchor" and applies an orthogonal transformation (via Singular Value Decomposition) to map all teacher features into a shared representation space.
  3. Token-wise Consensus Distillation (TCD): Instead of just averaging the teachers' outputs, FedE2E looks at individual "tokens" (parts of the feature map). If most teachers agree on a specific feature, it's given a higher weight. This "consensus" approach ensures the student learns the most reliable, shared knowledge across the fleet.
Fig. 1: Overview of the proposed FedE2E framework. In our setting, clients can be companies, research institutions, or model repositories that independently train models on their local datasets and upload only model weights to the server once, typically without participating in subsequent training. Once the client models are uploaded to the server, our framework distills knowledge from these models through multi-client calibration, anchor-based alignment, and token-wise consensus distillation to obtain a global model, which is subsequently evaluated in the CARLA simulator for closed-loop evaluation.

Datasets

The researchers evaluated FedE2E using the Bench2Drive benchmark within the CARLA 0.9.15 simulator.

  • Training Data: They used four distinct maps (Town03, Town12, Town13, and Town15) to simulate a Non-IID federated setting where each client has data from only one town.
  • Evaluation: The models were tested across 220 evaluation routes, focusing on complex scenarios like non-signalized junctions, pedestrian crossings, and highway exits.

Key Insights and Ideas

  • One-Shot is Enough: The paper demonstrates that you don't need constant communication to build a good model. A single upload of weights, if handled with sophisticated distillation, can outperform traditional multi-round methods.
  • The Power of Alignment: The ablation study showed that Anchor-based Alignment provided the single largest boost in performance (improving the Driving Score by over 5 points). This highlights that the "representation gap" is the biggest hurdle in merging heterogeneous models.
  • Consensus over Averaging: Simple parameter averaging (like FedAvg) often leads to "feature wash-out" in one-shot settings. By weighting features based on teacher agreement, FedE2E preserves sharp, actionable driving knowledge.
Fig. 2: Overview of token-wise consensus distillation. The server dataset is first fed into the calibrated teacher models and the student model to extract features. The teacher features are then mapped into the student feature space using the optimal orthogonal transformation matrices. Next, token-wise consensus is used to compute adaptive weights for teacher feature fusion. Finally, the student model is trained by minimizing the distillation loss between the fused teacher features and student features, together with the supervised driving action loss.

Key Results

FedE2E sets a new bar for one-shot federated driving:

  • Driving Score (DS): In the one-shot setting, it achieved a 55.85 DS, outperforming the previous best method (FedDF) by 4.07 points.
  • Success Rate (SR): It reached a 24.7% success rate, a significant jump over the 20.6% of prior work.
  • Multi-round Efficiency: When allowed five rounds of communication, FedE2E reached a 60.35 DS. Notably, it achieved a better score in just 3 rounds than other methods did in 5, representing a 40% reduction in required communication.
  • Safety: It matched the Infraction Penalty (IP) of a centralized model (0.78), indicating it drives as safely as a model trained on all data combined.

Limitations and Future Work

While FedE2E is a major step forward, the authors note a few areas for growth:

  • Architecture Scaling: The current work uses a ResNet34 backbone. Testing how these distillation techniques scale to larger Vision Transformers (ViTs) or more complex multi-modal architectures is a logical next step.
  • Diverse Environments: While Bench2Drive is rigorous, testing on even more diverse, real-world-inspired maps would further validate the model's generalization capabilities.
  • Server-side Data: The method relies on a small proxy dataset at the server for distillation. Exploring "data-free" distillation methods could make the framework even more flexible.

Summary updated

Abstract

Federated learning enables collaborative model training without sharing datasets. However, existing federated learning methods for autonomous driving typically require multi-round communication and lack closed-loop evaluation, while one-shot methods mainly focus on perception tasks. To address these limitations, we propose \method, a one-shot federated learning method for end-to-end autonomous driving. It comprises multi-client calibration, anchor-based feature alignment, and token-wise consensus distillation to distill knowledge from heterogeneous models. Closed-loop evaluation on Bench2Drive demonstrates the effectiveness of \method in both one-shot and multi-round federated learning. In the one-shot setting, it achieves a Driving Score of 55.85, which is 4.07 points higher than prior work. In the multi-round setting, it reaches 60.35 after five communication rounds, the best performance among all compared methods under the same number of communication rounds.
BibTeX
@misc{lian2026fede,
  author = {Tao Lian and Antonio M. López},
  title = {{FedE2E: One-Shot Federated Learning for End-to-End Autonomous Driving with Closed-Loop Evaluation}},
  year = {2026},
  eprint = {2611.24254},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24254},
}
Figure