Standard federated learning relies on a simple but rigid assumption: every participating device must train the exact same neural network architecture. In algorithms like Federated Averaging (FedAvg), a central server aggregates local updates by performing element-wise averaging on the models' weight matrices.
This parameter-space aggregation creates a severe bottleneck. In the real world, edge devices are highly heterogeneous. A powerful cloud server or a modern smartphone can easily train a large model with billions of parameters, while a low-power IoT sensor might struggle with a tiny convolutional network. Forcing a "one-size-fits-all" architecture either excludes resource-constrained devices or forces capable devices to downscale to the lowest common denominator. Furthermore, organizations often want to keep their local model architectures proprietary.
To break this constraint, a powerful paradigm shift has emerged: moving collaboration out of the parameter space and into the functional space.
Standard Federated Learning (Parameter Space):
[Client A: Weights] ---\
[Client B: Weights] ----> [Server: Average Weights] (Requires identical architectures)
[Client C: Weights] ---/
Functional Space Federated Learning:
[Client A: Architecture 1] ---> [Predictions on Reference Data] ---\
[Client B: Architecture 2] ---> [Predictions on Reference Data] ----> [Server: Ensemble Distillation]
[Client C: Architecture 3] ---> [Predictions on Reference Data] ---/
Shifting to the Functional Space
The core intuition behind functional space collaboration is that two neural networks do not need to share the same internal structure to share knowledge. Instead of exchanging raw weight matrices, models can communicate by showing each other what they think about a common set of reference points.
By exchanging model outputs—such as prediction probabilities, logits, or low-dimensional embeddings—clients can transfer their learned expertise without exposing or aligning their internal parameters. As explained by Laiqiao Qin et al. in Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutions, focusing on output alignment rather than parameter synchronization allows a diverse ecosystem of devices to participate in the same training federation regardless of their specific model structures.
This functional alignment is typically achieved through Knowledge Distillation (KD), where the predictions of one or more "teacher" models serve as soft targets to train a "student" model.
Establishing Common Ground via Reference Data
To compare predictions and align functional behavior, heterogeneous models must evaluate the same data points. Because sharing private client data violates the core premise of federated learning, researchers use alternative data sources to serve as a translation layer.
Unlabeled Public Datasets
The most straightforward approach uses an auxiliary, unlabeled public dataset that roughly reflects the data domain. During a communication round, clients pass this public data through their unique local models to generate soft labels (logits).
In Ensemble Distillation for Robust Model Fusion in Federated Learning, Tao Lin et al. demonstrate how a central server can use these client-generated predictions to train a unified global student model via ensemble distillation. Similarly, Yae Jee Cho et al. introduce a weighted consensus distillation scheme in Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning, enabling resource-constrained clients to train small, diverse models while the server builds a single high-capacity model.
Synthetic and Generative Data
When a suitable public dataset is unavailable, generative models can synthesize the necessary reference data. The authors of Unlocking the Potential of Federated Learning: The Symphony of Dataset Distillation via Deep Generative Latents show that dataset distillation can decouple local client models from the global server model. By generating synthetic data trajectories in a shared latent space, small surrogate networks on mobile devices can effectively inform the training of much larger models on the server.
Taking this a step further, Xingjian Cao et al. propose in Personalized Federated Learning via Generative Adversarial Networks to exchange synthetic data samples directly, treating local models as complete black boxes and allowing for total architectural heterogeneity across the network.
Architectural Bridges
Instead of relying entirely on external reference datasets, some frameworks introduce lightweight architectural intermediaries that act as translators between disparate models.
Shared Auxiliary Models
One popular strategy involves co-training a small, standardized auxiliary model alongside each client's unique, hardware-optimized local architecture. As explored by Liping Yi et al. in Federated Model Heterogeneous Matryoshka Representation Learning, the server aggregates updates exclusively from these small shared models, while clients use local mutual distillation to transfer knowledge between the shared model and their proprietary architectures.
Proxy Models and Adapters
In highly asymmetrical environments—such as edge clients collaborating with a massive foundation model—clients can maintain a compact proxy model. Jiaqi Wang and Ziyi Yin leverage this in Asymmetrical Reciprocity-based Federated Learning for Resolving Disparities in Medical Diagnosis, using a dual-distillation strategy where a large primary model transfers expertise to a proxy, which then handles the federated aggregation process.
Alternatively, Gautam Jajoo proposes in Federated Learning with Heterogeneous LLMs: Integrating Small Student Client Models with a Large Hungry Model a mechanism to map low-dimensional weight updates from small client models directly into the higher-dimensional space of a large server model, creating a direct mathematical bridge between different model scales.
Why Functional Space Collaboration Works
Functional space collaboration succeeds because it separates what a model knows from how its architecture is structured.
- The Richness of Dark Knowledge: Standard training relies on hard labels (e.g., a picture is either a "cat" or a "dog"). Functional distillation exchanges soft labels—the full probability distribution across all classes. If a model outputs that an image is 70% cat, 25% dog, and 5% car, that distribution contains rich geometric information about how the model structures its decision boundaries. This "dark knowledge" allows student models to learn complex relationships much faster than they would from raw data alone.
- Extreme Communication Efficiency: Transmitting the weights of modern deep networks requires gigabytes of bandwidth. In contrast, transmitting prediction vectors over a compact reference dataset requires orders of magnitude less data. Felix Sattler demonstrates this efficiency in CFD: Communication-Efficient Federated Distillation via Soft-Label Quantization and Delta Coding, showing that exchanging quantized soft labels instead of raw parameters can slash communication overhead by up to four orders of magnitude while maintaining competitive accuracy.
- Robustness to Non-IID Data: When local data distributions are highly skewed (non-IID), averaging weights often leads to "gradient cancellation," where conflicting updates degrade the global model. Functional aggregation avoids this by treating client models as an ensemble of experts. The server aggregates their predictions rather than their internal mechanics, preserving specialized local insights while building a robust global consensus.