Abstract
In decentralized federated learning (DFL), devices collaborate to train models without a central server, mitigating bottlenecks and privacy risks. However, DFL faces significant challenges regarding communication overhead and the difficulty of monitoring model convergence across a peer-to-peer network. This paper presents a dual-pronged approach: Dynamic Communication Cost Reduction (DCCR) to minimize data transfer, and a visualization technique using Uniform Manifold Approximation and Projection (UMAP) to monitor convergence in function space. Experimental results on Fashion-MNIST and CIFAR-10 demonstrate that DCCR reduces communication costs by approximately 72% while maintaining model accuracy, and the UMAP-based visualizer provides qualitative proof of consensus among decentralized nodes.
1. Introduction
Federated Learning (FL) has emerged as a standard for privacy-preserving machine learning, typically relying on a central server to aggregate updates. Decentralized Federated Learning (DFL) removes this central point of failure, allowing devices to communicate directly with neighbors. While DFL offers increased robustness and privacy, it introduces two primary hurdles: the high cost of peer-to-peer communication and the "black box" nature of decentralized convergence. Unlike centralized FL, where a global model serves as a benchmark, DFL lacks a clear metric to verify if disparate models are reaching a consensus.
2. Methodology
The proposed framework builds upon Consensus-based Multi-hop Federated Distillation (CMFD). In Federated Distillation (FD), devices share model predictions (distilled values) on a common dataset rather than sharing large weight matrices.
2.1 Dynamic Communication Cost Reduction (DCCR)
To address communication overhead, the researchers introduced DCCR, which employs an "annealing" strategy for data sharing. At the beginning of the training process, devices exchange predictions for only a small subset of the shared dataset. As training progresses and model accuracy improves, the number of shared samples is gradually increased. This strategy suppresses traffic during the early stages when models are less refined and focuses communication resources when high-precision distillation is most beneficial.
To synchronize the selection of these samples without additional communication, the system utilizes a Pseudo-Random Number Generator (PRNG) with a pre-shared seed. This allows all devices to independently identify the same data indices for distillation at each round, effectively achieving "zero-cost" synchronization.
2.2 Function Space Visualization
Monitoring convergence in a decentralized network is challenging because neural network parameters are high-dimensional and non-convex. The authors propose visualizing the "function space"—the mapping of inputs to outputs—rather than the parameter space. By treating a model's outputs on a shared dataset as a high-dimensional vector, the researchers apply UMAP to project these vectors into a 2D or 3D space. This allows for real-time observation of model trajectories; as the points representing different devices cluster together, it provides visual evidence of model convergence.
3. Experimental Evaluation
The methodology was tested using a ring lattice topology consisting of 10 devices. Each device was assigned a non-IID (Independent and Identically Distributed) subset of the Fashion-MNIST or CIFAR-10 datasets, simulating a realistic and challenging decentralized environment.
3.1 Results
The experiments revealed that DCCR significantly optimizes communication efficiency. For the Fashion-MNIST dataset, DCCR achieved a target accuracy of 70% using only 84 MB of data. In comparison, the standard CMFD method required 300 MB to reach the same threshold—a reduction of approximately 72%.
Furthermore, the UMAP visualization confirmed that while models start with divergent predictions due to their local data distributions, they eventually move toward a tight cluster in the function space. This confirms that the decentralized distillation process successfully drives the models toward a functional consensus. However, the researchers noted that if communication is overly restricted (e.g., maintaining a constant 20% sample rate), the models fail to converge in the function space, highlighting the importance of the dynamic increase in DCCR.
4. Discussion and Conclusion
A key insight from this research is that while neural network parameters are difficult to optimize collectively, the task of aligning model functions can be treated as a convex optimization problem. This theoretical grounding explains why distillation is particularly effective for decentralized consensus.
While the current study demonstrates significant progress in communication efficiency and transparency, it focuses on static network topologies. Future research should address the challenges of dynamic networks where devices may join or leave (churn). Additionally, while UMAP provides excellent visualization, its computational requirements suggest a need for lighter-weight monitoring tools suitable for resource-constrained edge devices.
In summary, the combination of DCCR and UMAP visualization offers a more efficient and transparent path for decentralized AI, proving that leaner communication does not have to come at the cost of performance or visibility.
Summary updated