Abstract
Decentralized learning often fails to spread "out-of-distribution" (OOD) knowledge across a network, leaving most nodes ignorant of rare but important data. This paper introduces topology-aware aggregation strategies—weighting neighbors by their network centrality—to boost OOD knowledge propagation by up to 123% without sacrificing performance on standard data.
The Problem: The OOD Propagation Bottleneck
In decentralized learning (DL), there is no central server. Instead, devices (nodes) only communicate with their immediate neighbors in a graph. For a node to learn from a distant peer, that "knowledge" (model weights) must hop through the network via successive rounds of aggregation.
Existing algorithms are typically "topology-unaware," meaning they treat all neighbors equally or weight them only by local data size. The authors identify a major bottleneck: OOD knowledge propagation. While standard "IID" data (data common across the network) spreads easily, unique or niche data (OOD) often gets stuck near its source. If one sensor in a smart grid detects a rare fault, the rest of the network may fail to learn to recognize that fault quickly because the signal is "washed out" during the hopping process.
Methodology: Topology-Aware Aggregation
The paper proposes moving away from uniform weighting in favor of Topology-Aware Aggregation. When a node aggregates models from its neighbors, it assigns weights based on graph theory metrics:
- Degree Centrality (Local): Weighting neighbors by how many connections they have. High-degree nodes act as local "hubs" that can quickly ingest and disperse information.
- Betweenness Centrality (Global): Weighting neighbors by how often they lie on the shortest path between other pairs of nodes. These nodes act as "bridges" between different clusters in the network.
The authors use a Softmax with temperature ($\tau$) to turn these centrality scores into aggregation coefficients. They evaluated these strategies across five datasets, including vision tasks (MNIST, CIFAR10/100) and a synthetic language dataset (TinyMem) designed to test memorization. To simulate OOD data, they used a "backdoor" approach where a small percentage of data on a single node was modified with a specific trigger.
Key Results and Insights
The researchers found that the propagation of knowledge is highly sensitive to where the OOD data starts. If it originates in a peripheral node, the network may never learn it. However, nodes with high centrality act as "Information Super-spreaders."
- Massive Accuracy Gains: Topology-aware strategies improved OOD data accuracy by 123% on average compared to topology-unaware baselines.
- No Trade-offs: These gains did not come at the expense of IID accuracy; in many cases, IID performance also improved.
- Structural Impact: The benefits are most pronounced in "scale-free" networks where a few nodes have many connections. Conversely, modularity (isolated clusters) is a major enemy of knowledge spread.
Limitations and Future Work
While promising, the study notes several challenges for real-world deployment:
- Static Topologies: The current study assumes a fixed network structure, whereas real-world swarms or mobile networks are dynamic.
- Global Knowledge: Calculating metrics like Betweenness Centrality requires knowing the entire graph structure, which may not be feasible in fully decentralized systems.
- Security: Faster propagation of "good" knowledge could also mean faster spread of malicious data or poisoning attacks.
Future research will explore other centrality metrics, such as Eigenvector centrality, and test these strategies in online, streaming data environments.
Summary updated