Abstract
Federated Learning (FL) allows a number of agents to participate in training a global machine learning model without disclosing locally stored data. Compared to traditional distributed learning, the heterogeneity (non-IID) of the agents slows down the convergence in FL. Furthermore, many datasets, being too noisy or too small, are easily overfitted by complex models, such as deep neural networks. Here, we consider the problem of using FL regression on noisy, hierarchical and tabular datasets in which user distributions are significantly different. Inspired by Latent Class Regression (LCR), we propose a novel probabilistic model, Hierarchical Latent Class Regression (HLCR), and its extension to Federated Learning, FEDHLCR. FEDHLCR consists of a mixture of linear regression models, allowing better accuracy than simple linear regression, while at the same time maintaining its analytical properties and avoiding overfitting. Our inference algorithm, being derived from Bayesian theory, provides strong convergence guarantees and good robustness to overfitting. Experimental results show that FEDHLCR offers fast convergence even in non-IID datasets.
Introduction: The Challenge of Heterogeneous Hierarchical Data
Federated Learning (FL) enables collaborative model training across decentralized agents without raw data exchange. However, real-world applications often face "non-IID" data, where distributions vary significantly between users. A critical, often overlooked aspect is the hierarchical structure of data. For instance, in IoT scenarios, an agent (e.g., a smartphone) interacts with multiple entities (e.g., Wi-Fi routers), each generating numerous events. Standard FL methods struggle with this "agent-entity-event" hierarchy, often leading to overfitting on noisy or small datasets when using complex models like Deep Neural Networks (DNNs).
Methods: Hierarchical Latent Class Regression
To address these challenges, the authors introduce Hierarchical Latent Class Regression (HLCR) and its federated extension, FEDHLCR.
Model Evolution and Context
HLCR draws inspiration from Latent Dirichlet Allocation (LDA) and Latent Class Regression (LCR). While LCR clusters data points into linear regression models, HLCR introduces a hierarchical layer to better capture the dependencies within entities.
The HLCR Architecture
The core idea of HLCR is to treat data as a mixture of latent classes, where each class is represented by a simple linear regression model. This approach provides a non-linear global model that remains mathematically tractable and interpretable.
- Hierarchy: The model assumes that all events for a specific entity belong to the same latent cluster, helping the model better organize hierarchical relationships.
- Bayesian Inference via Collapsed Gibbs Sampling (CGS): Instead of standard gradient descent, the authors use a Bayesian approach. CGS allows them to "integrate out" certain variables, focusing only on assigning data to clusters. This provides strong convergence guarantees and makes the model robust against noise.
- Computational Efficiency: To make this work on edge devices, they utilize the Woodbury matrix identity, reducing the computational cost of updates from a cubic factor ($O(F^3)$) to a quadratic one ($O(F^2)$) relative to the number of features $F$.
Experimental Evaluation
The researchers evaluated their model using two primary sources:
- SYNTHHLCR: A custom synthetic dataset designed to simulate various levels of noise, cluster counts, and hierarchical densities.
- FEMNIST (Federated Extended MNIST): A standard benchmark containing handwritten characters organized by the writer (agent). They adapted this for a binary classification task (distinguishing letters from digits) treated as a regression problem.
Key Results and Insights
- Rapid Convergence: In synthetic tests, the model typically converged in fewer than 10 iterations, even with high data heterogeneity.
- Superior Accuracy: On the FEMNIST dataset, FEDHLCR (using 4 or 8 clusters) achieved an AUC of approximately 97%, significantly outperforming both standard linear regression (~94.7%) and a baseline CNN-only approach (~94.7%).
- Clustering within Agents: Unlike many FL methods that cluster agents, FEDHLCR allows a single agent to hold data belonging to multiple different clusters. This is much more realistic for users who interact with diverse environments.
- The "Sweet Spot" of Complexity: By using a mixture of linear models, the authors find a middle ground between the simplicity of linear regression and the "black box" complexity of neural networks. This prevents the model from "memorizing" noise in small datasets.
Conclusion and Future Work
FEDHLCR offers a robust solution for federated regression on hierarchical tabular data. The authors acknowledge areas for growth, such as extending clusters to non-linear models, integrating Differential Privacy for enhanced security, and exploring "soft" assignments where an entity could span multiple clusters.
Summary updated