Abstract: Non-negative matrix factorization (NMF) based topic modeling is widely used in natural language processing (NLP) to uncover hidden topics of short text documents. Usually, training a high-quality topic model requires large amount of textual data. In many real-world scenarios, customer textual data should be private and sensitive, precluding uploading to data centers. This paper proposes a Federated NMF (FedNMF) framework, which allows multiple clients to collaboratively train a high-quality NMF based topic model with locally stored data. However, standard federated learning will significantly undermine the performance of topic models in downstream tasks (e.g., text classification) when the data distribution over clients is heterogeneous. To alleviate this issue, we further propose FedNMF+MI, which simultaneously maximizes the mutual information (MI) between the count features of local texts and their topic weight vectors to mitigate the performance degradation. Experimental results show that our FedNMF+MI methods outperform Federated Latent Dirichlet Allocation (FedLDA) and the FedNMF without MI methods for short texts by a significant margin on both coherence score and classification F1 score.
Introduction
Topic modeling is a fundamental NLP task used to discover latent themes in document collections. Traditionally, this requires centralizing data, which increasingly conflicts with privacy regulations like GDPR. Federated Learning (FL) offers a decentralized alternative by training models locally on user devices. However, standard FL algorithms like FedAvg struggle with "non-IID" (Independent and Identically Distributed) data—where different users have disparate writing styles or topics. Furthermore, while Latent Dirichlet Allocation (LDA) is a standard for long-form text, it often fails on short texts (e.g., tweets or SMS), where Non-negative Matrix Factorization (NMF) is typically more effective.
The FedNMF+MI Framework
The researchers propose FedNMF, adapting NMF into a federated setting using Stochastic Gradient Descent (SGD). To address the performance degradation caused by data heterogeneity, they introduce FedNMF+MI.
Key Technical Components
- Matrix Factorization: The term-document matrix ($A$) is decomposed into two smaller matrices: $W$ (the topic dictionary) and $H$ (topic weights per document).
- Mutual Information (MI) Regularization: This core innovation maximizes the MI between raw text features and learned topic weights. This forces the compressed topic representation ($H$) to retain maximum information from the original text ($A$), acting as an anchor against local data bias.
- SMILE Estimator: Since direct MI calculation is mathematically intractable, the framework uses the Statistical Mutual Information Least-square Estimator (SMILE) to approximate it via a small neural network.
- Aggregation: The central server collects topic matrices ($W$) and MI-estimator parameters ($\theta$) from clients, averages them, and redistributes them for subsequent local training rounds.
Experimental Evaluation
The framework was evaluated on four diverse short-text datasets: Spam Ham (SMS), Tag.News (news snippets/tweets), Rt Polarity (movie reviews), and Stack Overflow (technical questions).
Key Insights
- NMF Superiority: Results confirm that NMF-based methods consistently outperform LDA-based methods (like FedLDA) for documents averaging under 30 words.
- MI as a Stabilizer: Mutual Information prevents local models from drifting into "nonsense" territory even when local datasets are highly biased.
- Faster Convergence: The MI regularizer significantly accelerates convergence, reducing the necessary communication rounds between server and clients—a critical factor for bandwidth-constrained mobile applications.
Key Results
FedNMF+MI was compared against several baselines, including FedLDA and standard FedNMF.
- Higher Coherence and Accuracy: FedNMF+MI achieved significantly higher topic coherence and classification scores. On the Tag.News dataset, FedNMF+MI reached an F1 score of 0.973, while FedLDA lagged at 0.508.
- Robustness to Heterogeneity: Even in extreme non-IID settings where each client holds only one category of data, the MI-enhanced model maintained high performance while baseline models failed.
Conclusion and Future Work
While FedNMF+MI effectively addresses data heterogeneity in short-text topic modeling, future research may explore the computational overhead of the SMILE estimator on low-power IoT devices and the integration of explicit Differential Privacy (DP) mechanisms to further secure model updates against gradient leakage attacks.
Summary updated