Distributed Semisupervised Fuzzy Regression With Interpolation Consistency Regularization

Ye Shi, Leijie Zhang, Zehong Cao, Muhammad Tanveer, Chin-Teng Lin

Abstract

In the modern data landscape, privacy concerns and data uncertainty often hinder the development of effective machine learning models. This paper presents a novel framework called Distributed Semi-supervised Fuzzy Regression (DSFR). By combining fuzzy logic to handle data ambiguity with Interpolation Consistency Regularization (ICR) to leverage unlabeled data, DSFR enables multiple decentralized agents to collaboratively train robust models without sharing raw, private data. Experimental results across various regression benchmarks demonstrate that DSFR significantly outperforms existing distributed learning methods in both accuracy and computational efficiency.

Introduction: The Privacy-Uncertainty Dilemma

Machine learning practitioners frequently face three major hurdles when working with real-world data:

  1. Privacy and Decentralization: Data is often siloed across different organizations (e.g., hospitals or financial institutions) that cannot share raw information due to legal and security constraints. Distributed Semi-supervised Learning (DSSL) is required to bridge these "islands" of data.
  2. Data Uncertainty: Real-world measurements are rarely "crisp"; they often contain errors or linguistic ambiguity. Standard distributed models struggle to account for this inherent fuzziness.
  3. The Unlabeled Data Problem: While labeled data is expensive and scarce, unlabeled data is abundant. Effectively utilizing this unlabeled data in a privacy-preserving, distributed manner without excessive communication overhead remains a significant challenge.

The DSFR Framework

The proposed DSFR framework utilizes a two-stage approach built upon the Alternating Direction Method of Multipliers (ADMM), a mathematical optimization technique that allows agents to reach a global consensus on model parameters through local computation and minimal information exchange.

The architecture of the Distributed Semi-supervised Fuzzy Regression (DSFR) framework, illustrating the interaction between local agents and the consensus mechanism.

Core Components

  • Distributed Fuzzy C-Means (DFCM): This component handles the "antecedent" (the "IF" part) of the fuzzy rules. It clusters data across all nodes to identify global patterns and the underlying shape of the data distribution without moving the raw data itself.
  • Distributed Interpolation Consistency Regularization (DICR): This is the framework's primary innovation, handling the "consequent" (the "THEN" part). Inspired by "MixUp" techniques, it creates virtual samples by interpolating between unlabeled data points. The model is then forced to produce consistent predictions for these virtual points, effectively smoothing decision boundaries and improving generalization in low-density data regions.
  • Fuzzy Takagi-Sugeno (T-S) Model: Unlike standard neural networks, the T-S model uses fuzzy if-then rules, which are naturally suited for handling the gray areas and uncertainties present in complex datasets.

Experimental Evaluation

The DSFR model was evaluated against several state-of-the-art algorithms using one synthetic dataset and five real-world regression benchmarks, including NASA's Airfoil Self-Noise data, California Housing prices, and game telemetry from WarCraft Master Skill Level (WMSL).

Key Results

As shown in the performance comparison below, the proposed SFR-ICR method consistently achieves superior accuracy across all tested environments.

Performance comparison of centralized and distributed methods. The proposed SFR-ICR method (bolded) shows superior NRMSE across various datasets.
  • Superior Accuracy: On the Airfoil dataset, the DSFR-ICR model achieved a Test NRMSE (Normalized Root Mean Square Error) of 0.7506, vastly outperforming the baseline LapWNN (1.4428).
  • Significant Error Reduction: Compared to other distributed fuzzy methods like SFR-G, the ICR-based approach reduced NRMSE values by an average of 19.44% to 23.19% across all benchmarks.
  • Computational Efficiency: Because the model utilizes closed-form solutions for weights rather than iterative backpropagation, it converges rapidly. The DFCM component typically stabilizes in fewer than 25 iterations, while the DICR component reaches consensus within 300 iterations.

Conclusion and Future Work

DSFR provides a robust solution for privacy-preserving regression in uncertain environments. By leveraging ICR, the framework effectively utilizes unlabeled data to boost performance beyond what is possible with traditional distributed fuzzy systems.

While the current model performs excellently on standard regression tasks, the authors note that "shallow" fuzzy systems can encounter difficulties with extremely high-dimensional data, such as raw high-resolution images. Future research will focus on developing a "Deep DSFR" variant and extending the framework to handle more complex, non-standard network topologies.

Summary updated

Abstract

Recently, distributed semisupervised learning (DSSL) algorithms have shown their effectiveness in leveraging unlabeled samples over interconnected networks, where agents cannot share their original data with each other and can only communicate nonsensitive information with their neighbors. However, existing DSSL algorithms cannot cope with data uncertainties and may suffer from high computation and communication overhead problems. To handle these issues, we propose a distributed semisupervised fuzzy regression (DSFR) model with fuzzy if-then rules and interpolation consistency regularization (ICR). The ICR, which was proposed recently for semisupervised problem, can force decision boundaries to pass through sparse data areas, thus increasing model robustness. However, its application in distributed scenarios has not been considered yet. In this work, we proposed a distributed fuzzy C-means (DFCM) method and a distributed interpolation consistency regularization (DICR) built on the well-known alternating direction method of multipliers to respectively locate parameters in antecedent and consequent components of DSFR. Notably, the DSFR model converges very fast since it does not involve back-propagation procedure and is scalable to large-scale datasets benefiting from the utilization of DFCM and DICR. Experiments results on both artificial and real-world datasets show that the proposed DSFR model can achieve much better performance than the state-of-the-art DSSL algorithm in terms of both loss value and computational cost. Our code is available online. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
BibTeX
@article{shi2022distributed,
  author = {Ye Shi and Leijie Zhang and Zehong Cao and Muhammad Tanveer and Chin-Teng Lin},
  title = {{Distributed Semisupervised Fuzzy Regression With Interpolation Consistency Regularization}},
  journal = {IEEE Trans. Fuzzy Systems},
  year = {2022},
  doi = {10.1109/tfuzz.2021.3104339},
}
Figure