Distributed Semi-supervised Fuzzy Regression with Interpolation Consistency Regularization

Ye Shi, Leijie Zhang, Zehong Cao, Muhammad Tanveer, Chin-Teng Lin

Abstract

Recently, distributed semi-supervised learning (DSSL) algorithms have shown their effectiveness in leveraging unlabeled samples over interconnected networks, where agents cannot share their original data with each other and can only communicate non-sensitive information with their neighbors. However, existing DSSL algorithms cannot cope with data uncertainties and may suffer from high computation and communication overhead problems. To handle these issues, we propose a distributed semi-supervised fuzzy regression (DSFR) model with fuzzy if-then rules and interpolation consistency regularization (ICR). The ICR can force decision boundaries to pass through sparse data areas, thus increasing model robustness. We proposed a distributed Fuzzy C-means (DFCM) method and a distributed interpolation consistency regularization (DICR) built on the well-known alternating direction method of multipliers to respectively locate parameters in antecedent and consequent components of DSFR. Notably, the DSFR model converges very fast since it does not involve back-propagation and is scalable to large-scale datasets. Experimental results show that the proposed DSFR model can achieve much better performance than state-of-the-art DSSL algorithms in terms of both loss value and computational cost.

Introduction

In the modern era of "Big Data," two major hurdles often stall machine learning progress: data privacy and data uncertainty. Data is often siloed across different organizations (agents) that cannot share raw datasets due to legal or security concerns. Distributed Semi-supervised Learning (DSSL) aims to learn from these silos, but existing methods often suffer from high communication overhead or struggle to scale. Furthermore, real-world data is often messy, containing noise from measurement errors or linguistic ambiguity.

The authors bridge these gaps by creating a distributed model that is both "fuzzy" (to handle uncertainty) and "semi-supervised" (to use unlabeled data via ICR). By leveraging the Alternating Direction Method of Multipliers (ADMM), the proposed DSFR framework allows multiple agents to collaboratively train a robust regression model without ever sharing raw, sensitive information.

Methodology

The DSFR model consists of two main stages, both optimized using ADMM to ensure all agents reach a consensus on the model parameters.

The DSFR framework architecture showing distributed agents achieving consensus through local FCM and ICR components.
  • Distributed Fuzzy C-means (DFCM): This handles the "antecedent" (the IF-part) of the fuzzy rules. It clusters data across the network to find global cluster centers without moving the data itself.
  • Distributed Interpolation Consistency Regularization (DICR): This handles the "consequent" (the THEN-part). It applies ICR, inspired by MixUp, creating "virtual" samples by interpolating between unlabeled points. This forces the model to make consistent predictions across these interpolations, pushing decision boundaries into low-density areas and increasing robustness.
  • Consensus Strategy: Instead of a central server, agents communicate non-sensitive "dual variables" and local estimates to their neighbors, eventually converging on a single global model.

Experimental Setup

The model was evaluated on one artificial dataset and five diverse real-world regression benchmarks:

  • Airfoil: NASA noise test data.
  • WMSL (WarCraft Master Skill Level): Player performance data.
  • CCPP (Combined Cycle Power Plant): Electrical power output prediction.
  • CH (California Housing): Median house value prediction.
  • KCH (King County Housing): Home sales prices in Seattle/King County.

Key Results

Experimental results demonstrate that DSFR consistently outperforms existing DSSL methods in both accuracy and efficiency.

Performance comparison of DSFR-ICR against other centralized and distributed methods across multiple datasets.
  • Superior Accuracy: DSFR-ICR consistently achieved lower Normalized Root Mean Square Error (NRMSE) compared to state-of-the-art algorithms like LapWNN. For example, on the Airfoil dataset, it significantly reduced NRMSE compared to other distributed methods.
  • Efficiency: The model converges very quickly—often within a few dozen iterations for clustering and under 300 for parameter learning. Because it uses closed-form solutions within the ADMM iterations rather than back-propagation, it maintains lower communication costs.
  • Scalability: The parallelized computation across agents allows the model to scale effectively to large datasets that would be cumbersome for centralized fuzzy systems.

Conclusion and Future Work

While DSFR is highly effective for handling uncertainty in distributed environments, fuzzy systems can struggle with the "curse of dimensionality" in very high-dimensional feature spaces. Future research will explore developing Deep DSFR variants to further improve performance on complex data and extending these methods to dynamic network topologies where agents may join or leave during training.

Summary updated

Abstract

Recently, distributed semi-supervised learning (DSSL) algorithms have shown their effectiveness in leveraging unlabeled samples over interconnected networks, where agents cannot share their original data with each other and can only communicate non-sensitive information with their neighbors. However, existing DSSL algorithms cannot cope with data uncertainties and may suffer from high computation and communication overhead problems. To handle these issues, we propose a distributed semi-supervised fuzzy regression (DSFR) model with fuzzy if-then rules and interpolation consistency regularization (ICR). The ICR, which was proposed recently for semi-supervised problem, can force decision boundaries to pass through sparse data areas, thus increasing model robustness. However, its application in distributed scenarios has not been considered yet. In this work, we proposed a distributed Fuzzy C-means (DFCM) method and a distributed interpolation consistency regularization (DICR) built on the well-known alternating direction method of multipliers to respectively locate parameters in antecedent and consequent components of DSFR. Notably, the DSFR model converges very fast since it does not involve back-propagation procedure and is scalable to large-scale datasets benefiting from the utilization of DFCM and DICR. Experiments results on both artificial and real-world datasets show that the proposed DSFR model can achieve much better performance than the state-of-the-art DSSL algorithm in terms of both loss value and computational cost.
BibTeX
@article{shi2022distributed,
  author = {Ye Shi and Leijie Zhang and Zehong Cao and Muhammad Tanveer and Chin-Teng Lin},
  title = {{Distributed Semi-supervised Fuzzy Regression with Interpolation Consistency Regularization}},
  journal = {IEEE Trans. Fuzzy Systems},
  year = {2022},
  doi = {10.1109/tfuzz.2021.3104339},
}
Figure