TL;DR
This paper performs a rigorous "reproducibility audit" of Bayesian Takagi–Sugeno–Kang (TSK) fuzzy regression, identifying a critical implementation flaw—the "membership-spread bug"—that previously led to numerical instability. After rebuilding the pipeline with corrected code and evaluating it on both standard benchmarks and real-world building energy data, the author finds that while sophisticated spike-and-slab model averaging is a valid and reproducible tool for uncertainty quantification, it does not provide a significant accuracy or calibration advantage over simpler dense TSK baselines or even basic persistence models.
What Problem Does This Paper Address?
In the world of soft computing, Takagi–Sugeno–Kang (TSK) fuzzy systems are popular because they combine the interpretability of "if-then" rules with the mathematical power of linear regression. However, the field often struggles with "silent" implementation errors that undermine reproducibility.
The author identifies a specific, subtle bug: recomputing membership function spreads during the inference phase rather than "freezing" them at training time. This mismatch changes the underlying model representation after the coefficients have already been fitted, leading to unstable predictions and unfair comparisons. Furthermore, there is a need to understand how Bayesian methods for rule selection (like spike-and-slab priors) actually perform when these implementation details are handled correctly, especially in data-limited fields like building energy modeling where uncertainty estimates are crucial for decision-making.
Methods
The paper revisits first-order TSK regression and introduces a corrected, auditable pipeline. Key methodological features include:
- Frozen Partitions: Ensuring that Gaussian membership function centers and spreads are stored during training and reused verbatim during inference (Algorithm 1).
- Bayesian Estimators: The author implements a dense conjugate Bayesian baseline and a more complex rule-level spike-and-slab estimator. The latter uses a block-Gibbs sampler to determine which fuzzy rules are "active" (the slab) or "inactive" (the spike).
- Bayesian Model Averaging (BMA): Instead of picking one "best" rule set, the model averages predictions across multiple posterior draws, capturing both parameter and model-selection uncertainty.
- Log-Domain Evaluation: Firing strengths are calculated in the log domain to improve numerical stability.
- Comparison Baselines: The TSK methods are compared against standard machine learning references like Gaussian Processes (GP), Support Vector Regression (SVR), and Random Forests, as well as a "persistence" model (using the last observed value) for temporal data.
Datasets
The study utilizes two distinct types of data:
- Controlled Tabular Benchmarks: Two UCI regression datasets: Energy Efficiency (predicting heating and cooling loads for simulated buildings) and Concrete Compressive Strength.
- Measured Building Data: A sample from the Building Data Genome Project 2 (BDG2), featuring hourly electricity meter data from 158 buildings across 17 sites. This represents a challenging real-world scenario with temporal dependencies and missing data.
The paper introduces the "Route C" protocol for the BDG2 data, a standardized evaluation framework designed to ensure all models are tested on the exact same rows with the same training exposure.
Key Insights and Ideas
- Correctness is a Prerequisite for Performance: The "membership-spread bug" demonstrates that a model can be numerically plausible but scientifically invalid if its internal representation shifts between training and testing.
- Sparsity $\neq$ Accuracy: While spike-and-slab priors successfully identify active rules, this "rule selection" didn't actually improve point-prediction accuracy in the tested datasets. The dense TSK models performed just as well.
- Uncertainty Trade-offs: The paper highlights that different uncertainty objects (Bayesian posterior intervals vs. Conformal prediction intervals) answer different questions. Bayesian TSK intervals tended to "overcover" (providing 98% coverage for a 90% nominal interval), making them wider and more conservative than conformal alternatives.
- The "Persistence" Hurdle: In real-world building energy forecasting, simple persistence models are surprisingly hard to beat. The Bayesian TSK models failed to demonstrate a clear advantage over this baseline in the BDG2 common-unit evaluation.
Key Results
- UCI Benchmarks: On the Energy-Heating target, corrected TSK-LS and Bayesian TSK both achieved an $R^2$ of 0.957. However, non-fuzzy references like Gaussian Processes were stronger, reaching an $R^2$ of 0.998.
- Building Energy (BDG2): The persistence model achieved the lowest Root Mean Square Error (RMSE) of 27.710. Both TSK-LS and Bayesian TSK trailed slightly with an RMSE of 29.083.
- Calibration: For a 90% nominal interval, Bayesian TSK produced a Prediction Interval Coverage Probability (PICP) of 0.980 with a Mean Prediction Interval Width (MPIW) of 116.469. In contrast, the Ridge Conformal baseline was closer to the nominal target (PICP 0.901) with a much narrower width (MPIW 30.635).
Limitations and Future Work
The author acknowledges several bounds to this study:
- Scope of Sparsity: Sparsity was only applied to the rule consequents; joint selection of both antecedents and consequents remains an open area for this specific pipeline.
- Dataset Variety: The controlled evaluation was limited to two UCI datasets and a specific sample of BDG2. Results might differ in extremely high-dimensional settings.
- MCMC Convergence: While predictive diagnostics were stable, some raw coefficient diagnostics suggested that finite-chain uncertainty remains a factor in complex Gibbs sampling.
- Future Directions: The author suggests exploring alternative shrinkage priors, such as the Horseshoe or Spike-and-Slab LASSO, which might handle collinearity in TSK design matrices differently.
Summary updated