Abstract
Partition-wise models offer a flexible approach for modeling complex and multidimensional data and are capable of producing interpretable results. They are based on partitioning the observed data into regions, each of which is modeled with a simple submodel. The success of this approach highly depends on the quality of the partition, as too large a region could lead to a nonsimple submodel, while too small a region could inflate estimation variance. This paper proposes an automatic procedure for choosing the partition (i.e., the number of regions and the boundaries between regions) as well as the submodels for the regions. It is shown that under the assumption of the existence of a true partition, as well as a set of known significant predictors, the proposed partition estimator is statistically consistent. Features of the proposed methodology are highlighted for both regression and classification problems on synthetic and real data.
Introduction: Balancing Complexity and Interpretability
In contemporary data science, practitioners often face a trade-off between the interpretability of simple linear models and the predictive power of complex "black-box" architectures like deep neural networks. Partition-wise models provide a compelling middle ground by dividing the feature space into distinct regions and applying simple, interpretable submodels (such as linear regression) within each.
However, two primary challenges persist in this domain. First is the "Goldilocks Problem": determining the optimal size and number of regions to avoid both high bias (underfitting) and high variance (overfitting). Second is the lack of theoretical guarantees; while methods like Classification and Regression Trees (CART) are popular, proving that their estimated boundaries converge to the true underlying structure as data increases is mathematically difficult.
Methodology: MDL and Grid-Based Partitioning
The authors address these challenges by framing model selection as a data compression problem using the Minimum Description Length (MDL) principle. The "best" model is defined as the one that provides the shortest total description of the data, balancing model complexity (number of regions and variables) against the description of residuals (error).
Unlike recursive splitting methods, this framework utilizes a grid-based partitioning structure. When a change point is detected on a specific predictor, it influences the entire data space along that dimension, facilitating more rigorous mathematical analysis and enabling the proof of statistical consistency. To navigate the complex optimization landscape of the MDL criterion, the authors employ Binary Particle Swarm Optimization (BPSO), a metaheuristic inspired by collective biological behaviors.
Key Insights and Empirical Performance
A significant contribution of this work is the theoretical proof of consistency: if a true underlying partition exists, the MDL estimator is guaranteed to identify it as the sample size $n$ approaches infinity. Furthermore, the model performs automatic variable selection within each region, recognizing that a predictor's relevance may vary across different parts of the data space.
The methodology was validated on several datasets:
- Concrete Compressive Strength: In this regression task, the proposed method achieved a Root Mean Squared Error (RMSE) of 6.41, outperforming classical regression (10.52) and regression trees (12.24). It approached the accuracy of neural networks (5.97) while maintaining high interpretability.
- Red Wine Quality: For this classification task, the logistic version of the model achieved a misclassification error rate of 0.149, the lowest among all tested methods, including CART (0.214) and Neural Networks (0.202).
Conclusion and Future Directions
The proposed framework demonstrates that it is possible to achieve state-of-the-art predictive performance without sacrificing interpretability or theoretical rigor. While the current grid-based constraint facilitates consistency proofs, future work may explore more flexible partitioning schemes. Additionally, extending these proofs to a broader range of Generalized Linear Models (GLMs), such as Poisson regression for count data, remains a promising avenue for research.
Summary updated