Abstract Gradient Boosting (GB) typically relies on fixed-depth decision trees, which often struggle to balance high-resolution detail with noise robustness. This paper introduces Geometric Wavelets Gradient Boosting (GWGB), a novel approach that decomposes decision trees into geometric wavelets. By pruning based on a mathematical "wavelet norm" rather than hierarchical depth, GWGB allows for adaptive, non-contiguous selection of significant nodes. Experimental results demonstrate that GWGB significantly outperforms standard methods on imbalanced datasets and shows superior robustness to label noise.
1. Introduction
Gradient Boosting is a cornerstone of modern machine learning for structured data. However, standard implementations often use rigid pruning strategies—typically limiting trees to a fixed depth between 4 and 8 levels. This creates a "resolution gap" where rare class features deep in a tree are discarded, or a "noise sensitivity" where mislabeled data creates overfitting spikes. The authors of "Wavelet Decomposition of Gradient Boosting" propose a multiscale approach to tree pruning that treats a decision tree as a sum of local differences, or wavelets, allowing for surgical precision in model refinement.
2. Methodology: Geometric Wavelet Decomposition
The core of the proposed method is the Geometric Wavelet (GW) decomposition. Rather than viewing a tree as a hierarchy of rules, it is treated as a telescopic sum of functions representing the change in prediction between parent and child nodes.
2.1 The Wavelet Norm and Significance
For each node, a "wavelet norm" is calculated to quantify its contribution to the model's predictive power. High-norm wavelets represent significant shifts in data distribution, while low-norm wavelets often correspond to noise or redundant information. This allows the model to prioritize significance over structural position.
2.2 Adaptive M-Term Approximation
In each boosting iteration, GWGB grows a deep tree (e.g., depth 8) and decomposes it. Instead of pruning entire levels, the algorithm uses Out-Of-Bag (OOB) data to greedily select the $M$ most significant wavelets based on their norms.
This non-hierarchical pruning allows the model to capture high-resolution details early in the boosting process if they are statistically significant, effectively filtering out noisy "spikes" that would otherwise lead to overfitting.
3. Experimental Results
The authors evaluated GWGB across three primary domains: class imbalance, regression, and robustness to label noise.
3.1 Class Imbalance and Regression
On 44 real-world datasets from the KEEL repository characterized by high class imbalance, GWGB consistently outperformed state-of-the-art bagging and boosting methods in terms of AUC. In regression tasks (e.g., Housing, Diabetes), GWGB achieved lower Root Mean Square Error (RMSE) than both standard decision stump boosting and vanilla neural networks. For the Housing dataset, RMSE was reduced to 3.38, compared to 4.02 for the next best method.
3.2 Robustness to Noise
To test robustness, the authors injected 10% and 30% random label noise into datasets like Heart and PID. GWGB maintained higher accuracy than "robust" variants like rAdaBoost. For instance, with 10% noise on the Heart dataset, GWGB achieved 80.7% accuracy versus 78.4% for rAdaBoost. This is attributed to the wavelet norm naturally assigning low significance to isolated, non-representative data points.
4. Conclusion
GWGB represents a paradigm shift in how decision trees are integrated into gradient boosting. By prioritizing significance over structure, the model achieves a more flexible and robust representation of complex data distributions. While the computational cost of growing deeper trees and performing decomposition is higher, the gains in accuracy—particularly for imbalanced and noisy data—make it a compelling alternative to traditional Gradient Boosting Machines.
Summary updated