Abstract
Nonparametric regression is a powerful tool for modeling complex datasets where the model structure adapts to the data. However, traditional methods struggle with high-dimensional feature spaces (the "curse of dimensionality") and the communication overhead required for distributed training. We introduce Silo (Supervised Local modeling), a framework that leverages Random Forests to discover adaptive, supervised neighborhoods and applies local linear refinement to improve precision. Silo achieves superior predictive accuracy while allowing for communication-free training on distributed clusters.
Introduction
Modern datasets often involve complex, non-linear relationships and high-dimensional feature spaces. While nonparametric regression is theoretically ideal for such tasks, two primary obstacles hinder its practical application at scale:
- The Curse of Dimensionality: Methods like $k$-Nearest Neighbors ($k$-NN) rely on Euclidean distance, which becomes uninformative in high dimensions where many features may be irrelevant.
- Distributed Scalability: Training global models across clusters typically requires frequent synchronization of parameters, leading to significant communication bottlenecks.
Silo addresses these issues by combining the feature-selection capabilities of Random Forests with the bias-correction properties of local linear models.
Methodology: The Silo Framework
Silo operates through a two-stage process designed for both accuracy and distributed efficiency.
1. Supervised Neighborhood Discovery
Instead of using fixed Euclidean metrics, Silo utilizes a Random Forest (RF) to define neighborhoods. In this context, two points are considered "neighbors" if they frequently co-occur in the same leaf nodes across the forest. Because RF splits are chosen to minimize prediction error, the resulting neighborhoods are "supervised"—they automatically expand along irrelevant dimensions and contract along important ones, effectively bypassing the curse of dimensionality.
2. Local Linear Refinement
Standard Random Forests predict values by averaging outcomes within a leaf (a local constant model). Silo improves upon this by using the RF-derived weights to fit a local linear regression. This refinement corrects for the boundary bias often found in tree-based methods and allows for better extrapolation.
Distributed Implementation
In a distributed setting, each worker node independently trains a Random Forest on its local data partition. During the training phase, no communication between workers is required. At inference time, each worker computes local "sufficient statistics" (small covariance matrices) for a query point. These statistics are aggregated by a master node to produce the final prediction, ensuring high scalability.
Experimental Results
The authors evaluated Silo against several benchmarks, including the Friedman1 simulation, a high-dimensional Gaussian Process, and the Million Song Dataset.
Accuracy and Bias Reduction
Silo consistently outperformed standard Random Forests and traditional local methods like LOESS and $k$-NN. A key driver of this performance is the reduction in bias. By fitting a local linear model rather than a local constant, Silo significantly lowers the squared bias without a proportional increase in variance.
Scalability
In large-scale tests using Apache Spark, Silo demonstrated a 2x speedup over standard Spark MLlib Random Forest implementations. Furthermore, Silo's accuracy remained robust even as the number of worker nodes increased, unlike naive averaging techniques which often suffer from "model drift" when data is partitioned.
Discussion and Future Work
Silo represents a significant step forward in making nonparametric regression practical for big data. By treating Random Forests as adaptive kernels, it bridges the gap between flexible local modeling and robust global feature selection.
While Silo offers substantial training-time savings, it introduces higher latency during inference due to the need to solve a linear system for each prediction. Future research may focus on optimizing test-time performance through batching or approximation techniques, as well as extending the framework to classification and sparse feature regimes.
Summary updated