Traditional decision tree algorithms, such as CART or C4.5, partition feature space using axis-aligned splits (also known as axis-parallel or univariate splits). An axis-aligned split evaluates a single feature against a threshold (e.g., $x_1 > 10$). Geometrically, this means every decision boundary must be perpendicular to one of the coordinate axes, carving the input space into rigid, rectangular hyper-regions.
While computationally efficient to discover during greedy training, this geometric constraint introduces severe representational inefficiencies when dealing with complex, real-world data distributions.
The Staircase Effect and Structural Bloat
The most prominent consequence of the axis-aligned constraint is the staircase effect. When the true boundary separating two classes is diagonal, curved, or depends on a linear combination of multiple features, an axis-aligned tree cannot capture this relationship with a single split. Instead, it must approximate the smooth or oblique boundary using a jagged series of orthogonal steps.
As Geoffrey E. Hinton et al. observe in Using Pairs of Data-Points to Define Splits for Decision Trees, trying to represent diagonal boundaries between correlated features using axis-aligned cuts results in complex staircase approximations that require an excessive number of nodes. This forces the tree to grow significantly deeper and larger than necessary.
This structural bloat has two major downsides:
- Loss of Interpretability: Decision trees are often chosen for their transparency. However, as noted in An Incremental Method for Finding Multivariate Splits for Decision Trees by Carla E. Brodley et al., when a concept is inherently defined by relationships between multiple variables, restricting nodes to univariate tests creates massive, convoluted trees that obscure the underlying logic from human operators.
- Poor Generalization: Brittle, step-like boundaries rarely match physical reality. In geographic modeling, for instance, Jochen De Weerdt demonstrates in An Evolutionary Geospatial Regression Tree that axis-parallel splits force models to approximate naturally curved spatial boundaries using a jagged staircase of rectangular partitions, leading to poor representations of geographic reality.
Failure on Correlated Features and Low-Dimensional Manifolds
The geometric limitations of axis-aligned partitioning become even more acute when features are highly correlated or when the data resides on a low-dimensional manifold that is tilted relative to the coordinate axes.
In Which Spatial Partition Trees are Adaptive to Intrinsic Dimension?, Nakul Verma et al. demonstrate that axis-aligned splitting rules are fundamentally unable to efficiently isolate data structures lying on tilted manifolds. Because the tree cannot align its partitions with the intrinsic directionality of the data, it must use many more levels of depth to isolate points.
Similarly, Xuhui Fan et al. point out in Binary Space Partitioning Forest that standard axis-aligned methods cannot naturally separate correlated data. This lack of representational efficiency necessitates a high volume of splits to approximate simple linear relationships, resulting in deeper, more complex trees that struggle to generalize.
Moving Beyond the Axis: Oblique Splits
To overcome these constraints, researchers design oblique decision trees (or multivariate trees). Instead of testing a single variable, an oblique split evaluates a linear combination of features ($\sum w_i x_i > c$), allowing the splitting hyperplane to sit at any arbitrary angle.
The theoretical benefits of this shift are clear. Kristin P. Bennett et al. show in Enlarging the Margins in Perceptron Decision Trees that allowing non-axis-parallel splits enables models to capture complex feature interactions with significantly shallower architectures, reducing overfitting by maximizing the margin between classes at each node.
The primary reason axis-aligned trees remain the default despite these limitations is optimization complexity. Finding the best axis-aligned split requires a simple, discrete scan over individual feature thresholds. In contrast, finding the optimal oblique split requires searching an infinite space of continuous weight combinations, making it computationally expensive.
To bridge this gap, single-author works focus on making oblique trees scalable. For example, Arman Zharmagambetov introduces global optimization strategies in Smaller, more accurate regression forests using tree alternating optimization to produce compact oblique forests, while Jun Xu proposes a mathematically grounded Newton method in Hinge Regression Tree: A Newton Method for Oblique Regression Tree Splitting to efficiently optimize non-axis-aligned splits without relying on expensive brute-force searches.