Trees to Flows and Back: Unifying Decision Trees and Diffusion Models

Sai Niranjan Ramachandran, Suvrit Sra

TL;DR

This paper bridges the gap between two seemingly disparate machine learning models: decision trees (discrete, hierarchical) and diffusion models (continuous, dynamic). By establishing a formal mathematical link, they unify these models under a single optimization principle, "Global Trajectory Score Matching" (GTSM). This unification leads to novel algorithms that achieve state-of-the-art performance on tabular data, offering both improved generation quality and faster training.

What Problem Does This Paper Address?

The paper tackles the long-standing disconnect between two highly successful but domain-specific classes of machine learning models: decision trees (and their ensembles like gradient boosting) excelling in structured, tabular data, and diffusion models dominating continuous data generation (images, audio). While decision trees partition data hierarchically, diffusion models learn to reverse a continuous process of noise addition. This fundamental difference in formulation has led to separate research communities and a lack of cross-pollination. The authors aim to bridge this divide by showing they are not fundamentally different but rather different perspectives on the same underlying generative and discriminative principles.

Methods

The core of the paper is the Tree-Flow Correspondence. They demonstrate that:

  1. Trees to Flows: A hierarchical decision tree's partitioning process can be viewed as a discrete-time Markov process. By applying a "dyadic refinement" procedure (inserting intermediate splits), this discrete process can be made continuous, leading to a deterministic Probability Flow Ordinary Differential Equation (PF-ODE).
  2. Flows to Trees: Conversely, a continuous-time diffusion process (specifically, an entropically homogeneous SDE converging to a stationary distribution) implicitly defines a hierarchical clustering. This hierarchy is revealed by tracking how modes of the data distribution merge over time, forming a structure isomorphic to a decision tree.

This correspondence unifies the two models under a single optimization principle: Global Trajectory Score Matching (GTSM). They show that:

  • Gradient Boosting acts as an asymptotically optimal discrete solver for GTSM.
  • Diffusion Model Training acts as a continuous solver for GTSM.

Based on this, they introduce two novel algorithms:

  • TREEFLOW: Leverages decision tree partitions as a conditioning mechanism for Conditional Flow Matching (CFM) to generate high-fidelity tabular data more efficiently.
  • DSM-TREE: Distills the entire hierarchical decision logic of a tree into a neural network, going beyond just leaf predictions, for discriminative tasks.

Datasets

The paper uses a variety of datasets for evaluation:

  • For Verifying Equivalence (Experiments 1 & 2):
    • Synthetic 2D datasets (4_Corners, 9_Grid, 8_Gaussians) for visualizing cluster structures and hierarchy discovery.
  • MNIST, Fashion-MNIST, and USPS for information decay analysis.
  • For Algorithmic Instantiations (Experiments 3 & 4):
    • DSM-TREE: UCI tabular datasets: Digits (8x8), German Credit, Boston Housing (classification), Heart Disease, and Abalone (classification).
  • TREEFLOW: Standard tabular datasets: Adult, Breast Cancer, Diabetes, Wine, and a synthetic California Housing dataset.

No new benchmarks were introduced; existing ones were used.

Key Insights and Ideas

  • Unified Framework: The most significant insight is that decision trees and diffusion models are not fundamentally different but rather two sides of the same coin, unified by the GTSM objective. This provides a powerful new lens for understanding both.
  • GTSM as a Master Objective: The Global Trajectory Score Matching (GTSM) objective acts as a unifying principle, showing how seemingly different training methods (like gradient boosting and diffusion model training) are actually solving the same underlying problem of matching a complex data distribution's trajectory.
  • Trees as Inductive Bias for Flows: Decision trees provide a natural hierarchical structure that can be leveraged as an inductive bias for continuous generative models like diffusion models, leading to improved performance and efficiency, especially on tabular data.
  • Distilling Hierarchical Logic: DSM-TREE demonstrates that it's possible to distill not just the final predictions of a decision tree but its entire hierarchical decision-making process into a neural network, preserving interpretability and structure.
  • Boosting as Optimal GTSM Solver: The paper rigorously proves that the greedy, stage-wise nature of gradient boosting is not a heuristic but a provably optimal strategy for solving the discrete GTSM objective.

Key Results

  • TREEFLOW: Achieves competitive generation quality on tabular data, obtaining the highest TSTR accuracy on 3/5 benchmarks, lowest Wasserstein distance on 4/5, and lowest correlation error on 3/5. Crucially, it is 2x faster than the strong TabDDPM baseline.
  • DSM-TREE: Successfully distills hierarchical decision logic into neural networks, matching teacher performance within 2% on most benchmarks and even exceeding it by 3.7% on the Heart Disease dataset.
  • Theoretical Equivalence: The paper formally establishes the bidirectional correspondence between hierarchical decision trees and PF-ODEs/SDEs.

Limitations and Future Work

  • Continuous Feature Spaces: The theoretical framework and algorithmic instantiations primarily focus on continuous feature spaces. Extending the theory to handle intrinsic discontinuities (e.g., via Lévy processes) is a key area for future work.
  • Generalizing to Heterogeneous Domains: The authors suggest combining data-driven adaptive partitioning of trees with the expressive power of diffusion models to build novel foundation models for complex, heterogeneous domains (e.g., tabular and sequential data) that currently struggle with multiple modalities.
  • Practicality of Tree-Flow: While TREEFLOW shows promise, the complexity of learning tree structures for very deep trees might still be a challenge.
  • Real-world Applicability: While the paper demonstrates strong results on tabular data, further exploration of its applicability to other domains where decision trees and diffusion models are currently used separately is warranted.

Summary updated

Abstract

Decision trees and diffusion models are ostensibly disparate model classes, one discrete and hierarchical, the other continuous and dynamic. This work unifies the two by establishing a crisp mathematical correspondence between hierarchical decision trees and diffusion processes in appropriate limiting regimes. Our unification reveals a shared optimization principle: \emph{Global Trajectory Score Matching (GTSM)}, for which gradient boosting (in an idealized version) is asymptotically optimal. We underscore the conceptual value of our work through two key practical instantiations: treeflow, which achieves competitive generation quality on tabular data with higher fidelity and a 2× computational speedup, and dsmtree, a novel distillation method that transfers hierarchical decision logic into neural networks, matching teacher performance within 2\% on many benchmarks.
BibTeX
@inproceedings{ramachandran2026trees,
  author = {Sai Niranjan Ramachandran and Suvrit Sra},
  title = {{Trees to Flows and Back: Unifying Decision Trees and Diffusion Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=gW7NZN8zJu},
}

Loading related content...

Figure