Interpretable Mixture Density Estimation by use of Differentiable Tree-module

Ryuichi Kanoh, Tomu Yanabe

Abstract

Mixture Density Networks (MDNs) are essential for modeling complex, multi-modal probability distributions in real-world applications. However, their "black-box" nature and computational intensity limit their use in interpretable or real-time systems. This paper introduces a differentiable "Soft-Tree" module that structures the mixture weight calculation as a hierarchical decision process. By decoupling time-variant routing from time-invariant distribution parameters, the proposed architecture enhances both interpretability and inference speed. Experimental results on a large-scale taxi ride-hailing dataset demonstrate that this approach outperforms standard MDNs in probabilistic accuracy while providing a clear visual map of the model's decision-making process.

Introduction

In many real-world applications, such as predicting taxi travel times or financial risks, a single "point prediction" is insufficient. We require an understanding of uncertainty. Probabilistic models like Mixture Density Networks (MDNs) excel here by outputting a "mixture" of several distributions (e.g., multiple Gaussians) to represent complex, multi-modal outcomes.

However, standard MDNs are typically opaque, making it difficult for humans to understand why a specific distribution was chosen for a given input. Furthermore, calculating these complex mixtures in real-time can be computationally expensive. This paper addresses the dual challenge of interpretability—understanding the model's logic—and inference speed—making it viable for real-time services.

The Soft-Tree Module

The core innovation is a novel architecture that replaces the standard hidden layers of an MDN with a Differentiable Tree-Module (or "Soft-Tree"). Unlike traditional decision trees that make "hard" binary choices, this model utilizes "soft" splits.

Hierarchical routing logic within the differentiable Soft-Tree module. Each node calculates a probability $g_i$ to determine the flow of data.

As shown in the figure above, each node uses a trainable filter and a sigmoid activation to determine the probability ($g_i$) of data flowing to specific branches. This formulation ensures the entire tree remains differentiable and trainable via standard backpropagation. The final mixture weights ($\alpha$) are calculated as the product of probabilities along the path to each leaf.

Architecture and Caching

The proposed architecture effectively separates the task of defining potential outcomes from the task of choosing which outcome is likely.

The proposed Soft-Tree MDN architecture (b) separates time-variant routing from time-invariant leaf modules.
  1. Leaf-Level MDNs: Each leaf in the tree represents a candidate distribution.
  2. Time-Invariant Caching: The authors categorize input features into two groups: time-variant (e.g., current weather, time of day) and time-invariant (e.g., road length, location coordinates). By feeding only time-invariant data into the leaf modules, the parameters of the candidate distributions (mean $\mu$ and variance $\sigma$) can be pre-computed and cached.

During real-time inference, the model only needs to calculate the tree's routing logic based on time-variant features, significantly boosting execution speed without sacrificing the ability to handle complex data.

Experimental Evaluation

The model was evaluated using a massive real-world dataset from GO (formerly MOV), a Japanese taxi ride-hailing service. The task was to predict the duration for a taxi to traverse specific road segments, involving nearly 1 million records and 14 features.

The researchers compared the proposed model against a standard MDN and a Soft-Tree with constant (non-input-dependent) leaves using Negative Log-Likelihood (NLL). Lower scores indicate better probabilistic accuracy.

Model NLL (Lower is Better)
(1) Standard MDN $-0.1937 \pm 0.2144$
(2) Soft-Tree (Constant Leaves) $0.9926 \pm 0.1148$
(3) Proposed Method $-0.4512 \pm 0.1925$

The proposed method not only outperformed the standard MDN in accuracy but also required fewer parameters for real-time inference while providing significantly higher interpretability.

Conclusion

By transforming the mixture weight calculation into a readable flowchart, the Soft-Tree MDN moves away from "black-box" predictions toward transparent, visual logic. While future work may explore deeper trees or more complex leaf distributions, this research provides a robust framework for balancing performance with the need for human-understandable AI in AI systems. The ability to cache time-invariant components further ensures that these interpretable models remain fast enough for demanding real-world production environments.

Summary updated

Abstract

In order to develop reliable services using machine learning, it is important to understand the uncertainty of the model outputs. Often the probability distribution that the prediction target follows has a complex shape, and a mixture distribution is assumed as a distribution that uncertainty follows. Since the output of mixture density estimation is complicated, its interpretability becomes important when considering its use in real services. In this paper, we propose a method for mixture density estimation that utilizes an interpretable tree structure. Further, a fast inference procedure based on time-invariant information cache achieves both high speed and interpretability.
BibTeX
@misc{kanoh2021interpretable,
  author = {Ryuichi Kanoh and Tomu Yanabe},
  title = {{Interpretable Mixture Density Estimation by use of Differentiable Tree-module}},
  year = {2021},
  eprint = {2105.03616},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2105.03616},
}
Figure