Discerning Decision-Making Process of Deep Neural Networks with Hierarchical Voting Transformation

Ying Sun, Hengshu Zhu, Chuan Qin, Fuzhen Zhuang, Qing He, Hui Xiong

Deep neural networks are the powerhouses of modern AI, but they are notoriously tight-lipped about their internal logic. We often treat them as "black boxes" where data goes in and a prediction comes out, with the intermediate reasoning buried under millions of coupled parameters. At NeurIPS 2021, researchers from Baidu and the Chinese Academy of Sciences presented VOTEN (Voting Transformation-based Explainable Neural Network), a model designed to be transparent by design without sacrificing the "deep" performance we've come to expect.

The Explainability Trade-Off

The central challenge in Explainable AI (XAI) is the trade-off between performance and interpretability. Classic models like decision trees are easy to understand but struggle with complex data. Deep learning models (like Multi-Layer Perceptrons or MLPs) excel at complex data because they create massive, intertwined combinations of features. However, this "coupling" makes it nearly impossible to isolate why a specific feature led to a specific output.

Existing "post-hoc" explanation methods (like SHAP or LRP) try to guess what a trained model is doing from the outside. The authors of VOTEN argue that we should instead build models that are intrinsically explainable—meaning the decision-making process is built into the architecture itself in a way that mirrors human reasoning.

Methods: The VOTEN Architecture

The core idea of VOTEN is to treat the transformation from input to output as a hierarchical voting process.

VOTEN Architecture illustrating hierarchical voting channels, counting layers, and model pruning
  1. Explicit Concepts: Instead of abstract hidden units, VOTEN induces "concepts" layer-by-layer.
  2. Voting Channels: Each concept in a lower layer acts as a "voter." It uses a single-valued nonlinear function (a "voting network") to cast a vote for a concept in the next layer.
  3. Counting Layers: These layers aggregate the votes using weighted averages to determine the value of the higher-level concept.
  4. Mathematical Grounding: The authors theoretically prove that a standard MLP is actually a "degenerated" version of VOTEN. While an MLP uses simple linear weights and a shared activation function, VOTEN allows each "voter" to have its own expressive, nonlinear voting function. This extra expressiveness in the channels allows the model to use far fewer concepts to achieve the same (or better) results.

The researchers also developed specific algorithms for Decision Path Recognition (tracing the most influential votes for a specific prediction) and Model Pruning (removing unimportant voting channels to simplify the model further).

Key Insights and Ideas

  • Decoupling is Key: By forcing each vote to be a single-valued function of one concept, VOTEN prevents the "feature coupling" that makes MLPs so hard to read. You can look at a single voting channel and see exactly how a feature like "Price" influences a higher-level "Subway" score.
  • Expressiveness vs. Complexity: Because VOTEN's voting functions are powerful, the model needs fewer layers and hidden units. In a multi-modal transportation dataset, an MLP required 7 layers and 128 concepts per layer, leading to $2^{49}$ possible decision paths. VOTEN achieved better accuracy with just 2 layers and 16 concepts, resulting in only 256 possible paths.
  • Meaningful Latent Concepts: Through case studies, the authors showed that the hidden concepts VOTEN learns actually make sense.
Visualization of learned voting functions and the resulting decision path tree

For example, in transportation recommendation, the model automatically induced concepts representing "distance-cost performance" and "availability of flexible subway plans" based purely on the shapes of the learned voting functions.

Datasets and Key Results

The authors evaluated VOTEN against five strong baselines (including LightGBM and MLP) across six diverse real-world datasets, ranging from search conversion rate prediction to Higgs boson classification.

  • Superior Performance: VOTEN consistently outperformed or matched the best-performing baselines (like LightGBM) across all datasets in terms of AUC and Average Precision (AP).
  • Drastic Simplification: As noted above, VOTEN reduces the number of possible decision paths by many orders of magnitude compared to standard MLPs, making the model's logic highly traceable.
  • Effective Pruning: The model supports "link pruning." On the transportation dataset, the researchers could delete nearly 50% of the voting channels while maintaining an AUC of 0.924, proving that the model's logic is robust and sparse.

Limitations and Future Work

While VOTEN extracts concepts automatically, human effort is still required to look at the voting functions and "label" what those concepts mean (e.g., identifying that a specific node represents "distance-cost performance").

The authors suggest future work could focus on:

  1. Automatic Alignment: Developing ways to automatically align learned concepts with human-understandable labels.
  2. Task-Specific Architectures: Extending the VOTEN "voting" logic to more specialized structures like Convolutional Neural Networks (CNNs) for images or Transformers for text.
  3. User Studies: Testing how much these explanations actually help domain experts (like business analysts or doctors) make better, more informed decisions.

Summary updated

Abstract

Neural network based deep learning techniques have shown great success for numerous applications. While it is expected to understand their intrinsic decision-making processes, these deep neural networks often work in a black-box way. To this end, in this paper, we aim to discern the decision-making processes of neural networks through a hierarchical voting strategy by developing an explainable deep learning model, namely Voting Transformation-based Explainable Neural Network (VOTEN). Specifically, instead of relying on massive feature combinations, VOTEN creatively models expressive single-valued voting functions between explicitly modeled latent concepts to achieve high fitting ability. Along this line, we first theoretically analyze the major components of VOTEN and prove the relationship and advantages of VOTEN compared with Multi-Layer Perceptron (MLP), the basic structure of deep neural networks. Moreover, we design efficient algorithms to improve the model usability by explicitly showing the decision processes of VOTEN. Finally, extensive experiments on multiple real-world datasets clearly validate the performances and explainability of VOTEN.
BibTeX
@inproceedings{sun2021discerning,
  author = {Ying Sun and Hengshu Zhu and Chuan Qin and Fuzhen Zhuang and Qing He and Hui Xiong},
  title = {{Discerning Decision-Making Process of Deep Neural Networks with Hierarchical Voting Transformation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
}
Figure