Abstract
We consider the problem of the extraction of semantic attributes, supervised only with classification labels. For example, when learning to classify images of birds into species, we would like to observe the emergence of features that zoologists use to classify birds. To tackle this problem, we propose training a neural network with discrete features in the last layer, which is followed by two heads: a multi-layered perceptron (MLP) and a decision tree. Since decision trees utilize simple binary decision stumps we expect those discrete features to obtain semantic meaning. We present a theoretical analysis as well as a practical method for learning in the intersection of two hypothesis classes. Our results on multiple benchmarks show an improved ability to extract a set of features that are highly correlated with the set of unseen attributes.
Introduction
Deep Neural Networks (DNNs) have achieved state-of-the-art performance in classification tasks but often operate as "black boxes." While a model may correctly identify a bird species, the underlying features it uses are typically abstract mathematical representations rather than the semantic attributes (e.g., "has a red beak" or "is furry") used by human experts. Manually labeling these attributes for training is prohibitively expensive and time-consuming. This work addresses the challenge of "Weakly Supervised Recovery of Semantic Attributes," aiming to extract human-interpretable features using only high-level class labels as supervision.
Methodology
The proposed approach utilizes a hybrid architecture that integrates the representational power of DNNs with the inherent interpretability of decision trees.
As illustrated in the architecture, the model consists of a feature extractor (ResNet-50) followed by a quantizer that transforms continuous activations into a discrete, binary-like vector. This vector serves as the input to two parallel heads: a standard Multi-Layer Perceptron (MLP) and a Decision Tree.
The core innovation, Intersection Regularization, addresses the non-differentiability of decision trees. The neural network is trained to minimize its distance from the decision tree's logic, while the tree is trained to approximate the network's output. This "soft intersection" pressures the DNN to abandon complex, non-linear features in favor of "stump-friendly" features that naturally align with semantic attributes. Additionally, $L_1$ regularization is applied to ensure the learned attribute vector remains sparse, mimicking the sparsity of human-defined attributes.
Key Results
The method was evaluated on four major computer vision benchmarks: aYahoo, aPascal, AwA2 (Animals with Attributes 2), and CUB-200-2011. These datasets include ground-truth attribute labels used exclusively for evaluation, not training.
As shown in the comparative results, the proposed method significantly outperforms existing baselines in Feature Fidelity ($d_D(F)$), which measures the correlation between learned features and real-world attributes. For instance, on the AwA2 dataset, the method achieved a fidelity score of 84.67, surpassing the next best baseline at 83.31.
Crucially, this gain in interpretability does not require a significant sacrifice in classification accuracy. On the aYahoo dataset, the model reached 96.86% accuracy, which is nearly identical to standard non-interpretable models. This suggests that semantic features are often the most efficient way to distinguish between classes when the model is constrained by the simplicity of a decision tree.
Discussion and Conclusion
The success of this method stems from "Interpretability Pressure." By forcing a complex DNN to agree with a decision tree, the model discovers semantic features naturally as they represent the optimal bottleneck for classification. Qualitative analysis on the AwA2 dataset confirmed this, showing the recovery of 48 out of 85 attributes with high correlation.
While effective, the problem remains inherently ill-posed due to potential attribute ambiguity, where highly correlated features (e.g., "red" and "round") may be merged. Future work will focus on refining hyperparameter selection for feature dimensionality and extending the framework to non-visual domains, such as medicine and law, where understanding the "why" behind a decision is as critical as the decision itself.
Summary updated