Abstract
We consider the problem of the extraction of semantic attributes, supervised only with classification labels. For example, when learning to classify images of birds into species, we would like to observe the emergence of features that zoologists use to classify birds. To tackle this problem, we propose training a neural network with discrete features in the last layer, which is followed by two heads: a multi-layered perceptron (MLP) and a decision tree. Since decision trees utilize simple binary decision stumps we expect those discrete features to obtain semantic meaning. We present a theoretical analysis as well as a practical method for learning in the intersection of two hypothesis classes. Our results on multiple benchmarks show an improved ability to extract a set of features that are highly correlated with the set of unseen attributes.
Introduction
While deep neural networks (DNNs) are world-class classifiers, they are notoriously "black boxes." A zoologist classifies a bird based on specific dichotomous keys: the shape of the beak, the color of the wings, or the texture of the feathers. A standard DNN, however, might rely on obscure pixel correlations that have no meaning to a human expert.
The challenge addressed here is Weakly Unsupervised Recovery of Semantic Attributes. The goal is to enable a model to "discover" the same attributes experts use, without the painstaking process of labeling thousands of images with every possible attribute. This research explores whether a network can be forced to learn these meaningful intermediate features using only final species labels.
Methodology
The proposed approach utilizes a hybrid architecture that forces the network to produce representations compatible with simple logic.
- The Architecture: A feature extractor (ResNet-50) produces a representation that is passed through a Quantizer, creating a discrete, low-bit vector.
- The Dual Heads: This discrete vector feeds into two separate classifiers: a standard Multi-Layer Perceptron (MLP) for high-performance classification and a Decision Tree, which is inherently interpretable because it relies on simple binary "stumps."
- Intersection Regularization: By training the network to minimize the distance between the MLP's logic and the Decision Tree's logic, the model is forced to find a representation in the "intersection" of both hypothesis classes.
- Sparsity and Optimization: $L_1$ regularization is applied to the quantized activations to encourage sparsity, reflecting human-defined attributes. To handle the non-differentiable nature of quantization, the Straight-Through Estimator (STE) is employed.
Experimental Results
The model was evaluated on four major computer vision benchmarks: aYahoo, aPascal, AwA2 (Animals with Attributes 2), and CUB-200-2011. These datasets include ground-truth attribute labels used strictly for evaluation.
The primary metric is Feature Fidelity ($d_D(F)$)—how highly correlated the learned hidden features are with actual human-defined attributes.
Key findings include:
- Higher Fidelity: On the AwA2 dataset, the method achieved a feature fidelity score of 84.67, outperforming standard quantized networks (80.32).
- Minimal Accuracy Trade-off: While interpretability often comes at the cost of performance, this method maintained high accuracy. On aYahoo, it achieved 96.86% accuracy while significantly improving attribute recovery.
- Interpretability Pressure: Forcing a powerful DNN to produce features usable by a simple decision tree creates a "pressure" that filters out complex, non-interpretable noise in favor of simple, robust features.
Discussion and Limitations
Despite its success, the method faces challenges. The problem is fundamentally "ill-posed" because, without ground truth during training, the model might recover spurious correlations (e.g., "Sky" as an attribute for "Bird" if all birds are pictured in the sky). Additionally, determining the optimal dimensionality for the discrete feature vector $F$ without prior knowledge of the attribute space remains an open question.
Future work may explore applying "Intersection Regularization" to other pairs of hypothesis classes beyond MLPs and Decision Trees to discover different types of structured knowledge.
Summary updated