Abstract
Interpreting the learned features of vision models has posed a longstanding challenge in the field of machine learning. To address this issue, we propose a novel method that leverages the capabilities of language models to interpret the learned features of pre-trained image classifiers. Our method, called TExplain, tackles this task by training a neural network to establish a connection between the feature space of image classifiers and language models. During inference, our approach generates a vast number of sentences to explain the features learned by the classifier for a given image. These sentences are then used to extract the most frequent words, providing a comprehensive understanding of the learned features and patterns within the classifier. Our method, for the first time, utilizes these frequent words corresponding to a visual representation to provide insights into the decision-making process of the independently trained classifier, enabling the detection of spurious correlations, biases, and a deeper comprehension of its behavior. To validate the effectiveness of our approach, we conduct experiments on diverse datasets, including ImageNet-9L and Waterbirds. The results demonstrate the potential of our method to enhance the interpretability and robustness of image classifiers.
Introduction
Deep learning vision models, while highly accurate, are often criticized for being "black boxes." Understanding the internal logic of a model—why it classifies an image in a certain way—is crucial for trust and safety in real-world applications. Traditional interpretability tools like Grad-CAM provide spatial heatmaps indicating where a model is looking, but they fail to explain the semantic concepts the model associates with those regions. This lack of transparency often hides "spurious correlations," where a model learns to identify objects based on irrelevant background features, such as identifying a fish solely by the presence of water. TExplain addresses this by translating visual latent spaces into human-readable language.
The TExplain Framework
TExplain introduces a novel pipeline to translate abstract visual embeddings into human-readable text. The architecture consists of three primary components: a frozen image classifier (typically a Vision Transformer), a frozen language model (such as BERT), and a trainable Multi-Layer Perceptron (MLP) translator.
The translator is trained to map visual feature vectors into the embedding space of the language model using a large-scale dataset of 14 million image-text pairs. During inference, TExplain takes a feature vector and generates 1,000 candidate sentences via Nucleus Sampling. Instead of relying on a single sentence, the system extracts the most frequent descriptive words across these samples to produce a word cloud. This frequency-based approach ensures that the resulting explanation represents the most robust semantic features the model has actually learned, bypassing the common issue of language model hallucinations.
Identifying Spurious Correlations and Model Bias
A significant advantage of TExplain is its ability to reveal the "shortcuts" models take during training. By analyzing the generated word clouds, researchers can determine if a model is fixated on the correct object or merely its context.
As illustrated in the example above, for the category "train," the model's internal representations were more closely associated with "tracks" and "smoke" than the physical locomotive. This insight allows for targeted debugging. In experiments on the Waterbirds dataset, the authors used TExplain to identify problematic samples where the model relied on spurious background features. By masking these regions and fine-tuning, they improved worst-group accuracy from 89.79% to 95.63% while only masking 26% of the training data, significantly outperforming random masking strategies.
Conclusion
TExplain provides a powerful new lens for interpreting vision models by treating visual features as a language to be translated. While currently focused on general object classification, the framework is theoretically applicable to any encoder-decoder pair, including generative or multi-modal systems. By giving a voice to the internal representations of vision models, TExplain moves us closer to building more transparent, robust, and trustworthy AI systems.
Summary updated