Neurons to Words: A Novel Method for Automated Neural Network Interpretability and Alignment

Lukas-Santo Puglisi, Fabio Valdés, Jakob Johannes MetzgerAAAI 2025 · DOI

Introduction

As artificial intelligence models scale to billions of parameters, they increasingly function as "black boxes," making their internal decision-making processes opaque. Researchers have developed a scalable method to automatically interpret neural networks by using Large Language Models (LLMs) to translate internal activations into human-readable descriptions. By encoding raw activation values into a compact character-based format, they trained an "interpreter" model capable of analyzing a network's internal state and explaining the specific features—such as shapes or patterns—the network perceives.

The Interpretability Challenge

Traditional interpretability methods, such as saliency maps or surrogate models, often require significant manual effort or provide only localized explanations for individual predictions. A critical gap exists: the lack of a generic, automated way to understand a network's high-level reasoning. This is particularly vital for superalignment—ensuring superhuman AI systems provide correct answers for the right reasons. Without the ability to "read the mind" of a model via its activations, trust in high-stakes fields like medicine or autonomous systems remains elusive.

Methodology: The Neural Interpreter Pipeline

The proposed "Neural Interpreter" follows a three-phase pipeline to bridge the gap between neural activations and natural language.

  1. Target Network & Data Collection: A standard Convolutional Neural Network (CNN) was trained on MNIST handwritten digits. Low-level features (diagonals, corners, checkerboards) were identified, and a synthetic dataset of these shapes paired with text descriptions was created.
  2. The Encoding Scheme: Since LLM tokenizers are optimized for text rather than 32-bit floating-point numbers, a char-encoding binning scheme was developed. Activation values were mapped to five characters representing intensity: z (zero), l (low), m (medium), h (high), and v (very high). This compression allowed LLMs to process neural activations as a structured sequence.
  3. Fine-Tuning the Interpreter: Open-source LLMs (Mistral-7B and Mixtral-8x7B) were fine-tuned to perform an autoregressive task: given a task prompt and a sequence of encoded activations, predict the human-readable description of the represented features.
The Neural Interpreter pipeline: from raw activations to character-encoded sequences and final text descriptions.

Datasets

The study utilized two primary datasets:

  • Synthetic Shapes ($D_{synth}$): 15,000 images containing random combinations of seven primitive shapes (e.g., vertical, horizontal, ellipse).
  • MNIST ($D_{mnist}$): The classic handwritten digit dataset, with a subset of 100 images manually annotated with structural descriptions to test generalization.

Key Results

The interpreter models demonstrated high accuracy and generalization capabilities. On synthetic data, the Mistral-7B interpreter achieved 80% accuracy in shape identification with a ROUGE-1 score of 0.9269. When tested on real MNIST digits, the models successfully identified complex structural elements, such as describing a handwritten "5" through its constituent horizontals and diagonals.

Training and validation loss curves for Mistral-7B and Mixtral-8x7B during the interpreter fine-tuning process.

While Mistral-7B showed slightly better precision on synthetic data, the larger Mixtral-8x7B exhibited superior behavioral traits, such as better adherence to output formats and more reliable termination of text generation.

Key Insights and Future Work

The core innovation lies in treating neural activations as a translatable language, leveraging the pre-existing geometric and logical knowledge within LLMs. This automated approach offers significant scalability compared to manual mechanistic interpretability.

However, limitations remain. The current method flattens 2D activation maps into 1D sequences, losing spatial context. Additionally, the 5-level binning scheme is a lossy compression that may overlook nuanced features. Future work aims to scale this to higher resolutions and more complex data types, such as sentiment analysis in text-processing models.

Summary updated

BibTeX
@inproceedings{puglisi2025neurons,
  author = {Lukas-Santo Puglisi and Fabio Valdés and Jakob Johannes Metzger},
  title = {{Neurons to Words: A Novel Method for Automated Neural Network Interpretability and Alignment}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i26.34972},
}
Figure