TaxoSAE: Rethinking Sparse Autoencoders through Taxonomic Logic

Xinyu Guo, Minglai Yang, Clayton T. Morrison, Eduardo Blanco, Mihai SurdeanuarXiv 2026 · arXiv

TL;DR

TaxoSAE moves beyond the "flat" feature spaces of standard Sparse Autoencoders (SAEs) by explicitly encoding hierarchical "is-a" relationships into LLM representations. By stacking SAEs to mirror real-world taxonomies and using a novel taxonomic alignment loss, it achieves significantly more consistent knowledge readout and precise, granular control over internal model states.

What Problem Does This Paper Address?

Mechanistic interpretability aims to crack open the "black box" of Large Language Models (LLMs). Sparse Autoencoders (SAEs) have emerged as a leading tool for this, decomposing complex activations into sparse, interpretable features. However, current SAEs treat every feature as a standalone, independent unit.

In reality, human knowledge is taxonomically organized. A "T cell" is a type of "lymphocyte," which is a type of "immune cell." Standard SAEs ignore these logical dependencies, leading to two major issues:

  1. Taxonomic Inconsistency: A readout might show the "T cell" feature is active while the "lymphocyte" feature remains silent—a logical impossibility in a proper taxonomy.
  2. Lack of Control Granularity: Because features are "flat," editing a specific concept (like replacing a T cell with a B cell) doesn't naturally respect the broader category (they are both lymphocytes). You cannot easily specify the scope or scale of an edit.

Methods

The authors propose TaxoSAE, which bridges the gap between neural activations and symbolic logic through two main components:

  • Stacked SAEs: Instead of one giant SAE, TaxoSAE uses a stack of SAEs, where each level corresponds to a level in a target taxonomy (e.g., Species $\to$ Class $\to$ Phylum). The first SAE reads the LLM's hidden state and maps it to specific terminal concepts. Each subsequent SAE reads the reconstruction of the level below and maps it to higher-level hypernyms.
  • Taxonomic Alignment Loss: This is the core innovation. It enforces the "is-a" logic: a hypernym slot is trained to fire whenever any of its hyponyms (children) are active. This turns a logical rule into a differentiable training objective.
Figure 2: Overview of T axoSAE. We train stacked SAEs on a frozen LLM’s last-token hidden state. (1) Stacked SAEs: one SAE per taxonomy level (L1,...,K in Fig. 1), each mapping the reconstruction of the level below onto its hypernyms. (2) Taxonomic alignment loss: each taxonomic slot is trained to fire when the input involves its node or any of the node’s hyponyms. Complement slots bind the complementary element; reconstruction, sparsity, and orthogonality losses complete the objective. Figure 8 shows the full slot layout of each level on both taxonomies.

To handle the complexity of factual questions, each SAE level uses three types of "slots":

  1. Taxonomic slots: Dedicated to specific nodes in the taxonomy.
  2. Complement slots: Dedicated to the "other" part of a fact (e.g., if the taxonomy is animals, the complement might be "lifespan" or "diet").
  3. Free slots: Standard unsupervised slots to capture the remaining variance in the representation.

Datasets

The researchers evaluated TaxoSAE on two custom-built factual QA taxonomies:

  • Bio: A relation taxonomy based on Freebase biographical attributes (18 terminal relations like "birth city," grouped into 6 intermediate and 3 top-level categories).
  • Animal: An entity taxonomy based on ConceptNet (18 terminal animals like "cow" or "frog," grouped into 6 intermediate classes and 2 top-level classes).

These datasets were used to fine-tune base models (GPT-2 small and Qwen2.5-0.5B) to ensure they actually "knew" the facts before the SAEs were trained to interpret them.

Key Insights and Ideas

  • Logic as a Geometric Constraint: By enforcing taxonomic alignment, the authors essentially force the SAE to learn a feature geometry that respects logical hierarchies.
  • Generalization to the Unseen: One of the most striking findings is that TaxoSAE can identify the category of concepts it was never trained on. If the model sees a question about a "beetle" (unseen in SAE training), the higher-level SAEs can still correctly activate the "insect" slot because the underlying LLM representation for "beetle" is geometrically similar to other insects.
  • Concentrated Activations: Unlike unsupervised SAEs, which often fragment a single concept across dozens of features, TaxoSAE concentrates information into single, dedicated slots. This makes the features much more "readable" and "editable."
Figure 3: t-SNE of L2 activations on Animal, colored by L2 node: squares are trained animals, triangles unseen ones. A triangle inside its color’s contour is placed under the right hypernym. T axoSAE does this for most unseen animals; DiffMean’s regions overlap. Mollusc is omitted due to heavy overlap in the baseline ( App. C).

Key Results

  • Superior Readout: TaxoSAE outperformed baselines by +31.3 percentage points in reading out hypernyms for abstract concepts that were never seen during SAE training.
  • Precise Control: In "swap" experiments (e.g., forcing the model to think a "cow" is an "eagle"), TaxoSAE achieved a +25.5 point improvement in success rate.
  • Selective Suppression: Deactivating a specific taxonomic slot (like "mammal") successfully suppressed answers for all mammals (0-2% accuracy) while leaving other categories (like "birds") completely unaffected (91-100% accuracy).
  • Monosemanticity: As shown in Figure 5, TaxoSAE activations are highly concentrated, reaching near-ideal "Top1C" scores (the share of activation held by the strongest unit), whereas unsupervised SAEs remain fragmented.
Figure 5: Activation fragmentation (EffFeat) and con- centration (Top1C) per taxonomy level on Animal and Bio; dashed lines mark the ideal single unit. T axoSAE activations are the most concentrated.

Limitations and Future Work

  • Scale: The taxonomies used were relatively small (18 terminal nodes). Real-world taxonomies (like the Tree of Life or WordNet) are orders of magnitude larger.
  • Language: The study was limited to English-language models and datasets.
  • Complexity of Relations: The current work focuses strictly on "is-a" (hierarchical) relations. Future work could extend this logic to other types of relationships, such as "part-of" or causal dependencies.
  • Model Size: While tested on GPT-2 and Qwen2.5-0.5B, the approach needs validation on frontier-scale models (e.g., Llama-3 70B+).

Summary updated

Abstract

Recent Sparse Autoencoders (SAEs) decompose the representations of large language models (LLMs) into interpretable features, enabling precise readout and control of encoded knowledge. However, existing SAEs treat each feature as a standalone unit, whereas real-world knowledge is often taxonomically organized. As a result, readout carries no guarantee of taxonomic consistency, and control lacks the granularity to specify the scope of an edit. We propose TaxoSAE, which explicitly encodes taxonomical information, built on is-a (hypernym-hyponym) relations, into the SAE feature space. TaxoSAE mirrors a taxonomy with a stack of SAEs, one per taxonomy level, and introduces a taxonomic alignment loss that trains each concept to be active whenever one of its hyponyms is. On two factual QA taxonomies, one over biographical relations and one over animals, TaxoSAE enables (1) more accurate readout through the taxonomy with concentrated activations, e.g., +31.3 points in reading out the hypernyms of abstract concepts never queried in training; (2) more precise control, with +25.5 points in swap success, while suppressing a concept removes only its own answers (0-2% accuracy vs. 91-100% on other concepts); and, more importantly, (3) readout of unseen concepts through their hypernyms, with up to +37.9 points in hypernym readout accuracy.
BibTeX
@misc{guo2026taxosae,
  author = {Xinyu Guo and Minglai Yang and Clayton T. Morrison and Eduardo Blanco and Mihai Surdeanu},
  title = {{TaxoSAE: Rethinking Sparse Autoencoders through Taxonomic Logic}},
  year = {2026},
  eprint = {2610.31894},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.31894},
}
Figure