TL;DR
LLMs are increasingly used for ontology engineering, but do they actually "understand" the underlying taxonomic relationships, or are they just reciting memorized patterns? This paper introduces MIOE, a new benchmark that uses mechanistic interpretability to show that while ontological knowledge is causally accessible within models like Qwen3, it remains messy and deeply entangled with other subject-specific attributes.
What Problem Does This Paper Address?
While Large Language Models (LLMs) show promise in ontology tasks—like identifying that "Amsterdam" is a "City" (entity typing) or that "City" is a "Geographical Feature" (subsumption)—most evaluations only look at the final output. This "behavioral" approach doesn't tell us if the model is using a dedicated internal mechanism for these relationships or just relying on surface-level associations.
Furthermore, existing mechanistic interpretability research has focused heavily on simple factual recall (e.g., "The capital of France is..."). Ontological knowledge is different; it’s schema-level information that defines the very nature and constraints of an entity. The authors set out to determine if LLMs have a "faithful" internal representation of these taxonomic structures that can be isolated and manipulated without breaking other facts the model knows about an entity.
Methods
The researchers frame the problem as one of causal abstraction. They use interchange interventions: they identify a specific internal representation (a "target") and swap it between two different inputs to see if the model's behavior changes in a predictable way.
For example, if they swap the "type" representation of Amsterdam (a city) with that of Australia (a country), a faithful mechanism should cause the model to now say Amsterdam is a country, while still correctly remembering that Amsterdam is in the Netherlands (preserving auxiliary information).
They evaluated several state-of-the-art interpretability methods:
- Sparse Autoencoders (SAEs): Decomposing activations into sparse, interpretable features.
- Distributed Alignment Search (DAS): Finding low-dimensional linear subspaces that correspond to high-level concepts.
- Multi-task variants (MDAS/MDBM): Training the intervention to not only change the target answer but explicitly preserve unrelated auxiliary facts.
- Multi-layer interventions: Extending these searches across contiguous windows of transformer layers rather than just a single site.
Datasets
The authors introduced the MIOE (Mechanistic Interpretability for Ontology Engineering) benchmark. It is built from Wikidata, WordNet, and schema.org and includes:
- Two core tasks: Entity typing (instance of) and hierarchy prediction (subclass of).
- 54 classes for entity typing and 39 classes for subsumption.
- Auxiliary attributes: For every subject, the dataset includes non-target facts (e.g., for a city: population, official language) to test if interventions are truly disentangled.
- Prompt variants: Both controlled templates and naturalistic Wikipedia-derived contexts.
Key Insights and Ideas
- Ontology is "Middle-Management": The internal mechanisms for ontological relations aren't found at the very beginning or end of the model. As shown in Figure 1, the "Cause" effect (the ability to change the model's answer) peaks in the middle-to-late layers.
- The Disentanglement Struggle: It is difficult to change an entity's type without "leaking" into other attributes. There is a clear tension between changing the target answer and preserving auxiliary facts.
- Class-Level Personality: Not all ontology classes are created equal. Concrete categories with regular attribute profiles (like airport or currency) are much easier to isolate than broad, heterogeneous ones (like body of water or biological process).
- Role Asymmetry: A class might be a great "source" for an intervention (easy to graft onto others) but a poor "base" (hard to overwrite), or vice versa. For instance, protein and video game are strong sources, while national park is a stronger base.
Key Results
- MDAS Wins: The Multi-task Distributed Alignment Search (MDAS) achieved the best balance of changing the target while preserving auxiliary info, reaching a disentanglement score of 0.592 for instance of on Qwen3-8B.
- Multi-layer is Better: Intervening on a window of 4 layers outperformed single-layer interventions, suggesting that ontological computation is distributed across a short interval of the residual stream rather than a single point.
- Supervision Matters: Methods that didn't use explicit preservation supervision (like standard SAEs or DAS) often "broke" the model's auxiliary knowledge while trying to change the taxonomic type, as illustrated by the trade-offs in Figure 2.
Limitations and Future Work
The authors conclude that while ontological variables are "causally accessible," they are not yet "cleanly disentangled." Current methods can't perfectly isolate the "city-ness" of an entity from its other properties.
Future directions include: * Testing richer ontological constraints beyond simple typing and subsumption. * Evaluating a broader range of model families. * Investigating whether these internal mechanisms remain stable across even more complex reasoning contexts or different domains.
Summary updated