Mechanistic Interpretability of Antibody Language Models Using SAEs

Reina Haque, Oliver M. Turnbull, Anisha Parsan, Nithin Parsan, John Yang, Charlotte M. Deane

Abstract

Sparse autoencoders (SAEs) are a mechanistic interpretability technique that have been used to provide insight into learned concepts within large protein language models. Here, we employ TopK and Ordered SAEs to investigate an autoregressive antibody language model, p-IgGen, and steer its generation. We show that TopK SAEs can reveal biologically meaningful latent features, but high feature concept correlation does not guarantee causal control over generation. In contrast, Ordered SAEs impose an hierarchical structure that reliably identifies steerable features, but at the expense of more complex and less interpretable activation patterns. These findings advance the mechanistic interpretability of domain-specific protein language models and suggest that, while TopK SAEs are sufficient for mapping latent features to concepts, Ordered SAEs are preferable when precise generative steering is required.

Introduction

Protein language models (pLMs) have become essential tools in antibody drug discovery, assisting in tasks such as library generation and humanization. However, these models often function as "black boxes," making it difficult to verify if they are utilizing biologically relevant features or merely identifying statistical noise. Mechanistic interpretability seeks to address this by deconstructing model activations into human-understandable components. This study applies Sparse Autoencoders (SAEs) to p-IgGen, a 17M-parameter antibody language model, to extract biological features and evaluate their capacity for generative steering.

Methods

The researchers analyzed the hidden layer activations of p-IgGen using two distinct SAE architectures:

  1. TopK SAEs: A standard approach that enforces sparsity by activating only the top $k$ (e.g., 32) latents, favoring the discovery of localized, monosemantic features.
  2. Ordered SAEs (O-SAEs): A nested architecture that imposes a strict hierarchy. Early latents are forced to capture broad, global sequence structures, while later latents capture fine-grained details.

SAEs were trained on 1.8 million antibody sequences from the Observed Antibody Space (OAS). Linear probes were then used to map these latents to known biological concepts, such as Complementarity-Determining Region (CDR) identities and V/J germline gene classifications.

Key Insights: Disentangling Biological Concepts

A primary advantage of SAEs is their ability to disentangle "polysemantic" neurons—which respond to multiple unrelated concepts—into "monosemantic" latents. In p-IgGen, raw hidden neurons often exhibit noisy activation patterns across the entire sequence. In contrast, TopK SAEs isolate features that correspond precisely to structural regions like the CDRH3 or the Heavy J region.

Comparison of raw neuron activations vs. disentangled SAE latents. (a) Latent 4280 targets CDRH3; (b) Neuron 45 is polysemantic; (c) Latents 3054, 532, and 3098 target the Heavy J region.

However, the study identified a critical "Feature Splitting" phenomenon in TopK SAEs. High-level biological concepts (like a specific gene identity) are often decomposed into extremely localized features, such as a specific amino acid at a single position. While these features are highly predictive, they often lack the causal weight necessary to steer the model's global generation process.

Results: Predictive Power vs. Generative Control

The SAEs demonstrated exceptional accuracy in mapping latent features to biology. TopK latents achieved a 0.99 accuracy in predicting CDR identity and a 0.93 macro F1-score for heavy J gene (IGHJ) identity. Despite this predictive success, generative steering—intervening in the model's generation by adding latent vectors to hidden states—yielded divergent results:

  • TopK Steering Failure: Amplifying TopK latents correlated with the IGHJ4 gene failed to increase the proportion of IGHJ4 in generated sequences. This is likely because TopK features are too localized; steering a single residue (e.g., at position 119) is insufficient to shift the model's entire generative trajectory.
Example of a highly localized TopK latent (4720) activating almost exclusively at IMGT position 119.
  • Ordered SAE Steering Success: O-SAEs, by capturing hierarchical and more abstract representations, provided robust causal control. Steering "Latent 12" showed a strong positive correlation (Pearson's R = 0.939) with the generation of IGHJ4 antibodies, demonstrating that hierarchical features can successfully drive global model behavior.

Discussion and Limitations

This research highlights a fundamental trade-off: architectures that produce the most intuitive, localized features (TopK) are not necessarily the most effective for model control. Hierarchical models like O-SAEs capture the abstract "global" concepts required for steering, even if their activation patterns are more difficult for humans to interpret visually. Future work will focus on scaling these techniques to identify domain-specific abstractions that allow for the precise design of antibodies with optimized clinical properties.

Summary updated

Abstract

Sparse autoencoders (SAEs) are a mechanistic interpretability technique that have been used to provide insight into learned concepts within large protein language models. Here, we employ TopK and Ordered SAEs to investigate an autoregressive antibody language model, p-IgGen, and steer its generation. We show that TopK SAEs can reveal biologically meaningful latent features, but high feature concept correlation does not guarantee causal control over generation. In contrast, Ordered SAEs impose an hierarchical structure that reliably identifies steerable features, but at the expense of more complex and less interpretable activation patterns. These findings advance the mechanistic interpretability of domain-specific protein language models and suggest that, while TopK SAEs are sufficient for mapping latent features to concepts, Ordered SAEs are preferable when precise generative steering is required.
BibTeX
@inproceedings{haque2026mechanistic,
  author = {Reina Haque and Oliver M. Turnbull and Anisha Parsan and Nithin Parsan and John Yang and Charlotte M. Deane},
  title = {{Mechanistic Interpretability of Antibody Language Models Using SAEs}},
  booktitle = {LMRL @ ICLR},
  year = {2026},
  url = {https://openreview.net/forum?id=LlYKVD7iso},
}
Figure