Abstract
Sparse autoencoders (SAEs) are a mechanistic interpretability technique that have been used to provide insight into learned concepts within large protein language models. Here, we employ TopK and Ordered SAEs to investigate an autoregressive antibody language model, p-IgGen, and steer its generation. We show that TopK SAEs can reveal biologically meaningful latent features, but high feature concept correlation does not guarantee causal control over generation. In contrast, Ordered SAEs impose an hierarchical structure that reliably identifies steerable features, but at the expense of more complex and less interpretable activation patterns. These findings advance the mechanistic interpretability of domain-specific protein language models and suggest that, while TopK SAEs are sufficient for mapping latent features to concepts, Ordered SAEs are preferable when precise generative steering is required.
Introduction
Protein language models (pLMs) have become essential tools in antibody drug discovery, assisting in tasks such as library generation and humanization. However, these models often function as "black boxes," making it difficult to verify if they are utilizing biologically relevant features or merely identifying statistical noise. Mechanistic interpretability seeks to address this by deconstructing model activations into human-understandable components. This study applies Sparse Autoencoders (SAEs) to p-IgGen, a 17M-parameter antibody language model, to extract biological features and evaluate their capacity for generative steering.
Methods
The researchers analyzed the hidden layer activations of p-IgGen using two distinct SAE architectures:
- TopK SAEs: A standard approach that enforces sparsity by activating only the top $k$ (e.g., 32) latents, favoring the discovery of localized, monosemantic features.
- Ordered SAEs (O-SAEs): A nested architecture that imposes a strict hierarchy. Early latents are forced to capture broad, global sequence structures, while later latents capture fine-grained details.
SAEs were trained on 1.8 million antibody sequences from the Observed Antibody Space (OAS). Linear probes were then used to map these latents to known biological concepts, such as Complementarity-Determining Region (CDR) identities and V/J germline gene classifications.
Key Insights: Disentangling Biological Concepts
A primary advantage of SAEs is their ability to disentangle "polysemantic" neurons—which respond to multiple unrelated concepts—into "monosemantic" latents. In p-IgGen, raw hidden neurons often exhibit noisy activation patterns across the entire sequence. In contrast, TopK SAEs isolate features that correspond precisely to structural regions like the CDRH3 or the Heavy J region.
However, the study identified a critical "Feature Splitting" phenomenon in TopK SAEs. High-level biological concepts (like a specific gene identity) are often decomposed into extremely localized features, such as a specific amino acid at a single position. While these features are highly predictive, they often lack the causal weight necessary to steer the model's global generation process.
Results: Predictive Power vs. Generative Control
The SAEs demonstrated exceptional accuracy in mapping latent features to biology. TopK latents achieved a 0.99 accuracy in predicting CDR identity and a 0.93 macro F1-score for heavy J gene (IGHJ) identity. Despite this predictive success, generative steering—intervening in the model's generation by adding latent vectors to hidden states—yielded divergent results:
- TopK Steering Failure: Amplifying TopK latents correlated with the IGHJ4 gene failed to increase the proportion of IGHJ4 in generated sequences. This is likely because TopK features are too localized; steering a single residue (e.g., at position 119) is insufficient to shift the model's entire generative trajectory.
- Ordered SAE Steering Success: O-SAEs, by capturing hierarchical and more abstract representations, provided robust causal control. Steering "Latent 12" showed a strong positive correlation (Pearson's R = 0.939) with the generation of IGHJ4 antibodies, demonstrating that hierarchical features can successfully drive global model behavior.
Discussion and Limitations
This research highlights a fundamental trade-off: architectures that produce the most intuitive, localized features (TopK) are not necessarily the most effective for model control. Hierarchical models like O-SAEs capture the abstract "global" concepts required for steering, even if their activation patterns are more difficult for humans to interpret visually. Future work will focus on scaling these techniques to identify domain-specific abstractions that allow for the precise design of antibodies with optimized clinical properties.
Summary updated