TL;DR
This paper applies Sparse Autoencoders (SAEs) to autoregressive antibody language models to decode their internal representations and steer sequence generation. The authors discover a striking trade-off: while standard TopK SAEs excel at mapping clean, human-interpretable features to biological concepts, Ordered SAEs (O-SAEs) are required to actually exert precise, causal control over the generated antibody properties.
What Problem Does This Paper Address?
Antibody language models hold immense promise for accelerating drug discovery by generating diverse, developable therapeutic candidates. However, these models operate as "black boxes," making it difficult for researchers to trust their predictions or intentionally guide the generation of specific structural features.
While mechanistic interpretability tools like Sparse Autoencoders (SAEs) have successfully disentangled concepts in general protein language models, they hadn't yet been applied to autoregressive, antibody-specific architectures. The authors set out to bridge this gap, aiming to identify where and how critical features—like germline gene identities and complementarity-determining regions (CDRs)—are encoded, and whether they can be manipulated to rationally design new antibodies.
Methods
The researchers investigated two distinct SAE architectures applied to p-IgGen, a 17-million-parameter, GPT-like decoder-only antibody language model:
- TopK SAEs: A standard approach that forces sparsity by allowing only the top $k$ most active latent features to fire at any given time.
- Ordered SAEs (O-SAEs): A nested architecture that enforces a strict hierarchical ordering of latents. Early latents are penalized more heavily if they fail to capture coarse, global structures, forcing the model to learn abstract concepts first and fine-grained details later.
To validate their findings across scales and architectures, they also replicated key experiments on two other models: IgLM (13M parameters) and ProGen2-OAS (764M parameters).
Features were identified using linear probes (logistic regression) trained on latent activations to predict specific biological properties. Generative steering was achieved by scaling the decoder vector of a target latent feature and adding it directly back into the model's hidden states during generation.
Datasets
- SAE Training: Hidden layer activations were extracted from 1,800,545 VH/VL paired sequences sourced from the Observed Antibody Space (OAS) database.
- Targeted Feature Identification: A curated dataset of 149,069 sequences compiled from OAS, the Coronavirus Antibody Database (CoV-AbDab), and the Patent and Literature Antibody Database (PLAbDab). Sequences were clustered using CD-HIT based on CDR sequence similarity to ensure robust evaluation.
Key Insights and Ideas
- Correlation $\neq$ Causation in Interpretability: High feature-concept correlation does not guarantee generative control. TopK SAEs yielded features that perfectly correlated with specific germline genes, but amplifying these features failed to predictably alter the generated output.
- The Culprit is Feature Splitting: TopK SAEs suffer from "feature splitting," where higher-order, abstract concepts (like sequence-level gene identity) get broken down into hyper-specific, residue-level contextual examples. Activating just one of these fragments isn't enough to steer the whole model.
- Hierarchy Enables Control: Ordered SAEs overcome feature splitting by enforcing a nested structure. This forces the network to capture global, abstract concepts in its first few latents, making them highly effective handles for steering model generation.
- The Interpretability Trade-Off: There is a literal price to pay for steerability. While TopK latents activate in clean, localized, and intuitive sequence positions, O-SAE latents exhibit complex, scattered activation patterns that are much harder for humans to visually interpret.
Key Results
- Information Preservation: TopK SAE latents successfully compress model activations without losing vital data, achieving a validation accuracy of 0.99 for residue-level CDR identity (outperforming raw hidden neurons at 0.98) and a macro $F_1$ score of 0.93 for heavy J gene prediction.
- Visual Interpretability: TopK latents cleanly isolate biological regions. For example, latents tracking the CDRH3 region or the IGHJ3 gene activate strictly within those specific sequence boundaries, whereas raw neurons fire chaotically across the entire sequence.
- Successful Generative Steering: Using O-SAEs, the authors successfully steered
p-IgGengeneration. Positively steering on O-SAE latent 12 predictably increased the proportion of IGHJ4 genes in the generated library (Pearson $R = 0.943$), while steering on latent 49 successfully decreased it (Pearson $R = -0.980$).
- Maintaining Naturalness: The steered antibody libraries remained highly diverse and biologically plausible, showing only minor shifts in Hamming distance and maintaining high ESM-2 protein-likeness scores compared to natural baselines.
Limitations and Future Work
The primary bottleneck highlighted by the authors is the severe lack of large-scale, functionally annotated antibody datasets. Unlike general protein models that can leverage massive databases like UniRef, antibody models lack the labeled data required for automated feature annotation.
Additionally, while O-SAEs unlock steerability, their scattered activation patterns make them difficult to interpret visually. Future work will need to focus on scaling SAE training to more expansive datasets (such as FLAb) and developing methods that combine the crisp, localized interpretability of TopK SAEs with the robust causal control of Ordered SAEs.
Summary updated