Towards functional annotation with latent protein language model features

Jake Silberg, Elana Simon, James Zou

TL;DR

This paper introduces a novel pipeline for annotating protein functions using interpretable features derived from protein language models (PLMs). By combining expanded database matching, structural analysis, and large language models (LLMs), they significantly improve the ability to identify and understand these features, leading to the discovery of more granular functional insights and the detection of previously missed annotations.

What Problem Does This Paper Address?

The rapid growth of protein sequence data, especially from metagenomics, has outpaced our ability to functionally annotate these proteins. Traditional methods, relying on sequence similarity, struggle with distantly related proteins. While protein language models (PLMs) like ESM-2 can capture evolutionary patterns, their raw embeddings are not easily interpretable for functional annotation. Sparse Autoencoders (SAEs) applied to PLMs can generate interpretable features, but identifying which of these features are reliable and correspond to meaningful biological concepts for annotation has been a major challenge. Many features can be "polysemantic" (activating on multiple unrelated concepts) or too abstract for human understanding.

Methods

The authors developed a three-pronged validation pipeline to identify cohesive and interpretable SAE features for protein annotation:

  1. Expanded Database Matching: They integrated a comprehensive set of over 20 annotation sources, including hierarchical codes (like Pfam clans and CATH topologies), to assess feature correspondence with known biological annotations. This goes beyond single-source matching and captures broader biological concepts.
  2. Feature-Guided Local Structural Alignment: This novel approach uses the activation patterns of SAE features to guide the alignment of protein structures. By focusing on regions where a feature strongly activates, they can identify structurally consistent subdomains, even if these subdomains are not explicitly annotated in existing databases. They use backbone RMSD100 for alignment, which is more lenient for longer conserved structures.
  3. LLM-Based Feature Description Generation: Large Language Models (LLMs), specifically Claude-3.5 Sonnet, were used to generate natural language descriptions of the biological patterns that trigger feature activation. This helps in interpreting features that might be missed by purely database-driven or structural methods.

Datasets

  • Training Data: The SAE features were derived from the Layer 18 of ESM-2 650M, resulting in 10,240 features.
  • Annotation Databases: The study utilized a comprehensive set of over 20 annotation sources, including UniProtKB, Pfam, CATH/Gene3D, and others.
  • Structural Data: AlphaFold-predicted structures were used for structural alignment.
  • Metagenomic Data: NMPfamsDB (Novel Metagenomic PfamsDB) was used to evaluate the pipeline's ability to annotate uncharacterized metagenomic sequences.

Key Insights and Ideas

  • Granularity Beyond Databases: SAE features can capture functional patterns at a finer resolution than existing protein databases, allowing for the identification of sub-domains within larger annotated regions.
  • Automated Discovery of Missing Annotations: The pipeline can systematically identify proteins that exhibit recognizable structural motifs (detected by SAE features) but lack corresponding database labels, significantly aiding in the discovery of missing annotations.
  • Bridging Sequence and Structure: By combining feature activation patterns with structural alignment, the method can link sequence-based features to specific 3D structural elements, providing a more holistic understanding.
  • LLMs as Interpreters: LLMs can effectively translate abstract feature activations into human-readable descriptions, uncovering biological insights that might be missed by traditional methods.

Key Results

  • The developed pipeline doubles annotation coverage, with over 60% of features showing strong correspondence to biological annotations (F1 > 0.8).
  • An additional 2.4% of features were identified through structural alignment, even with weaker database correspondence.
  • The pipeline automatically identified at least 491 missing CATH topology annotations.
  • It enabled rapid structural matching for metagenomic proteins, identifying 615 features with strong structural similarity to characterized proteins and allowing structural matches for at least 8,077 metagenomic proteins.
  • The LLM-based descriptions showed a high correlation with the F1 score of the best single annotation code, but also provided unique insights for features that span multiple annotation categories.

Limitations and Future Work

  • Manual Validation for LLM Insights: While LLMs can suggest literature, identifying the precise functional roles and verifying these insights often requires manual effort due to potential "hallucinations" and the need for careful verification.
  • Structural Motif Limitations: The local structural validation approach might miss features that span distant regions or involve highly flexible structural motifs.
  • Model Specificity: The current work focuses on a single layer of ESM-2 and Swiss-Prot for annotation validation, limiting the scope of analysis.

Future work will involve:

  • Expanding analysis across multiple layers of PLMs.
  • Systematically evaluating how latent feature quality influences structural consistency.
  • Conducting more rigorous benchmarking against existing tools like FoldSeek and Merizo-search.
  • Exploring advances in LLM capabilities and protein representations to further enhance annotation.

Summary updated

Abstract

Protein Language Models (PLMs) create high-dimensional embeddings that can be transformed into interpretable sparse features using Sparse Autoencoders (SAEs), where each feature activates on specific protein elements or patterns. However, scalably identifying which features are cohesive and reliable enough for protein annotation remains challenging. We address this by developing a validation pipeline combining three complementary methods: (1) expanded database matching across 20+ annotation sources including hierarchical codes, (2) feature-guided local structural alignment to identify structurally consistent activation regions, and (3) LLM-based feature description generation. Our annotation pipeline demonstrates three key properties of SAE features that make them a useful source of functional annotation complementary to existing methods. First, they can represent more granular patterns than existing protein databases, enabling the identification of sub-domains within proteins. Second, they can detect missing annotations by finding proteins that display recognizable structural motifs but lack corresponding database labels. Here, we automatically identify at least 491 missing CATH topology annotations with our pipeline. Third, they can maintain structural consistency across unseen proteins. Of our 10,240 SAE features, we find 615 that are consistently structurally similar in unannotated metagenomic proteins, allowing us to structurally match at least 8,077 metagenomic proteins to characterized proteins. This provides a rapid annotation pipeline with constant time search regardless of database size, that automatically includes structural and function information about the feature that triggered the match.
BibTeX
@inproceedings{silberg2025towards,
  author = {Jake Silberg and Elana Simon and James Zou},
  title = {{Towards functional annotation with latent protein language model features}},
  booktitle = {GenBio @ ICML},
  year = {2025},
  url = {https://openreview.net/forum?id=NmP8bJx9z3},
}

Loading related content...

Figure