Scaling Up ESM2 Architectures for Long Protein Sequences Analysis: Long and Quantized Approaches

Gabriel Bianchin de Oliveira, Hélio Pedrini, Zanoni Dias

Abstract

State-of-the-art protein Transformer models, such as ESM2, are typically limited by a hard input sequence length of 1,024 tokens due to the quadratic computational cost of global self-attention. This study introduces "Long" and "Quantized" variants of the ESM2 architecture to address these scalability challenges. By implementing local sliding window attention, the researchers doubled the sequence capacity to 2,048 amino acids while maintaining linear complexity. Furthermore, 4-bit quantization significantly reduced the memory footprint, enabling the deployment of large-scale models on modest hardware. Experimental results on protein function prediction tasks demonstrate that these adaptations not only handle longer sequences natively but also frequently outperform standard models in predictive accuracy.

Introduction

In bioinformatics, Transformer models have become the standard for interpreting the "language of life." However, the ESM2 (Evolutionary Scale Modeling) family faces a significant bottleneck: a 1,024-token input limit. Because standard Transformers utilize global self-attention, computational requirements grow quadratically ($O(n^2)$) with sequence length. Processing longer proteins previously required lossy truncation or inefficient sliding window techniques. This research explores native architectural scaling to accommodate longer protein sequences without prohibitive GPU memory costs.

Methodology

Local Attention (The "Long" Version)

To overcome the quadratic scaling of global attention, the authors adopted a local sliding window mechanism inspired by the Longformer architecture. In this setup, each amino acid token only attends to its immediate neighbors within a fixed window (e.g., 1,024 tokens). This modification transitions the attention complexity from quadratic to linear ($O(nk)$), allowing the model to process up to 2,048 amino acids.

Comparison of Global Attention (a) and Local Attention (b) mechanisms.

Quantization and Optimization

To democratize access to these models, the researchers implemented 4-bit integer (int4) quantization. By converting standard 32-bit floating-point weights and utilizing Low-Rank Adaptation (LoRA) with bfloat16 computation, they maintained high accuracy while drastically reducing the model's memory footprint. Additionally, the study found that "warm-starting" the new 2,048-position models by copying context representations from original ESM2 weights was significantly more effective than random initialization.

Experimental Results

Performance in Protein Function Prediction

The models were evaluated using the CAFA5 challenge dataset, focusing on Gene Ontology (GO) categories: Biological Process (BPO), Cellular Component (CCO), and Molecular Function (MFO). The "Long" and "Quantized" versions consistently achieved higher $F_{max}$ scores than original ESM2 models, particularly for sequences exceeding 1,024 amino acids. For instance, on the T12 architecture for BPO, the Long version reached an $F_{max}$ of 0.533, surpassing the standard model's 0.493.

Memory Efficiency

Quantization proved highly effective for scaling. As shown in the table below, the memory required to load the T36 model dropped from over 11 GB to just 1.75 GB.

Memory requirements (in MB) for Standard, Long, and Quantized ESM2 variants across different model scales.

Conclusion

This work demonstrates that local attention and quantization are viable strategies for scaling protein language models. By doubling the sequence length and shrinking the hardware requirements, these models provide a more robust toolset for analyzing complex, long-form proteins. Future research will focus on extending these techniques to the largest architectures (T48) and evaluating performance on structural tasks like secondary structure prediction and contact map generation.

Summary updated

BibTeX
@inproceedings{oliveira2024scaling,
  author = {Gabriel Bianchin de Oliveira and Hélio Pedrini and Zanoni Dias},
  title = {{Scaling Up ESM2 Architectures for Long Protein Sequences Analysis: Long and Quantized Approaches}},
  booktitle = {BSB},
  year = {2024},
  doi = {10.5753/bsb.2024.244804},
}
Figure