Abstract
Unlike the predictable scaling laws in natural language processing and computer vision, protein language models (PLMs) scale poorly: for many tasks, models within the same family plateau or even decrease in performance, with mid-sized models often outperforming the largest in the family. We introduce Reverse Distillation, a principled framework that decomposes large PLM representations into orthogonal subspaces guided by smaller models of the same family. The resulting embeddings have a nested, Matryoshka-style structure: the first $k$ dimensions of a larger model's embedding are exactly the representation from the smaller model. This ensures that larger reverse-distilled models consistently outperform smaller ones. A motivating intuition is that smaller models, constrained by capacity, preferentially encode broadly-shared protein features. Reverse distillation isolates these shared features and orthogonally extracts additional contributions from larger models, preventing interference between the two. On ProteinGym benchmarks, reverse-distilled ESM-2 variants outperform their respective baselines at the same embedding dimensionality, with the reverse-distilled 15 billion parameter model achieving the strongest performance.
The Protein Scaling Paradox
In natural language processing, the "scaling law" is a well-established phenomenon: increasing parameters and data consistently leads to better performance. However, Protein Language Models (PLMs) frequently hit a "scaling wall." In the widely used ESM-2 family, for instance, the 15-billion-parameter model often underperforms the 650-million-parameter version on functional prediction tasks.
The authors hypothesize that this occurs due to "feature interference." Small models, limited by capacity, are forced to learn fundamental, broadly shared biological features (such as secondary structure). Large models possess the capacity to learn complex, rare patterns (such as specific enzyme motifs), but these high-level features often become "entangled" with basic ones. This entanglement creates noise that can degrade the performance of linear classifiers used to probe these models.
The Reverse Distillation Framework
To address this, the researchers propose Reverse Distillation, which inverts the traditional knowledge distillation paradigm. Instead of a small student mimicking a large teacher, the representations of smaller models are used as a foundational "basis" for larger ones.
The framework follows a four-step process:
- Compute Model Representations: Generate embeddings for the same protein sequence across different model scales (e.g., ESM2-8M and ESM2-150M).
- Learn Linear Mapping: Use Principal Component Regression (PCR) to map the small model's embedding into the larger model's space, identifying the information shared between them.
- Summarize Residual Subspace: Calculate the "residual"—the unique information the large model possesses that the small model lacks—and summarize it using Singular Value Decomposition (SVD).
- Construct Matryoshka-style Embedding: Create a nested embedding where the first $k$ dimensions of a large model's representation are identical to the smaller model's representation. This process can be "chained" across an entire family (8M → 35M → ... → 15B), ensuring each step adds a layer of unique, orthogonal information.
Restoring Scaling and Biological Utility
By forcing the large model's unique features to be orthogonal to the small model's features, the framework prevents destructive interference. This restores monotonic scaling: on ProteinGym benchmarks (measuring deep mutational scanning effects), the reverse-distilled (rd) models showed consistent improvement with size. Notably, the rd.15B model achieved the strongest performance, finally surpassing the 650M and 3B versions that previously outperformed the baseline 15B.
Beyond performance, the method offers significant practical advantages. Because the embeddings are nested, users can truncate a large embedding to a smaller size and still retain a high-performing representation corresponding to a smaller model scale. Furthermore, the computational overhead is low; running the entire chain for a 15B model takes only about 1.7x the time of running the 15B model alone, as smaller models are highly efficient.
Conclusion and Future Work
Reverse Distillation provides a mathematically clean way to disentangle biological features across model scales. Analysis using Sparse Autoencoders (SAEs) confirmed that rd-models capture more specific and functionally relevant Gene Ontology terms than their baseline counterparts. While the current linear approach is highly interpretable, future work may explore non-linear relationships and parameter-efficient fine-tuning (like LoRA) to further optimize the generation of these distilled embeddings.
Summary updated