Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models' Transferability

Wei-Tsung Kao, Hung-yi Lee

Abstract

Large language models (LLMs) like BERT are typically celebrated for their mastery of human linguistics. However, recent research presented at EMNLP 2021 suggests that their utility extends far beyond natural language. This paper investigates "discipline adaptability"—the ability of models pre-trained on text to transfer their learned representations to non-linguistic domains such as genomics, proteomics, and symbolic music. Our findings indicate that BERT-based architectures consistently outperform models trained from scratch on these diverse tasks, often approaching the performance of domain-specific expert models. This suggests that pre-training on natural language imparts universal structural principles that transcend human syntax.

Introduction

In Natural Language Processing (NLP), the benefits of transfer learning are well-documented: pre-training on a massive general corpus like Wikipedia significantly boosts performance on specialized downstream tasks. But a fundamental question remains: is this transferability limited to human language? This research explores whether the "knowledge" captured by BERT is purely linguistic or if it encompasses a more fundamental understanding of sequence structure that can be applied to entirely different disciplines.

The Concept of Discipline Adaptability

The authors define discipline adaptability as the capacity of a model pre-trained in one discipline (e.g., natural language) to adapt to another (e.g., biological sequences) where token distributions and underlying logic differ fundamentally. This is particularly relevant for fields like biology or music theory, where labeled data is often scarce, but raw sequence data is plentiful.

Experimental Methodology

To test this adaptability, the researchers employed two primary strategies to bridge the gap between natural language and "alien" domains:

  1. Synthetic Permutation: To isolate the importance of "meaning" versus "structure," the researchers applied a deterministic one-to-one mapping to standard GLUE benchmark tokens. This scrambled the vocabulary while preserving the underlying graph structure of the language.
  2. Cross-Domain Mapping: For real-world non-text data, discrete tokens from other disciplines—such as the 20 amino acids in proteins or the 4 bases in DNA—were mapped directly to BERT’s existing vocabulary.
Mapping strategies for synthetic text permutation and amino acid sequences.

These "language-heavy" models were then fine-tuned on target tasks and compared against two critical baselines: models initialized from scratch and "discipline-specific" models pre-trained directly on the target domain (e.g., PLUS for proteins or Hilbert-CNN for DNA).

Key Results and Performance

The empirical results across DNA, protein, and music tasks were striking. Pre-trained models (BERT, RoBERTa, ALBERT) consistently outperformed the "from-scratch" baselines across all tested domains.

Performance comparison across synthetic GLUE, protein, DNA, and music tasks.

As shown in Figure 2, the pre-trained models (blue bars) not only beat the scratch models (orange bars) but also remained highly competitive with domain-expert models (indicated by red lines). For instance, in protein classification, BERT performed only slightly below PLUS-TFM, a model pre-trained on millions of biological sequences.

Furthermore, the study highlighted the importance of the initial embedding layer. A "re-embedding" test (green bars), where the first layer was randomly re-initialized while keeping the rest of the pre-trained weights, resulted in degraded performance compared to the full pre-trained model. This suggests that the relationships learned even at the token-embedding level in natural language contribute to cross-disciplinary transfer.

Insights into Structural Universality

Why does a model trained on English Wikipedia help predict protein stability? The authors propose several insights:

  • Structural Universality: The success on scrambled synthetic data implies that BERT learns high-level logical or structural patterns inherent in sequences, rather than just memorizing a dictionary.
  • The "Head Start" Effect: Pre-trained models converged significantly faster than scratch models, often reaching lower loss values in a fraction of the training steps.
  • Latent Similarity: Using PWCCA to compare neural representations, the researchers found that BERT’s internal representations of protein sequences before fine-tuning were more similar to protein-specific models than to random initializations.

Limitations and Future Directions

Despite the clear evidence of transferability, the exact mechanism remains elusive. The researchers tested hypotheses regarding hierarchical structures and gradient stability (dynamical isometry), but neither provided a definitive explanation. Future work is needed to determine which specific properties of natural language—such as Zipf’s Law or recursive properties—serve as the "universal" foundation for this cross-disciplinary prowess. Understanding these properties could pave the way for more efficient, generalized pre-training strategies across all sequence-based sciences.

Summary updated

Abstract

This paper investigates whether the power of the models pre-trained on text data, such as BERT, can be transferred to general token sequence classification applications. To verify pre-trained models' transferability, we test the pre-trained models on text classification tasks with meanings of tokens mismatches, and real-world non-text token sequence classification data, including amino acid, DNA, and music. We find that even on non-text data, the models pre-trained on text converge faster, perform better than the randomly initialized models, and only slightly worse than the models using task-specific knowledge. We also find that the representations of the text and non-text pre-trained models share non-trivial similarities.
BibTeX
@inproceedings{kao2021is,
  author = {Wei-Tsung Kao and Hung-yi Lee},
  title = {{Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models' Transferability}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=ezXWMdaDeFQ},
}
Figure