Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models' Transferability

Wei-Tsung Kao, Hung-Yi Lee

Abstract

This paper investigates whether the power of the models pre-trained on text data, such as BERT, can be transferred to general token sequence classification applications. To verify pre-trained models' transferability, we test the pre-trained models on text classification tasks with meanings of tokens mismatches, and real-world non-text token sequence classification data, including amino acid, DNA, and music. We find that even on non-text data, the models pre-trained on text converge faster, perform better than the randomly initialized models, and only slightly worse than the models using task-specific knowledge. We also find that the representations of the text and non-text pre-trained models share non-trivial similarities.

Introduction

In Natural Language Processing (NLP), pre-trained models like BERT have demonstrated exceptional performance through domain adaptation—transferring knowledge from general corpora to specialized text. However, a deeper question remains: is this transferability limited to human language? This research explores discipline adaptability, the capacity for models pre-trained on text to transfer knowledge to discrete token sequences with fundamentally different distributions and structures, such as biological sequences (DNA, proteins) and symbolic music. The study aims to determine if BERT is a specialized "linguist" or a more general "sequence pattern learner."

Methods

The study evaluates several pre-trained architectures, including BERT, RoBERTa, and ALBERT, across two distinct experimental paradigms:

  1. Synthetic Text Permutation: Standard GLUE benchmark datasets are modified by applying a deterministic permutation to tokens. This process preserves the underlying grammatical structure while completely altering the semantic meaning of individual words.
  2. Real-World Non-Text Mapping: Non-textual tokens, such as amino acids or DNA bases, are mapped to BERT’s existing subword vocabulary. This allows the model to process biological or musical sequences as if they were text.
Illustration of (a) permutation on text data and (b) mapping amino acid sequences to natural language tokens for model input.

To analyze the internal mechanisms of this transfer, the authors utilized Projection Weighted Canonical Correlation Analysis (PWCCA) to compare BERT’s internal representations with those of domain-specific models (e.g., the PLUS model for proteins). Additionally, "re-embedding" experiments were conducted to isolate the contribution of pre-trained embeddings versus the transformer layers.

Datasets

The evaluation spans multiple disciplines:

  • Synthetic GLUE: 8 tasks from the GLUE benchmark with permuted tokens.
  • Protein Classification: Tasks involving localization, stability, and fluorescence using amino acid sequences.
  • DNA Classification: Four tasks (H3, H4, H3K9ac, and Splice) based on DNA sub-sequences.
  • Music Composer Classification: Pitch sequences from the MAESTRO-v1 dataset.

Key Results

The experimental results consistently support the hypothesis of cross-disciplinary transferability:

  • Superior Performance: In all tested domains (DNA, Protein, Music), pre-trained models significantly outperformed models trained from scratch. For example, on protein localization, BERT achieved a score of 64.1, compared to 57.8 for a scratch model.
  • Competitive with Domain-Specific Models: BERT's performance was only marginally lower than models pre-trained specifically on domain data, such as PLUS-TFM for proteins.
  • Efficiency and Convergence: Pre-trained models exhibited much faster loss reduction and higher data efficiency. Even with only 1% of training data, pre-trained models maintained a significant lead over scratch models.
Performance comparison on DNA classification tasks, showing pre-trained models outperforming training from scratch.

Key Insights

The findings suggest that BERT learns universal structural priors during text pre-training. It appears to capture fundamental principles of processing discrete sequences—such as hierarchical and relational structures—that are not exclusive to human language. Interestingly, the pre-trained embedding layer provides a "geometric" advantage; resetting these embeddings (the "re-emb" condition) led to performance degradation, indicating that the pre-trained embedding space itself contains useful structural information.

Conclusion and Future Work

While the study confirms "discipline adaptability," the exact structural properties of language that enable this transfer remain a mystery. Standard deep learning theories like dynamical isometry do not fully explain BERT's success in these non-textual domains. Future research will focus on identifying the specific "structural knowledge" BERT extracts and exploring whether pre-training on other structured data, such as source code or mathematical proofs, can further enhance cross-disciplinary transfer.

Summary updated

Abstract

This paper investigates whether the power of the models pre-trained on text data, such as BERT, can be transferred to general token sequence classification applications. To verify pre-trained models' transferability, we test the pre-trained models on text classification tasks with meanings of tokens mismatches, and real-world non-text token sequence classification data, including amino acid, DNA, and music. We find that even on non-text data, the models pre-trained on text converge faster, perform better than the randomly initialized models, and only slightly worse than the models using task-specific knowledge. We also find that the representations of the text and non-text pre-trained models share non-trivial similarities.
BibTeX
@inproceedings{kao2021is,
  author = {Wei-Tsung Kao and Hung-Yi Lee},
  title = {{Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models' Transferability}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=ezXWMdaDeFQ},
}

Loading related content...

Figure