Abstract
This paper investigates whether the power of the models pre-trained on text data, such as BERT, can be transferred to general token sequence classification applications. To verify pre-trained models' transferability, we test the pre-trained models on text classification tasks with meanings of tokens mismatches, and real-world non-text token sequence classification data, including amino acid, DNA, and music. We find that even on non-text data, the models pre-trained on text converge faster, perform better than the randomly initialized models, and only slightly worse than the models using task-specific knowledge. We also find that the representations of the text and non-text pre-trained models share non-trivial similarities.
Introduction
In Natural Language Processing (NLP), pre-trained models like BERT have demonstrated exceptional performance through domain adaptation—transferring knowledge from general corpora to specialized text. However, a deeper question remains: is this transferability limited to human language? This research explores discipline adaptability, the capacity for models pre-trained on text to transfer knowledge to discrete token sequences with fundamentally different distributions and structures, such as biological sequences (DNA, proteins) and symbolic music. The study aims to determine if BERT is a specialized "linguist" or a more general "sequence pattern learner."
Methods
The study evaluates several pre-trained architectures, including BERT, RoBERTa, and ALBERT, across two distinct experimental paradigms:
- Synthetic Text Permutation: Standard GLUE benchmark datasets are modified by applying a deterministic permutation to tokens. This process preserves the underlying grammatical structure while completely altering the semantic meaning of individual words.
- Real-World Non-Text Mapping: Non-textual tokens, such as amino acids or DNA bases, are mapped to BERT’s existing subword vocabulary. This allows the model to process biological or musical sequences as if they were text.
To analyze the internal mechanisms of this transfer, the authors utilized Projection Weighted Canonical Correlation Analysis (PWCCA) to compare BERT’s internal representations with those of domain-specific models (e.g., the PLUS model for proteins). Additionally, "re-embedding" experiments were conducted to isolate the contribution of pre-trained embeddings versus the transformer layers.
Datasets
The evaluation spans multiple disciplines:
- Synthetic GLUE: 8 tasks from the GLUE benchmark with permuted tokens.
- Protein Classification: Tasks involving localization, stability, and fluorescence using amino acid sequences.
- DNA Classification: Four tasks (H3, H4, H3K9ac, and Splice) based on DNA sub-sequences.
- Music Composer Classification: Pitch sequences from the MAESTRO-v1 dataset.
Key Results
The experimental results consistently support the hypothesis of cross-disciplinary transferability:
- Superior Performance: In all tested domains (DNA, Protein, Music), pre-trained models significantly outperformed models trained from scratch. For example, on protein localization, BERT achieved a score of 64.1, compared to 57.8 for a scratch model.
- Competitive with Domain-Specific Models: BERT's performance was only marginally lower than models pre-trained specifically on domain data, such as PLUS-TFM for proteins.
- Efficiency and Convergence: Pre-trained models exhibited much faster loss reduction and higher data efficiency. Even with only 1% of training data, pre-trained models maintained a significant lead over scratch models.
Key Insights
The findings suggest that BERT learns universal structural priors during text pre-training. It appears to capture fundamental principles of processing discrete sequences—such as hierarchical and relational structures—that are not exclusive to human language. Interestingly, the pre-trained embedding layer provides a "geometric" advantage; resetting these embeddings (the "re-emb" condition) led to performance degradation, indicating that the pre-trained embedding space itself contains useful structural information.
Conclusion and Future Work
While the study confirms "discipline adaptability," the exact structural properties of language that enable this transfer remain a mystery. Standard deep learning theories like dynamical isometry do not fully explain BERT's success in these non-textual domains. Future research will focus on identifying the specific "structural knowledge" BERT extracts and exploring whether pre-training on other structured data, such as source code or mathematical proofs, can further enhance cross-disciplinary transfer.
Summary updated