Introduction
Modern Natural Language Processing (NLP) is predominantly built upon subword tokenization methods like Byte Pair Encoding (BPE) or WordPiece. While these methods offer a balance between vocabulary size and semantic coverage, they exhibit significant "brittleness" when encountering noisy text, typos, or low-resource languages. In such cases, subword tokenizers often decompose words into semantically opaque fragments. Conversely, character-level models provide extreme morphological robustness but often lack the deep semantic abstractions captured by subword-based pretraining.
This paper introduces a novel "entanglement model" designed to bridge this gap. By treating characters and subwords as distinct yet complementary modalities, the proposed architecture generates mutually informed representations that leverage the strengths of both granularities.
The Entanglement Model
The core innovation of this work is the Entanglement Model, which facilitates bidirectional information flow between subword and character encoders. Unlike previous hybrid approaches that often treat one granularity as secondary, this model treats them as equal partners in a cross-modal framework.
The architecture consists of:
- Dual Encoders: The model utilizes separate pretrained backbones—typically RoBERTa or XLM-R for subwords and CANINE for characters.
- Co-Attention Modules: Inspired by vision-language architectures like ViLBERT, the model employs $m$ layers of co-attention. In these layers, subword representations query character representations to extract fine-grained morphological features, while character representations query subwords for broader semantic context.
- Iterative Refinement: Through these layers, the representations become "entangled," resulting in character-aware subword embeddings and subword-aware character embeddings.
Experimental Evaluation
The model was evaluated across a diverse set of challenging NLP tasks:
- English Sequence Labeling: Tested on WNUT-17 (noisy social media) and CoNLL-03 (formal news).
- Multilingual NER: Evaluated on the MasakhaNER dataset, which covers 10 low-resource African languages.
- Text Classification: Benchmarked on WNUT-20 (COVID-19 tweets) and TweetEval.
- Character-level Labeling: Applied to intraword code-switching datasets (Spanish-Wixarika and German-Turkish).
Key Results and Insights
The entanglement model demonstrates superior performance, particularly in scenarios where standard subword models struggle.
State-of-the-Art in Multilingual NER
Using XLM-R-large as a backbone, the model achieved state-of-the-art results on the MasakhaNER dataset. It outperformed standalone models and previous benchmarks on 6 out of 10 African languages, demonstrating the power of character-level information in low-resource settings.
Robustness to Noise and Out-of-Distribution Data
On the WNUT-17 dataset, which is notorious for its noisy, informal text, the entanglement model significantly improved F1 scores compared to standalone subword or character models. The character-level side effectively serves as a "safety net," providing useful features when the subword tokenizer encounters out-of-distribution tokens or typos.
Emergent Alignment and Pretraining
A key finding was that the attention mechanism could learn spatial relationships between characters and subwords without explicit positional alignment. Furthermore, the researchers observed a "pretraining paradox": while the backbones are pretrained, extensive Masked Language Model (MLM) pretraining on the cross-attention layers themselves sometimes hindered performance, suggesting that fine-tuning is sufficient to connect the existing knowledge in the two encoders.
Conclusion
The entanglement model represents a significant step toward more robust and versatile NLP. By treating text granularities as modalities and allowing them to inform one another, the model achieves high performance on noisy, multilingual, and complex linguistic data. While the dual-encoder approach increases computational overhead, the gains in robustness and accuracy make it a compelling choice for challenging real-world applications.
Paper Citation: Wang, Y., Hu, X., & Gormley, M. (2024). Learning Mutually Informed Representations for Characters and Subwords. Findings of the Association for Computational Linguistics: NAACL 2024.
Summary updated