What is the best recipe for character-level encoder-only modelling?

Kris CaoACL 2023 · DOI

Subword tokenization, such as WordPiece or BPE, has become the standard in modern NLP. However, it introduces complexities, particularly in multilingual contexts where different languages compete for limited vocabulary space. This research deconstructs the design space of character-level language models to identify an optimal "recipe" for encoder-only architectures.

The Problem with Tokenization

Tokenization is often arbitrary. In multilingual settings, a fixed vocabulary size can lead to poor representation for certain languages. Character-level models avoid this by operating directly on raw characters or bytes. While models like CANINE, Charformer, and Perceiver exist, they were developed under different conditions, making it difficult to determine which architectural choices drive performance. This study aims to disentangle these variables.

Methods and Architecture

The researchers conducted a combinatorial experiment across several design dimensions: input representation, downsampling, upsampling, and pretraining objectives.

Downsampling and Upsampling

Since character sequences are significantly longer than subword sequences, models must compress them through downsampling. The study compared three primary methods:

  1. CANINE: Uses local attention and strided convolutions.
  2. Charformer: Employs gradient-based subword-like blocks.
  3. Perceiver: Utilizes cross-attention with latent queries.
Comparison of downsampling architectures: CANINE, Charformer, and Perceiver.

The process of mapping these compressed representations back to individual characters is known as upsampling. The study evaluated strategies from CANINE and Perceiver.

Upsampling strategies in CANINE and Perceiver architectures.

The resulting optimal configuration, named BORT (Bidirectional Orthographic Representation Technique), combines Charformer downsampling with CANINE upsampling.

Key Insights

  • The Tokenizer Paradox: The most effective way to train a "tokenizer-free" model is to use a tokenizer during pretraining for masking and prediction targets. This suggests that the inductive bias of morpheme-like units remains crucial.
  • Learnt vs. Hash Embeddings: Explicitly learned embeddings for common characters outperformed the random hash embeddings used in the original CANINE, providing better stability and performance.
  • Importance of Locality: Flexible architectures like Perceiver struggled with data efficiency. Local-attention and convolutional approaches (CANINE, Charformer) better capture linguistic morphology, which relies on units in close proximity.
  • Morphology-Semantics Trade-off: Models excelling at low-level morphological tasks often performed worse on high-level semantic tasks like Question Answering, suggesting a loss of fine-grained character information as semantic units are formed.

Results

BORT consistently outperformed the multilingual BERT (mBERT) baseline on downstream tasks. On TyDi-QA, BORT achieved an F1 score of 78.76 (vs. BERT's 76.97), and on WikiANN, it reached 90.65 F1 (vs. BERT's 90.29).

However, character-level models remain computationally demanding. BORT requires approximately 2.7x more FLOPs per forward pass than BERT and offers about 0.68x the throughput.

Conclusion and Future Work

While BORT provides a high-performance character-level alternative, it still relies on tokenizers during the pretraining phase. Future research will focus on eliminating this dependency and extending these findings to encoder-decoder or decoder-only architectures. The increased computational cost also remains a practical consideration for deployment.

Summary updated

BibTeX
@inproceedings{cao2023what,
  author = {Kris Cao},
  title = {{What is the best recipe for character-level encoder-only modelling?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.326},
}
Figure