Abstract
Tokenization is a central component of natural language processing in current large language models (LLMs), enabling models to convert raw text into processable units. Although learned tokenizers are widely adopted, they exhibit notable limitations, including their large, fixed vocabulary sizes and poor adaptability to new domains or languages. We present a family of models with up to 70 billion parameters based on the hierarchical autoregressive transformer (HAT) architecture. In HAT, an encoder transformer aggregates bytes into word embeddings and then feeds them to the backbone, a classical autoregressive transformer. The outputs of the backbone are then cross-attended by the decoder and converted back into bytes. We show that we can reuse available pre-trained models by converting the Llama 3.1 8B and 70B models into the HAT architecture: Llama-3.1-8B-TFree-HAT and Llama-3.1-70B-TFree-HAT are byte-level models whose encoder and decoder are trained from scratch, but where we adapt the pre-trained Llama backbone, i.e., the transformer blocks with the embedding matrix and head removed, to handle word embeddings instead of the original tokens. We also provide a 7B HAT model, Llama-TFree-HAT-Pretrained, trained entirely from scratch on nearly 4 trillion words. The HAT architecture improves text compression by reducing the number of required sequence positions and enhances robustness to intra-word variations, e.g., spelling differences. Through pre-training, as well as subsequent supervised fine-tuning and direct preference optimization in English and German, we show strong proficiency in both languages, improving on the original Llama 3.1 in most benchmarks. We release our models (including 200 pre-training checkpoints) on Hugging Face.
Introduction
Almost all modern Large Language Models (LLMs) rely on tokenization methods like BPE or WordPiece to bridge the gap between raw text and numerical processing. However, this reliance introduces several "original sins": static vocabularies that struggle with new domains, significant parameter waste in large embedding matrices, and fragility to minor character variations like typos. Furthermore, different languages exhibit varying "fertility" rates, leading to uneven computational costs for the same semantic content.
Aleph Alpha Research addresses these challenges with a "T-Free" (Tokenizer-Free) approach. By operating directly on raw UTF-8 bytes while maintaining the efficiency of word-level processing, these models achieve superior compression and robustness without the overhead of traditional tokenizers.
The HAT Architecture
The core innovation is the Hierarchical Autoregressive Transformer (HAT) architecture. Unlike standard "flat" transformers, HAT utilizes a three-part hierarchy:
- The Encoder: A small transformer that processes raw UTF-8 bytes using local causal attention. It aggregates these bytes into "word" embeddings based on a rule-based splitter.
- The Backbone: A standard autoregressive transformer (the "brain"). The authors demonstrate that existing models can be "HATified" by stripping the original embedding and output heads from a pre-trained backbone (like Llama 3.1) and training the HAT encoder/decoder around it.
- The Decoder: This component takes the backbone's high-level word predictions and converts them back into a sequence of bytes, using cross-attention to maintain fine-grained detail.
This hierarchy allows the heavy-duty backbone to operate on a compressed word level while the encoder and decoder manage the byte-level reality, reducing the non-backbone parameter footprint from 13% to less than 3% in models like Llama 3.1 8B.
Training and Evaluation
The model family includes a 7B parameter model trained from scratch on 4 trillion words (70% English, 7% German, 18% Code, 5% Math) and "HATified" versions of Llama 3.1 8B and 70B. Post-training involved Supervised Fine-Tuning (SFT) on approximately 2 million samples and Direct Preference Optimization (DPO) for alignment.
Evaluation was conducted across a comprehensive suite of benchmarks, including MMLU for knowledge, ARC and HellaSwag for reasoning, and German-specific variants like MMMLU and German ARC to test multilingual proficiency.
Key Results and Discussion
The HAT architecture demonstrates significant advantages in both performance and efficiency:
- Superior Performance: The Llama-3.1-8B-TFree-HAT outperformed the original Llama 3.1 8B on most benchmarks, despite having roughly 800 million fewer parameters. Gains were particularly notable in German benchmarks, where byte-level encoding better handles complex morphology.
- Enhanced Compression: HAT models consistently achieve higher "fertility," processing more bytes per sequence position (5.0–6.0) compared to standard tokenized models (3.5–4.5). This allows the backbone to be more information-dense.
- Intrinsic Robustness: Because the model operates on bytes, it can inherently recognize relationships between words (e.g., "car" and "cars") that token-based models must learn from scratch.
Conclusion
By liberating LLMs from static vocabularies, the HAT architecture provides a path toward more efficient, robust, and linguistically flexible models. While the hierarchical nature introduces new challenges for inference throughput and KV cache management, the performance gains and parameter savings suggest that tokenizer-free architectures are a viable and powerful alternative for the next generation of LLMs.
Summary updated