Abstract
This study presents a massive empirical evaluation of 473 models across 14 large-scale datasets to identify the most effective text encoding strategies for Chinese, Japanese, Korean (CJK), and English classification tasks. The findings highlight byte-level one-hot encoding (UTF-8) as a robust, language-agnostic approach that simplifies preprocessing without sacrificing performance.
Introduction
Natural Language Processing (NLP) has traditionally treated words as the fundamental unit of analysis. However, applying this paradigm to CJK languages introduces significant challenges. First, segmentation is difficult because these languages often lack explicit word boundaries, requiring complex tokenization. Second, the sheer volume of unique characters in CJK scripts makes standard one-hot encoding computationally prohibitive. This paper investigates whether a unified encoding mechanism can bypass these language-specific hurdles.
Methodology
The researchers evaluated three primary model families across multiple encoding levels, including UTF-8 bytes, characters, words, and their romanized counterparts.
Model Architectures
- Convolutional Networks (ConvNets):
- GlyphNet: This novel approach treats characters as 16x16 pixel images (glyphs), attempting to capture semantic information directly from visual strokes.
- OnehotNet: Utilizes one-hot encoding at the byte (UTF-8) or romanized character level.
- EmbedNet: Employs lookup-table embeddings for various units.
- Linear Models: Multinomial logistic regression using bag-of-words/characters and n-grams with TF-IDF.
- fastText: A shallow neural network architecture optimized for speed using hierarchical softmax and n-gram features.
Datasets
The study's scale is unprecedented, involving 14 datasets with millions of samples across four languages and multilingual "Joint" sets.
Key Insights
The Efficiency of Byte-Level Encoding
A major finding is that treating text as a raw sequence of UTF-8 bytes is highly effective for ConvNets. This approach ignores linguistic boundaries entirely, providing a "win" for simplicity by removing the need for language-specific segmenters or romanization tools.
Performance vs. Speed
While GlyphNet offers a creative way to interpret characters, it proved computationally expensive. Training on 1 million samples took over 37 hours on a GPU, whereas fastText completed the same task in just 2-10 seconds. However, ConvNets demonstrated superior robustness, showing a much smaller generalization gap between training and testing errors compared to fastText, which was prone to overfitting with high-order n-grams.
The Resilience of Linear Models
Despite the rise of deep learning, simple linear models using word n-grams remain remarkably competitive. Even in CJK languages where segmentation is imperfect, these models often hold their own against more complex neural architectures.
Conclusion
Byte-level one-hot encoding emerges as a powerful, language-independent solution for text classification. While specialized models like GlyphNet provide interesting visual interpretations of text, the simplicity and robustness of byte-level ConvNets make them a compelling choice for multilingual NLP pipelines. Future research may extend these findings to more complex reasoning tasks and modern architectures like Transformers.
Summary updated