Which Encoding is the Best for Text Classification in Chinese, English, Japanese and Korean?

Xiang Zhang, Yann LeCunarXiv 2017 · arXiv

Abstract

This study presents a massive empirical evaluation of 473 models across 14 large-scale datasets to identify the most effective text encoding strategies for Chinese, Japanese, Korean (CJK), and English classification tasks. The findings highlight byte-level one-hot encoding (UTF-8) as a robust, language-agnostic approach that simplifies preprocessing without sacrificing performance.

Introduction

Natural Language Processing (NLP) has traditionally treated words as the fundamental unit of analysis. However, applying this paradigm to CJK languages introduces significant challenges. First, segmentation is difficult because these languages often lack explicit word boundaries, requiring complex tokenization. Second, the sheer volume of unique characters in CJK scripts makes standard one-hot encoding computationally prohibitive. This paper investigates whether a unified encoding mechanism can bypass these language-specific hurdles.

Methodology

The researchers evaluated three primary model families across multiple encoding levels, including UTF-8 bytes, characters, words, and their romanized counterparts.

Model Architectures

  1. Convolutional Networks (ConvNets):
    • GlyphNet: This novel approach treats characters as 16x16 pixel images (glyphs), attempting to capture semantic information directly from visual strokes.
  2. OnehotNet: Utilizes one-hot encoding at the byte (UTF-8) or romanized character level.
  3. EmbedNet: Employs lookup-table embeddings for various units.
  4. Linear Models: Multinomial logistic regression using bag-of-words/characters and n-grams with TF-IDF.
  5. fastText: A shallow neural network architecture optimized for speed using hierarchical softmax and n-gram features.
Examples of character glyphs used in GlyphNet.

Datasets

The study's scale is unprecedented, involving 14 datasets with millions of samples across four languages and multilingual "Joint" sets.

Summary of the 14 large-scale datasets used in the study.

Key Insights

The Efficiency of Byte-Level Encoding

A major finding is that treating text as a raw sequence of UTF-8 bytes is highly effective for ConvNets. This approach ignores linguistic boundaries entirely, providing a "win" for simplicity by removing the need for language-specific segmenters or romanization tools.

Performance vs. Speed

While GlyphNet offers a creative way to interpret characters, it proved computationally expensive. Training on 1 million samples took over 37 hours on a GPU, whereas fastText completed the same task in just 2-10 seconds. However, ConvNets demonstrated superior robustness, showing a much smaller generalization gap between training and testing errors compared to fastText, which was prone to overfitting with high-order n-grams.

The Resilience of Linear Models

Despite the rise of deep learning, simple linear models using word n-grams remain remarkably competitive. Even in CJK languages where segmentation is imperfect, these models often hold their own against more complex neural architectures.

Conclusion

Byte-level one-hot encoding emerges as a powerful, language-independent solution for text classification. While specialized models like GlyphNet provide interesting visual interpretations of text, the simplicity and robustness of byte-level ConvNets make them a compelling choice for multilingual NLP pipelines. Future research may extend these findings to more complex reasoning tasks and modern architectures like Transformers.

Summary updated

Abstract

This article offers an empirical study on the different ways of encoding Chinese, Japanese, Korean (CJK) and English languages for text classification. Different encoding levels are studied, including UTF-8 bytes, characters, words, romanized characters and romanized words. For all encoding levels, whenever applicable, we provide comparisons with linear models, fastText and convolutional networks. For convolutional networks, we compare between encoding mechanisms using character glyph images, one-hot (or one-of-n) encoding, and embedding. In total there are 473 models, using 14 large-scale text classification datasets in 4 languages including Chinese, English, Japanese and Korean. Some conclusions from these results include that byte-level one-hot encoding based on UTF-8 consistently produces competitive results for convolutional networks, that word-level n-grams linear models are competitive even without perfect word segmentation, and that fastText provides the best result using character-level n-gram encoding but can overfit when the features are overly rich.
BibTeX
@misc{zhang2017which,
  author = {Xiang Zhang and Yann LeCun},
  title = {{Which Encoding is the Best for Text Classification in Chinese, English, Japanese and Korean?}},
  year = {2017},
  eprint = {1708.02657},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1708.02657},
}
Figure