Long Unit Word Tokenization and Bunsetsu Segmentation of Historical Japanese

Hiroaki Ozaki, Kanako Komiya, Masayuki Asahara, Toshinobu OgisoACL 2024

TL;DR

This paper introduces a deep learning-based parser for historical Japanese that can segment sentences into "bunsetsu" (natural phrase units) and identify "Long Unit Words" (LUWs), a more complex word definition. The model achieves performance comparable to contemporary Japanese parsers, offering a valuable tool for analyzing grammatical evolution over centuries.

What Problem Does This Paper Address?

Analyzing historical Japanese presents unique challenges. Unlike many Western languages, Japanese lacks explicit word delimiters, leading to ambiguity in word segmentation. Linguists often rely on "bunsetsu," natural minimal phrases, as fundamental units for grammatical analysis. However, defining "words" in Japanese is complex, with two main categories: Short Unit Words (SUWs), the smallest meaningful units, and Long Unit Words (LUWs), which can comprise multiple SUWs and are often defined by higher-level syntactic structures. While SUW dictionaries exist, LUW dictionaries are scarce, especially for historical periods. This gap hinders the study of linguistic change. This paper addresses this by developing a parser for historical Japanese (specifically the Heian period and beyond) that can simultaneously identify bunsetsu and LUWs, enabling deeper diachronic linguistic analysis.

Methods

The authors propose a Transformer-based Language Model (TLM) that performs joint sequential labeling. This model simultaneously predicts three types of labels for each SUW token: bunsetsu BI (Beginning/Inside) tags, LUW BI tags, and LUW Part-of-Speech (POS) tags. To achieve this, the TLM first processes subword tokens derived from SUWs. A pooling layer (sum, max, or head pooling) then aggregates these subword representations into a single SUW representation. This pooled representation is then fed into a fully connected layer with a softmax activation to output the label probabilities. Two methods are explored for incorporating POS information: concatenating POS embeddings with the pooled SUW representation, and prepending POS information to the input text before subword tokenization ("incontext").

Datasets

  • Corpus of Historical Japanese (CHJ): This is the primary dataset for historical analysis, collected by NINJAL and spanning from the Nara to Meiji periods. Bunsetsu and LUW annotations were performed on sampled sentences from this corpus.
  • UD-Japanese-GSD: A contemporary Japanese corpus from Universal Dependencies, used for model comparison and finding optimal hyperparameters, as it already contains bunsetsu and LUW annotations and has existing deep-learning-based parsers.

Key Insights and Ideas

  • Deep Learning for Historical Linguistics: The paper demonstrates the effectiveness of modern deep learning techniques (Transformers) for analyzing historical linguistic data, achieving performance comparable to contemporary models.
  • Joint Modeling is Key: Simultaneously predicting bunsetsu, LUW, and POS tags within a single model proves effective, suggesting these linguistic units are closely related.
  • Pooling Strategies Matter: Different pooling strategies (sum, max, head) for aggregating subword representations have varying impacts. Sum pooling generally performed well for overall representation, while head pooling was better for boundary prediction.
  • Grammaticalization's Impact: The study highlights how grammaticalization of morphemes can influence bunsetsu segmentation, particularly evident in the Muromachi period.
  • Muromachi Period as a Turning Point: Significant grammatical changes appear to have occurred during the Muromachi period, making models trained on earlier periods less effective for this later era.

Key Results

  • The developed parser achieved F1 scores ranging from 0.976 to 0.996 for both bunsetsu and LUW reconstruction on historical Japanese corpora, comparable to contemporary Japanese parsers.
  • Models trained on the Heian and Kamakura periods showed good transferability to each other, but struggled with the Muromachi period, indicating a linguistic shift.
  • The "incontext" method for POS embedding improved precision but reduced recall, likely due to increased input length exceeding Transformer limits.
  • Analysis of sentence-ending particles revealed new usages emerging in the Muromachi period, which were difficult for models trained on earlier periods to predict.

Limitations and Future Work

  • Hyperparameter Optimization: The authors acknowledge that they did not perform an exhaustive hyperparameter search, suggesting potential for further performance improvements.
  • Applicable Periods: The current study focuses on the Heian period and later. Future work aims to expand the applicable periods.
  • Syntactic Parsing: The authors plan to build a syntactic parser by annotating dependencies between bunsetsu segments, which would offer a more comprehensive linguistic analysis.
  • Subword Tokenization Limits: The "incontext" method's performance was limited by the maximum subword token length of the Transformer model.

Summary updated

BibTeX
@inproceedings{ozaki2024long,
  author = {Hiroaki Ozaki and Kanako Komiya and Masayuki Asahara and Toshinobu Ogiso},
  title = {{Long Unit Word Tokenization and Bunsetsu Segmentation of Historical Japanese}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure