JADES: New Text Simplification Dataset in Japanese Targeted at Non-Native Speakers

Akio Hayakawa, Tomoyuki Kajiwara, Hiroki Ouchi, Taro Watanabe2022 · DOI

Abstract

The user-dependency of Text Simplification makes its evaluation obscure. A targeted evaluation dataset clarifies the purpose of simplification, though its specification is hard to define. We built JADES (JApanese Dataset for the Evaluation of Simplification), a text simplification dataset targeted at non-native Japanese speakers, according to public vocabulary and grammar profiles. JADES comprises 3,907 complex-simple sentence pairs annotated by an expert. Analysis of JADES shows that wide and multiple rewriting operations were applied through simplification. Furthermore, we analyzed outputs on JADES from several benchmark systems and automatic and manual scores of them. Results of these analyses highlight differences between English and Japanese in operations and evaluations.

Introduction

Text Simplification (TS) aims to improve readability while preserving the core meaning of a text. However, the concept of "simplicity" is inherently subjective and depends heavily on the target audience—what is simple for a child may not be simple for a language learner. Most existing TS datasets treat simplicity as a one-size-fits-all metric, leading to inconsistent evaluations.

In the context of Japanese, the existing standard dataset, SNOW, has several limitations. It lacks specific grammatical guidance for annotators, relies on source material that is often already simple (textbooks), and its strict vocabulary limits can lead to verbose, unnatural phrasing. The authors of JADES argue that for TS to be truly useful, it must be targeted.

The JADES Dataset

To address these issues, the researchers introduced JADES (JApanese Dataset for the Evaluation of Simplification). This dataset is specifically targeted at non-native Japanese speakers at the JLPT N3 level, which corresponds to an ability to understand Japanese used in everyday situations (roughly CEFR B1).

JADES consists of 3,907 complex-simple sentence pairs. Unlike many datasets that rely on crowdsourcing, JADES was created by an expert in "Easy Japanese" to ensure strict adherence to JLPT N3 vocabulary (approx. 1,400 words) and grammar profiles. The source text for JADES comes from the WMT20 news translation task, providing a benchmark for simplifying real-world, complex news content.

Table 1: Comparison of dataset statistics between the existing SNOW dataset and the new JADES dataset.

As shown in Table 1, JADES features a significantly larger vocabulary in its complex sentences compared to SNOW, reflecting its use of more diverse source material. Notably, while over 25% of simplified sentences in SNOW are identical to the original, JADES contains zero identical pairs, indicating a much higher degree of transformation required for its target audience.

Analysis of Simplification Operations

A key finding from the JADES analysis is the necessity of "drastic rewriting." The authors categorized transformations into several operations, such as REPLACE, DELETE, and STRONG REWRITE.

Table 2: Frequency of sentences containing specific rewriting operations in JADES (J) compared to other datasets (N and S).

Table 2 highlights the diversity and intensity of operations in JADES. Nearly all sentences (97%) required at least one REPLACE operation, and 61% involved a STRONG REWRITE. This suggests that simple lexical substitution is insufficient for N3 learners; often, the entire sentence structure must be reimagined to meet the target criteria while maintaining meaning.

Experimental Results

The researchers evaluated several benchmark systems on JADES, including BART (a sequence-to-sequence transformer) and EditNTS (an edit-based model). The best-performing model, BART-SJ, utilized a two-step fine-tuning process: it was first trained on the large-scale SNOW dataset and then fine-tuned on the high-quality JADES dataset. This transfer learning approach proved more effective than training on either dataset in isolation.

While BART-SJ achieved the highest automatic scores (BLEU: 36.03, SARI: 58.12), human evaluation revealed a significant gap between machine outputs and human references. Models were rated relatively well for fluency but lagged behind human experts in simplicity. Furthermore, the study found that standard automatic metrics like SARI, while correlating best with simplicity, may not fully capture the nuances of Japanese simplification as effectively as they do for English.

Conclusion and Future Work

JADES provides a rigorous, targeted benchmark for Japanese text simplification. The dataset demonstrates that effective simplification for non-native speakers requires substantial structural and lexical changes. Future work will involve expanding the dataset with more annotators and conducting user studies with actual non-native speakers to validate the JLPT-based simplification criteria.

Summary updated

Abstract

The user-dependency of Text Simplification makes its evaluation obscure. A targeted evaluation dataset clarifies the purpose of simplification, though its specification is hard to define. We built JADES (JApanese Dataset for the Evaluation of Simplification), a text simplification dataset targeted at non-native Japanese speakers, according to public vocabulary and grammar profiles. JADES comprises 3,907 complex-simple sentence pairs annotated by an expert. Analysis of JADES shows that wide and multiple rewriting operations were applied through simplification. Furthermore, we analyzed outputs on JADES from several benchmark systems and automatic and manual scores of them. Results of these analyses highlight differences between English and Japanese in operations and evaluations.
BibTeX
@misc{hayakawa2022jades,
  author = {Akio Hayakawa and Tomoyuki Kajiwara and Hiroki Ouchi and Taro Watanabe},
  title = {{JADES: New Text Simplification Dataset in Japanese Targeted at Non-Native Speakers}},
  year = {2022},
  doi = {10.18653/v1/2022.tsar-1.17},
}
Figure