Learning a Formality-Aware Japanese Sentence Representation

Henry Li Xinyuan, Ray Lee, Jerry Chen, Kelly MarchisioarXiv 2023 · OpenReview

Abstract

While the way intermediate representations are generated in encoder-decoder sequence-to-sequence models typically allow them to preserve the semantics of the input sentence, input features such as formality might be left out. On the other hand, downstream tasks such as translation would benefit from working with a sentence representation that preserves formality in addition to semantics, so as to generate sentences with the appropriate level of social formality -- the difference between speaking to a friend versus speaking with a supervisor. We propose a sequence-to-sequence method for learning a formality-aware representation for Japanese sentences, where sentence generation is conditioned on both the original representation of the input sentence, and a side constraint which guides the sentence representation towards preserving formality information. Additionally, we propose augmenting the sentence representation with a learned representation of formality which facilitates the extraction of formality in downstream tasks. We address the lack of formality-annotated parallel data by adapting previous works on procedural formality classification of Japanese sentences. Experimental results suggest that our techniques not only helps the decoder recover the formality of the input sentence, but also slightly improves the preservation of input sentence semantics.

Introduction

In many languages, social context is as critical as literal meaning. In Japanese, this is explicitly encoded through grammar and morphology. Standard sequence-to-sequence models, however, are optimized for semantic reconstruction, often discarding stylistic nuances like formality. This loss of "social signal" can lead to translations that are factually accurate but socially inappropriate—for instance, using informal language with a superior. This research addresses the challenge of creating "formality-aware" sentence representations that preserve both meaning and social register.

Examples of formal and informal Japanese sentence endings.

Methodology

The researchers propose a modified Transformer architecture designed to "bake" formality into the intermediate sentence representation. The core of the approach involves a joint training objective and architectural augmentations.

Formality Augmentation

The model augments the standard encoder output with a learnable formality vector. Two primary methods were explored:

  1. Additive Augmentation: A "formal" or "informal" vector is added to the representation, effectively shifting its position in the embedding space.
  2. Multiplicative Augmentation: The representation is scaled by a formality mask.

Joint Objective Function

To ensure the model remains sensitive to style, a joint loss function is employed. The model is graded on two fronts:

  • Translation Quality Loss: Standard cross-entropy loss for word reconstruction.
  • Formality Awareness Loss: Bernoulli cross-entropy loss based on a pre-trained formality classifier.

A hyperparameter ($\lambda$) balances these two objectives, allowing for tuning between semantic fidelity and stylistic accuracy.

The proposed formality-aware sequence-to-sequence architecture.

Datasets and Labeling

A significant challenge in this domain is the lack of large-scale, formality-annotated parallel corpora. The authors addressed this by curating a dataset of approximately 53,000 Japanese sentences and applying an automated labeling script. This script identifies formality based on specific morphological markers, such as the presence of desu/masu (formal) versus short forms like da (informal).

Results and Discussion

The experimental results highlight the effectiveness of the proposed augmentations:

  • Classification Accuracy: The joint-training approach achieved a remarkable 99.5% accuracy in identifying formality, significantly outperforming a standard BERT-based classifier (79.4%).
  • Reconstruction Quality: Additive Augmentation yielded the best results for sentence reconstruction, achieving a BLEU score of 77.6 and a formality correctness of 83.0%. Interestingly, the formality objective also slightly improved semantic reconstruction, likely because it forces the model to attend more closely to sentence-final verb endings, which are semantically dense in Japanese.
  • Cross-Lingual Robustness: In back-translation tasks (Japanese $\rightarrow$ English $\rightarrow$ Japanese), Multiplicative Augmentation proved more robust. English often lacks explicit morphological markers for formality, making it a "lossy" intermediate step. The multiplicative mask helped maintain stylistic awareness even through this bottleneck.

Conclusion

This work demonstrates that stylistic features like formality can be effectively integrated into neural sentence representations without sacrificing semantic accuracy. By utilizing side constraints and learned augmentations, models can better preserve the social context necessary for natural communication in languages like Japanese. Future work may extend these techniques to more granular levels of honorifics (Keigo) or other stylistic markers such as emotional tone and reading level.

Summary updated

Abstract

While the way intermediate representations are generated in encoder-decoder sequence-to-sequence models typically allow them to preserve the semantics of the input sentence, input features such as formality might be left out. On the other hand, downstream tasks such as translation would benefit from working with a sentence representation that preserves formality in addition to semantics, so as to generate sentences with the appropriate level of social formality -- the difference between speaking to a friend versus speaking with a supervisor. We propose a sequence-to-sequence method for learning a formality-aware representation for Japanese sentences, where sentence generation is conditioned on both the original representation of the input sentence, and a side constraint which guides the sentence representation towards preserving formality information. Additionally, we propose augmenting the sentence representation with a learned representation of formality which facilitates the extraction of formality in downstream tasks. We address the lack of formality-annotated parallel data by adapting previous works on procedural formality classification of Japanese sentences. Experimental results suggest that our techniques not only helps the decoder recover the formality of the input sentence, but also slightly improves the preservation of input sentence semantics.
BibTeX
@misc{xinyuan2023learning,
  author = {Henry Li Xinyuan and Ray Lee and Jerry Chen and Kelly Marchisio},
  title = {{Learning a Formality-Aware Japanese Sentence Representation}},
  year = {2023},
  eprint = {2301.07209},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.07209},
}
Figure