Improving Formality Style Transfer with Context-Aware Rule Injection

Zonghai Yao, Hong Yu

Introduction

Large Language Models (LLMs) such as BERT and RoBERTa are predominantly pre-trained on formal, well-structured corpora like Wikipedia and published books. Consequently, their performance often degrades when processing user-generated content (UGC) from social media, which frequently contains slang, abbreviations, and non-standard syntax. Formality Style Transfer (FST) serves as a critical bridge, converting informal text into a formal equivalent to improve the robustness of downstream Natural Language Processing (NLP) tasks.

However, FST is inherently a low-resource task with limited parallel training data. While rule-based systems are often employed to augment data, they typically operate in a context-insensitive manner. For instance, a rule-based system might incorrectly map "extro" to "extra" instead of "extrovert" without considering the surrounding sentence, leading to semantic errors that propagate to downstream models.

Methodology: Context-Aware Rule Injection (CARI)

To address the limitations of context-insensitive methods, this paper proposes Context-Aware Rule Injection (CARI). CARI is an end-to-end BERT-based encoder-decoder model designed to resolve ambiguities in rule application by leveraging global sentence context.

Figure 1: Comparison between Context-Insensitive Rule Injection (CIRI) and Context-Aware Rule Injection (CARI). CARI allows the model to select the correct formal replacement based on local and global context.

The CARI framework involves four primary stages:

  1. Rule Detection: External APIs (e.g., Grammarbot) identify all potential formal replacements for informal tokens.
  2. Contextual Injection: Unlike previous "First Come First Served" approaches that select a single rule arbitrarily, CARI injects all detected candidate rules into the input stream.
  3. Input Formatting: Each candidate replacement is provided alongside its local context (a window of two tokens). These segments are concatenated to the original sentence using [SEP] tokens.
  4. Rule Selection: During fine-tuning, the BERT encoder-decoder learns to attend to the specific rule that best aligns with the global context of the sentence, effectively acting as a neural arbitrator for the rule-base.

Experimental Setup

The model was evaluated on both intrinsic and extrinsic metrics:

  • Intrinsic Evaluation: The authors utilized the GYAFC (Grammarly’s Yahoo Answers Formality Corpus), specifically the "Entertainment & Music" (E&M) and "Family & Relationships" (F&R) domains. The training set was augmented with 100,000 back-translated pairs.
  • Extrinsic Evaluation: To measure the impact on downstream tasks, the model was tested on SemEval-2018 Task 1 (Affect in Tweets) and Task 3 (Irony Detection).

Key Results

CARI achieved new state-of-the-art results on the GYAFC benchmark. BLEU scores improved from 72.7 to 74.31 in the Entertainment domain and from 77.2 to 78.05 in the Family domain.

Figure 3: Performance comparison (BLEU) across different training sizes for the Entertainment & Music (left) and Family & Relationship (right) domains.

As shown in Figure 3, while CARI requires more initial data to learn the selection mechanism compared to context-insensitive baselines, it achieves a significantly higher performance ceiling as training size increases. Furthermore, using CARI as a preprocessing step significantly boosted downstream performance; for example, RoBERTa's F1 score for Irony Detection improved from 72.6 to 73.7.

Discussion and Limitations

A key insight from the study is the "Context Window" sweet spot; a window size of 2 tokens was found to be optimal for providing sufficient disambiguation without over-cluttering the model's input. CARI also demonstrates a "safety net" capability, where the neural model can ignore erroneous suggestions from the rule-detection system if they contradict the global sentence context.

Despite its success, CARI remains dependent on the quality of the underlying rule-base. If a slang term is not recognized by the external API, the model cannot generate a correction. Future work may involve expanding the rule-base or integrating more diverse external knowledge sources to handle rarer linguistic variations.

Summary updated

Abstract

Models pre-trained on large-scale regular text corpora often do not work well for user-generated data where the language styles differ significantly from the mainstream text. Here we present Context-Aware Rule Injection (CARI), an innovative method for formality style transfer (FST). CARI injects multiple rules into an end-to-end BERT-based encoder and decoder model. It learns to select optimal rules based on context. The intrinsic evaluation showed that CARI achieved the new highest performance on the FST benchmark dataset. Our extrinsic evaluation showed that CARI can greatly improve the regular pre-trained models' performance on several tweet sentiment analysis tasks.
BibTeX
@inproceedings{yao2021improving,
  author = {Zonghai Yao and Hong Yu},
  title = {{Improving Formality Style Transfer with Context-Aware Rule Injection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.124},
}
Figure