Introduction
Large Language Models (LLMs) such as BERT and RoBERTa are predominantly pre-trained on formal, well-structured corpora like Wikipedia and published books. Consequently, their performance often degrades when processing user-generated content (UGC) from social media, which frequently contains slang, abbreviations, and non-standard syntax. Formality Style Transfer (FST) serves as a critical bridge, converting informal text into a formal equivalent to improve the robustness of downstream Natural Language Processing (NLP) tasks.
However, FST is inherently a low-resource task with limited parallel training data. While rule-based systems are often employed to augment data, they typically operate in a context-insensitive manner. For instance, a rule-based system might incorrectly map "extro" to "extra" instead of "extrovert" without considering the surrounding sentence, leading to semantic errors that propagate to downstream models.
Methodology: Context-Aware Rule Injection (CARI)
To address the limitations of context-insensitive methods, this paper proposes Context-Aware Rule Injection (CARI). CARI is an end-to-end BERT-based encoder-decoder model designed to resolve ambiguities in rule application by leveraging global sentence context.
The CARI framework involves four primary stages:
- Rule Detection: External APIs (e.g., Grammarbot) identify all potential formal replacements for informal tokens.
- Contextual Injection: Unlike previous "First Come First Served" approaches that select a single rule arbitrarily, CARI injects all detected candidate rules into the input stream.
- Input Formatting: Each candidate replacement is provided alongside its local context (a window of two tokens). These segments are concatenated to the original sentence using
[SEP]tokens. - Rule Selection: During fine-tuning, the BERT encoder-decoder learns to attend to the specific rule that best aligns with the global context of the sentence, effectively acting as a neural arbitrator for the rule-base.
Experimental Setup
The model was evaluated on both intrinsic and extrinsic metrics:
- Intrinsic Evaluation: The authors utilized the GYAFC (Grammarly’s Yahoo Answers Formality Corpus), specifically the "Entertainment & Music" (E&M) and "Family & Relationships" (F&R) domains. The training set was augmented with 100,000 back-translated pairs.
- Extrinsic Evaluation: To measure the impact on downstream tasks, the model was tested on SemEval-2018 Task 1 (Affect in Tweets) and Task 3 (Irony Detection).
Key Results
CARI achieved new state-of-the-art results on the GYAFC benchmark. BLEU scores improved from 72.7 to 74.31 in the Entertainment domain and from 77.2 to 78.05 in the Family domain.
As shown in Figure 3, while CARI requires more initial data to learn the selection mechanism compared to context-insensitive baselines, it achieves a significantly higher performance ceiling as training size increases. Furthermore, using CARI as a preprocessing step significantly boosted downstream performance; for example, RoBERTa's F1 score for Irony Detection improved from 72.6 to 73.7.
Discussion and Limitations
A key insight from the study is the "Context Window" sweet spot; a window size of 2 tokens was found to be optimal for providing sufficient disambiguation without over-cluttering the model's input. CARI also demonstrates a "safety net" capability, where the neural model can ignore erroneous suggestions from the rule-detection system if they contradict the global sentence context.
Despite its success, CARI remains dependent on the quality of the underlying rule-base. If a slang term is not recognized by the external API, the model cannot generate a correction. Future work may involve expanding the rule-base or integrating more diverse external knowledge sources to handle rarer linguistic variations.
Summary updated