Abstract
This paper describes SimpleNER, a model developed for the sentence simplification task at GEM-2021. Our system is a monolingual Seq2Seq Transformer architecture that uses control tokens pre-pended to the data, allowing the model to shape the generated simplifications according to user desired attributes. Additionally, we show that NER-tagging the training data before use helps stabilize the effect of the control tokens and significantly improves the overall performance of the system. We also employ pretrained embeddings to reduce data sparsity and allow the model to produce more generalizable outputs.
Introduction
Sentence simplification aims to reduce the linguistic complexity of a text while preserving its original meaning. Modern Transformer models, while effective, often struggle with two main issues: a lack of explicit control over simplification attributes and a tendency to "hallucinate" or incorrectly replace named entities. For example, a model might replace a rare city name with a generic pronoun, destroying factual accuracy. SimpleNER addresses these challenges by combining attribute-control tokens with a robust Named Entity Recognition (NER) preprocessing strategy.
Methodology
The SimpleNER system enhances a standard Seq2Seq Transformer architecture with three key components:
- Control Tokens: Prepending specific tokens to the input allows users to guide the generation process based on attributes like compression ratio, paraphrasing (Levenshtein similarity), and lexical complexity (word rank).
- NER Mapping: The core contribution is a "tag-and-replace" strategy. Named entities (people, dates, locations) are identified and replaced with indexed tags (e.g.,
person@1). This reduces the model's effective vocabulary and prevents it from incorrectly modifying proper nouns. The original entities are restored in a post-processing step. - Pre-trained Embeddings: The model is initialized with 300-dimensional FastText embeddings to mitigate data sparsity and improve semantic understanding.
Evaluation and Results
The system was trained on the WikiLarge dataset and evaluated on the ASSET and TurkCorpus benchmarks. The authors prioritize the SARI metric, which specifically rewards the operations of adding, deleting, and keeping words appropriately, over the traditional BLEU score.
As shown in the results, the SimpleNER configuration consistently outperformed the baseline across all test sets in terms of SARI score. On the ASSET test set, the SARI score rose from 36.71 (Baseline) to 39.55 (SimpleNER).
Qualitative Analysis
Qualitative evaluation demonstrates the system's ability to maintain factual integrity. By treating specific terms as protected tags, SimpleNER avoids common errors found in baseline models, such as changing dates or replacing specific nouns with generic pronouns.
In the examples provided, the baseline model incorrectly alters dates (e.g., changing "2008" to "2007") or replaces specific names with pronouns (e.g., "aracaju" to "it"). SimpleNER avoids these errors by preserving the entity placeholders. However, limitations remain, such as occasional tag mismatches or semantic errors (e.g., replacing "difficult" with "simple").
Conclusion
SimpleNER shows that NER-tagging not only preserves factual accuracy but also stabilizes the effect of control tokens. By reducing vocabulary complexity, the model focuses more effectively on syntactic and lexical simplification. Future work could involve moving to contextual embeddings like BERT or RoBERTa to further improve nuance and style control.
Summary updated