SimpleNER Sentence Simplification System for GEM 2021

Kv Aditya Srivatsa, Shachar Mirkin, Manish Shrivastava2021 · DOI

Abstract

This paper describes SimpleNER, a model developed for the sentence simplification task at GEM-2021. Our system is a monolingual Seq2Seq Transformer architecture that uses control tokens pre-pended to the data, allowing the model to shape the generated simplifications according to user desired attributes. Additionally, we show that NER-tagging the training data before use helps stabilize the effect of the control tokens and significantly improves the overall performance of the system. We also employ pretrained embeddings to reduce data sparsity and allow the model to produce more generalizable outputs.

Introduction

Sentence simplification aims to reduce the linguistic complexity of a text while preserving its original meaning. Modern Transformer models, while effective, often struggle with two main issues: a lack of explicit control over simplification attributes and a tendency to "hallucinate" or incorrectly replace named entities. For example, a model might replace a rare city name with a generic pronoun, destroying factual accuracy. SimpleNER addresses these challenges by combining attribute-control tokens with a robust Named Entity Recognition (NER) preprocessing strategy.

Methodology

The SimpleNER system enhances a standard Seq2Seq Transformer architecture with three key components:

  1. Control Tokens: Prepending specific tokens to the input allows users to guide the generation process based on attributes like compression ratio, paraphrasing (Levenshtein similarity), and lexical complexity (word rank).
  2. NER Mapping: The core contribution is a "tag-and-replace" strategy. Named entities (people, dates, locations) are identified and replaced with indexed tags (e.g., person@1). This reduces the model's effective vocabulary and prevents it from incorrectly modifying proper nouns. The original entities are restored in a post-processing step.
  3. Pre-trained Embeddings: The model is initialized with 300-dimensional FastText embeddings to mitigate data sparsity and improve semantic understanding.

Evaluation and Results

The system was trained on the WikiLarge dataset and evaluated on the ASSET and TurkCorpus benchmarks. The authors prioritize the SARI metric, which specifically rewards the operations of adding, deleting, and keeping words appropriately, over the traditional BLEU score.

Performance comparison of SimpleNER against baselines on ASSET and TurkCorpus datasets.

As shown in the results, the SimpleNER configuration consistently outperformed the baseline across all test sets in terms of SARI score. On the ASSET test set, the SARI score rose from 36.71 (Baseline) to 39.55 (SimpleNER).

Qualitative Analysis

Qualitative evaluation demonstrates the system's ability to maintain factual integrity. By treating specific terms as protected tags, SimpleNER avoids common errors found in baseline models, such as changing dates or replacing specific nouns with generic pronouns.

Qualitative examples comparing the source text, baseline model outputs, and SimpleNER outputs.

In the examples provided, the baseline model incorrectly alters dates (e.g., changing "2008" to "2007") or replaces specific names with pronouns (e.g., "aracaju" to "it"). SimpleNER avoids these errors by preserving the entity placeholders. However, limitations remain, such as occasional tag mismatches or semantic errors (e.g., replacing "difficult" with "simple").

Conclusion

SimpleNER shows that NER-tagging not only preserves factual accuracy but also stabilizes the effect of control tokens. By reducing vocabulary complexity, the model focuses more effectively on syntactic and lexical simplification. Future work could involve moving to contextual embeddings like BERT or RoBERTa to further improve nuance and style control.

Summary updated

Abstract

This paper describes SimpleNER, a model developed for the sentence simplification task at GEM-2021. Our system is a monolingual Seq2Seq Transformer architecture that uses control tokens pre-pended to the data, allowing the model to shape the generated simplifications according to user desired attributes. Additionally, we show that NER-tagging the training data before use helps stabilize the effect of the control tokens and significantly improves the overall performance of the system. We also employ pretrained embeddings to reduce data sparsity and allow the model to produce more generalizable outputs.
BibTeX
@misc{srivatsa2021simplener,
  author = {KV Aditya Srivatsa and Shachar Mirkin and Manish Shrivastava},
  title = {{SimpleNER Sentence Simplification System for GEM 2021}},
  year = {2021},
  doi = {10.18653/v1/2021.gem-1.14},
}
Figure