A Benchmark for Text Expansion: Datasets, Metrics, and Baselines

Yi Chen, Haiyun Jiang, Wei Bi, Rui Wang, Longyue Wang, Shuming Shi, Ruifeng XuarXiv 2023 · DOI · arXiv

Abstract

Text Expansion (TE) is a novel natural language processing task focused on enriching plain text by seamlessly inserting fine-grained modifiers—such as adjectives, adverbs, and idioms—to make writing more vivid and concrete. This paper presents a comprehensive benchmark for TE, featuring a massive dataset of 12 million pairs across English and Chinese, a novel evaluation metric called Info-Gain, and a robust "Locate & Infill" modeling paradigm. Our experiments demonstrate that while large language models like ChatGPT excel in fluency, specialized "Locate & Infill" models are superior in maintaining semantic integrity and achieving high fidelity.

Introduction: Defining Text Expansion

While modern writing assistants effectively correct grammar or paraphrase sentences, they often lack the capability for creative "polishing"—the addition of descriptive nuances that make human writing engaging. Existing tasks like "Text Infilling" or "Story Elaboration" typically involve fixed insertion points or allow for broad, wandering content.

In contrast, Text Expansion (TE) requires a model to:

  1. Identify flexible locations where modifiers can be added without disrupting the linguistic flow.
  2. Generate contextually rich modifiers that enhance the original sentence.
  3. Preserve core semantics, ensuring the fundamental meaning remains unchanged.
Examples of Text Expansion (TE) in Chinese and English, highlighting the insertion of various modifiers like ADJP, ADVP, and idioms.

Methods and Metrics

The researchers explored two primary modeling paradigms for TE:

  • Text2Text: A standard sequence-to-sequence approach using models like T5, BART, or GPT-2 to transform a source sentence into an expanded version.
  • Locate & Infill: A specialized two-step process. First, a model identifies "anchors" (typically nouns or verbs) and places mask tokens around them. Second, an infilling model populates these masks. This can be implemented as a Pipelined system (separate models) or a Joint system (a single model performing both tasks).

To evaluate these models, the authors introduced Info-Gain. Traditional metrics like BLEU or Perplexity often fail to distinguish between useful descriptive expansion and redundant "fluff." Info-Gain utilizes a pre-trained model to quantify how much the added modifiers assist in "recovering" the original context, thereby measuring the specific information added.

Dataset Construction

A massive benchmark was constructed for both English and Chinese, consisting of:

  • 12 Million Automatically Generated Pairs: These were created using four strategies: Neural Sentence Compression (NSC) (inverse summarization), Constituency Tree Pruning (CTP) (stripping modifiers via linguistic rules), IsA Relationship (IAR) (extracting descriptive phrases from hypernym patterns), and Masked Modifier Prediction (MMP) (proactively suggesting expansions).
  • 2K Human-Annotated Test Set: A high-quality, manually refined set used for rigorous final evaluation.

Key Results and the ChatGPT Paradox

The experimental results highlight a significant performance gap between specialized models and general-purpose LLMs.

Performance comparison of various models on English Text Expansion. Specialized "Locate & Infill" models achieve near-perfect fidelity compared to Text2Text baselines and ChatGPT.

The Superiority of Locate & Infill

In both languages, the Locate & Infill models achieved nearly 100% Fidelity, meaning they successfully preserved all original tokens without unauthorized deletions or rewrites. They also outperformed Text2Text baselines on the Info-Gain metric. Interestingly, the Joint model performed better for English, while the Pipelined model was superior for Chinese, likely due to structural differences in how modifiers are positioned in each language.

The ChatGPT Paradox

When evaluated on TE, ChatGPT produced long, fluent expansions. However, it failed significantly on Fidelity (scoring only ~2.4% in English). This is because ChatGPT tends to rewrite the entire sentence rather than strictly inserting modifiers, often changing the original wording and potentially altering the core semantics.

Conclusion and Future Work

This work establishes a solid foundation for Text Expansion. While the proposed models show high fidelity and informativeness, several challenges remain. Current models do not yet check for factual consistency (e.g., ensuring a modifier like "young" is historically accurate) and often default to "safe," generic modifiers. Future research will focus on improving the specificity of expansions and developing more granular evaluation metrics.

Summary updated

Abstract

This work presents a new task of Text Expansion (TE), which aims to insert fine-grained modifiers into proper locations of the plain text to concretize or vivify human writings. Different from existing insertion-based writing assistance tasks, TE requires the model to be more flexible in both locating and generation, and also more cautious in keeping basic semantics. We leverage four complementary approaches to construct a dataset with 12 million automatically generated instances and 2K human-annotated references for both English and Chinese. To facilitate automatic evaluation, we design various metrics from multiple perspectives. In particular, we propose Info-Gain to effectively measure the informativeness of expansions, which is an important quality dimension in TE. On top of a pre-trained text-infilling model, we build both pipelined and joint Locate&Infill models, which demonstrate the superiority over the Text2Text baselines, especially in expansion informativeness. Experiments verify the feasibility of the TE task and point out potential directions for future research toward better automatic text expansion.
BibTeX
@misc{chen2023benchmark,
  author = {Yi Chen and Haiyun Jiang and Wei Bi and Rui Wang and Longyue Wang and Shuming Shi and Ruifeng Xu},
  title = {{A Benchmark for Text Expansion: Datasets, Metrics, and Baselines}},
  year = {2023},
  eprint = {2309.09198},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.09198},
}
Figure