GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentation

Biyang Guo, Yeyun Gong, Yelong Shen, Songqiao Han, Hailiang Huang, Nan Duan, Weizhu ChenarXiv 2022 · DOI · arXiv

Abstract

GENIUS introduces a novel "reconstruction from sketch" pre-training objective that enables language models to generate high-quality, diverse text from a sparse outline of key phrases. By using an extreme masking strategy (up to 80%), the model acts as a powerful data augmentation tool (GeniusAug) that significantly boosts performance in low-resource and out-of-distribution NLP tasks.

Introduction

When humans write, we often start with a "sketch"—a few core ideas or keywords—and then fill in the surrounding context. However, most popular Pre-trained Transformer Models (PTMs) like BERT, BART, or T5 are trained to recover text from only minor corruptions (masking 15-30% of tokens).

This creates a gap: these models are excellent at "denoising" small errors but struggle with "sketch-based generation," where the majority of the content is missing. The authors aim to bridge this gap by creating a general-purpose model that can reconstruct entire documents from a skeleton of key information, which is particularly useful for generating synthetic data to train other models.

Methods

The researchers developed GENIUS (GENeratIon Using Sketch as input) using two primary innovations:

  1. Extreme and Selective Masking: Unlike standard models that mask random tokens, GENIUS uses an unsupervised keyword extraction algorithm (YAKE) to identify the most informative 20% of a text. The remaining 80% is replaced by single mask tokens [M]. This forces the model to learn how to bridge large gaps between semantically dense "islands" of text.
  2. GeniusAug Pipeline: To use GENIUS for data augmentation, the authors propose a "target-aware" extraction method. It uses Sentence-BERT to identify n-grams in a training sample that are most relevant to the specific task (e.g., the class label in sentiment analysis). GENIUS then generates new, diverse contexts around these essential parts.
  3. Sketch Mixup: A more advanced augmentation technique where the model combines key elements from multiple different training samples into a single "mixed" sketch to generate entirely new, label-preserving examples.
Overview of the GENIUS pre-training pipeline and its application in data augmentation across various NLP tasks.

The model architecture is based on a bidirectional encoder and an autoregressive decoder (initialized with BART weights).

Experimental Setup

The model was pre-trained on the C4-realnewslike corpus, consisting of roughly 27 million <sketch, text> pairs. Evaluation spanned several domains:

  • Text Classification: BBC, HuffPost, Yahoo, 20NG, IMDB, and SST2.
  • Named Entity Recognition (NER): CoNLL03.
  • Machine Reading Comprehension (MRC): SQuAD.

Key Insights and Results

The "Goldilocks" Zone of Augmentation

Existing methods are often either too "conservative" (making tiny changes that don't help diversity, like EDA) or too "aggressive" (generating entirely new text that might lose the original meaning, like GPT-2). GeniusAug finds a middle ground by keeping the "semantic core" (the sketch) while completely reimagining the surrounding "syntax and context."

Performance Gains

GENIUS achieved significantly lower perplexity (18.09 vs. 56.04 for BART-large) and higher ROUGE scores when reconstructing text from sketches. Using GeniusAug improved classification accuracy across the board. In "In-Distribution" (ID) settings, it provided a ~2-3% boost. In "Out-of-Distribution" (OOD) settings—where the model is tested on a different dataset than it was trained on—the gains were even larger, around 5-7%.

Comparison of ID and OOD accuracy across different augmentation methods as the number of labeled samples increases.

The model is particularly effective when labeled data is extremely scarce (e.g., only 50 labeled examples). Furthermore, the paper demonstrates that masking the less informative parts of a sentence is more effective for pre-training than random masking, as it teaches the model the underlying "connective tissue" of language.

Limitations and Future Work

While powerful, GENIUS requires more GPU resources for augmentation compared to simple rule-based methods. For tasks requiring strict logical reasoning, sketch extraction might miss subtle logical connectors, leading to noisy synthetic samples. In NER tasks, the model might "hallucinate" new entities not present in the original sketch. Future work includes exploring multilingual versions and precise control over generated text length.

Summary updated

Abstract

We introduce GENIUS: a conditional text generation model using sketches as input, which can fill in the missing contexts for a given sketch (key information consisting of textual spans, phrases, or words, concatenated by mask tokens). GENIUS is pre-trained on a large-scale textual corpus with a novel reconstruction from sketch objective using an extreme and selective masking strategy, enabling it to generate diverse and high-quality texts given sketches. Comparison with other competitive conditional language models (CLMs) reveals the superiority of GENIUS's text generation quality. We further show that GENIUS can be used as a strong and ready-to-use data augmentation tool for various natural language processing (NLP) tasks. Most existing textual data augmentation methods are either too conservative, by making small changes to the original text, or too aggressive, by creating entirely new samples. With GENIUS, we propose GeniusAug, which first extracts the target-aware sketches from the original training set and then generates new samples based on the sketches. Empirical experiments on 6 text classification datasets show that GeniusAug significantly improves the models' performance in both in-distribution (ID) and out-of-distribution (OOD) settings. We also demonstrate the effectiveness of GeniusAug on named entity recognition (NER) and machine reading comprehension (MRC) tasks. (Code and models are publicly available at https://github.com/microsoft/SCGLab and https://github.com/beyondguo/genius)
BibTeX
@misc{guo2022genius,
  author = {Biyang Guo and Yeyun Gong and Yelong Shen and Songqiao Han and Hailiang Huang and Nan Duan and Weizhu Chen},
  title = {{GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentation}},
  year = {2022},
  eprint = {2211.10330},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.10330},
}
Figure