Abstract
GENIUS introduces a novel "reconstruction from sketch" pre-training objective that enables language models to generate high-quality, diverse text from a sparse outline of key phrases. By using an extreme masking strategy (up to 80%), the model acts as a powerful data augmentation tool (GeniusAug) that significantly boosts performance in low-resource and out-of-distribution NLP tasks.
Introduction
When humans write, we often start with a "sketch"—a few core ideas or keywords—and then fill in the surrounding context. However, most popular Pre-trained Transformer Models (PTMs) like BERT, BART, or T5 are trained to recover text from only minor corruptions (masking 15-30% of tokens).
This creates a gap: these models are excellent at "denoising" small errors but struggle with "sketch-based generation," where the majority of the content is missing. The authors aim to bridge this gap by creating a general-purpose model that can reconstruct entire documents from a skeleton of key information, which is particularly useful for generating synthetic data to train other models.
Methods
The researchers developed GENIUS (GENeratIon Using Sketch as input) using two primary innovations:
- Extreme and Selective Masking: Unlike standard models that mask random tokens, GENIUS uses an unsupervised keyword extraction algorithm (YAKE) to identify the most informative 20% of a text. The remaining 80% is replaced by single mask tokens
[M]. This forces the model to learn how to bridge large gaps between semantically dense "islands" of text. - GeniusAug Pipeline: To use GENIUS for data augmentation, the authors propose a "target-aware" extraction method. It uses Sentence-BERT to identify n-grams in a training sample that are most relevant to the specific task (e.g., the class label in sentiment analysis). GENIUS then generates new, diverse contexts around these essential parts.
- Sketch Mixup: A more advanced augmentation technique where the model combines key elements from multiple different training samples into a single "mixed" sketch to generate entirely new, label-preserving examples.
The model architecture is based on a bidirectional encoder and an autoregressive decoder (initialized with BART weights).
Experimental Setup
The model was pre-trained on the C4-realnewslike corpus, consisting of roughly 27 million <sketch, text> pairs. Evaluation spanned several domains:
- Text Classification: BBC, HuffPost, Yahoo, 20NG, IMDB, and SST2.
- Named Entity Recognition (NER): CoNLL03.
- Machine Reading Comprehension (MRC): SQuAD.
Key Insights and Results
The "Goldilocks" Zone of Augmentation
Existing methods are often either too "conservative" (making tiny changes that don't help diversity, like EDA) or too "aggressive" (generating entirely new text that might lose the original meaning, like GPT-2). GeniusAug finds a middle ground by keeping the "semantic core" (the sketch) while completely reimagining the surrounding "syntax and context."
Performance Gains
GENIUS achieved significantly lower perplexity (18.09 vs. 56.04 for BART-large) and higher ROUGE scores when reconstructing text from sketches. Using GeniusAug improved classification accuracy across the board. In "In-Distribution" (ID) settings, it provided a ~2-3% boost. In "Out-of-Distribution" (OOD) settings—where the model is tested on a different dataset than it was trained on—the gains were even larger, around 5-7%.
The model is particularly effective when labeled data is extremely scarce (e.g., only 50 labeled examples). Furthermore, the paper demonstrates that masking the less informative parts of a sentence is more effective for pre-training than random masking, as it teaches the model the underlying "connective tissue" of language.
Limitations and Future Work
While powerful, GENIUS requires more GPU resources for augmentation compared to simple rule-based methods. For tasks requiring strict logical reasoning, sketch extraction might miss subtle logical connectors, leading to noisy synthetic samples. In NER tasks, the model might "hallucinate" new entities not present in the original sketch. Future work includes exploring multilingual versions and precise control over generated text length.
Summary updated