Introduction
Open-ended Long Text Generation (Open-LTG) is a cornerstone of creative AI applications, from storytelling to long-form article writing. Historically, this field has been dominated by Autoregressive (AR) models like GPT and BART. While these models produce high-quality text by generating tokens sequentially, they suffer from a "linear time tax"—the generation speed decreases significantly as the sequence length grows.
Non-autoregressive (NAR) models offer a parallelized alternative for speed, but they often struggle with "model collapse" in open-ended tasks, frequently producing repetitive or nonsensical output. This research introduces a novel approach using Masked Language Models (MLMs) to achieve the speed of NAR models without sacrificing the creative quality of AR baselines.
Methodology: Iterative Refinement and Attention Control
The authors utilize a RoBERTa-base backbone and an iterative "mask-predict" strategy. Instead of a single left-to-right pass, the model begins with a sequence of mask tokens and refines them over 6–8 iterations. To prevent the typical collapse seen in NAR models, two key innovations were introduced:
- Dynamic Sliding Window Attention (DSWA): Global attention in MLMs can become overwhelmed by uninformative mask tokens. DSWA restricts a token's attention to its immediate neighbors. This window is dynamic: larger in lower layers for context and smaller in higher layers for local precision.
- Linear Temperature Decay (LTD): To balance exploration and convergence, the model uses a high sampling temperature in early iterations to encourage diversity, which then decays linearly to ensure fluency and confidence in the final output.
Key Insights: The Mask Noise Hypothesis
A central finding of this work is the "Mask Noise" hypothesis. The authors argue that MLM failure in long-form generation is primarily an attention problem rather than a lack of model capacity. When surrounded by hundreds of masks, the global attention mechanism fails to find meaningful signals. By enforcing local constraints through DSWA, the model is forced to rely on nearby informative anchors, significantly improving coherence.
Experimental Results
The model was evaluated against BART-base on several benchmarks, including ROCStories, WritingPrompts, and WikiPlots.
Speed and Efficiency
The MLM approach demonstrates massive efficiency gains that scale with text length. On an NVIDIA A100, the model achieved a 13.3x speedup on the WikiPlots dataset compared to BART. Even on shorter tasks, it maintained a nearly 3x speed advantage.
Quality and Fluency
Automated metrics (BLEU, ROUGE, and BERTScore) showed the MLM outperforming BART. Notably, the model achieved significantly lower perplexity (PPL), indicating superior fluency. Human evaluations mirrored these results, with annotators preferring the MLM output for relevance (47.5% win rate) and coherence.
Conclusion and Future Work
By repurposing MLMs with specialized attention and sampling strategies, this research demonstrates that parallelized text generation is a viable and highly efficient alternative to autoregressive modeling for long-form creative writing. Future work will focus on improving global logical ordering and reducing minor grammatical artifacts inherent in parallel prediction.
Summary updated