Introduction
Sentence infilling is the task of predicting a missing sentence within a paragraph, requiring a model to maintain both local syntactic fluency and global discourse coherence. Unlike standard token-level infilling, which focuses on local context, sentence infilling acts as a "semantic bridge" between preceding and following text. At ACL 2020, researchers introduced INSET (INter-SEntential Transformer), a framework that addresses this challenge by decoupling semantic planning from surface realization.
Methodology: The Decoupled Framework
The core innovation of INSET is its separation of the infilling task into three distinct modules, moving away from direct token-to-token generation which often results in generic or irrelevant outputs.
- Understanding Module (BERT): A pre-trained BERT encoder maps context sentences into a latent semantic space, producing fixed-length vectors.
- Planning Module (Sentence-level Transformer): This module operates on the sequence of sentence vectors. Given a gap (e.g., at position 4 in a 7-sentence sequence), it predicts the latent vector for the missing sentence.
- Generation Module (GPT-2): A GPT-2 decoder takes the predicted semantic vector and "unpacks" it into natural language tokens.
To ensure the latent space effectively captures meaning, the authors employed a denoising autoencoder strategy during training. By forcing the model to reconstruct corrupted sentences, it learns robust semantic representations rather than simple token memorization. Crucially, the model can also incorporate lexical constraints (keywords) to guide the planning phase toward more specific content.
Evaluation and Results
The model was evaluated on two datasets: TripAdvisor (hotel reviews) and Recipe (procedural instructions). These datasets test the model's ability to handle subjective sentiment and strict logical flow, respectively.
As shown in the results table, INSET significantly outperforms token-level baselines across multiple automatic metrics. On the TripAdvisor dataset, INSET improved the BLEU-4 score from 0.54% to 0.96% in the unconstrained setting. When provided with lexical keyword constraints, performance reached 6.57%. Human evaluations corroborated these findings, with judges preferring INSET's coherence over the baseline by a margin of 54% to 32%.
Conclusion
INSET demonstrates that effective sentence infilling requires high-level discourse planning. By predicting "what" to say in a latent semantic space before deciding "how" to say it, the model achieves superior coherence and relevance compared to traditional word-by-word generation. Future work may extend this framework to handle multi-sentence gaps and explore more advanced latent space structures.
Summary updated