INSET: Sentence Infilling with INter-SEntential Transformer

Yichen Huang, Yizhe Zhang, Oussama Elachqar, Yu ChengACL 2020 · DOI

Introduction

Sentence infilling is the task of predicting a missing sentence within a paragraph, requiring a model to maintain both local syntactic fluency and global discourse coherence. Unlike standard token-level infilling, which focuses on local context, sentence infilling acts as a "semantic bridge" between preceding and following text. At ACL 2020, researchers introduced INSET (INter-SEntential Transformer), a framework that addresses this challenge by decoupling semantic planning from surface realization.

Methodology: The Decoupled Framework

The core innovation of INSET is its separation of the infilling task into three distinct modules, moving away from direct token-to-token generation which often results in generic or irrelevant outputs.

  1. Understanding Module (BERT): A pre-trained BERT encoder maps context sentences into a latent semantic space, producing fixed-length vectors.
  2. Planning Module (Sentence-level Transformer): This module operates on the sequence of sentence vectors. Given a gap (e.g., at position 4 in a 7-sentence sequence), it predicts the latent vector for the missing sentence.
  3. Generation Module (GPT-2): A GPT-2 decoder takes the predicted semantic vector and "unpacks" it into natural language tokens.
The INSET architecture decouples sentence encoding/decoding (left) from sentence-level semantic planning (right).

To ensure the latent space effectively captures meaning, the authors employed a denoising autoencoder strategy during training. By forcing the model to reconstruct corrupted sentences, it learns robust semantic representations rather than simple token memorization. Crucially, the model can also incorporate lexical constraints (keywords) to guide the planning phase toward more specific content.

Evaluation and Results

The model was evaluated on two datasets: TripAdvisor (hotel reviews) and Recipe (procedural instructions). These datasets test the model's ability to handle subjective sentiment and strict logical flow, respectively.

Performance comparison of INSET against baselines across TripAdvisor and Recipe datasets.

As shown in the results table, INSET significantly outperforms token-level baselines across multiple automatic metrics. On the TripAdvisor dataset, INSET improved the BLEU-4 score from 0.54% to 0.96% in the unconstrained setting. When provided with lexical keyword constraints, performance reached 6.57%. Human evaluations corroborated these findings, with judges preferring INSET's coherence over the baseline by a margin of 54% to 32%.

Conclusion

INSET demonstrates that effective sentence infilling requires high-level discourse planning. By predicting "what" to say in a latent semantic space before deciding "how" to say it, the model achieves superior coherence and relevance compared to traditional word-by-word generation. Future work may extend this framework to handle multi-sentence gaps and explore more advanced latent space structures.

Summary updated

BibTeX
@inproceedings{huang2020inset,
  author = {Yichen Huang and Yizhe Zhang and Oussama Elachqar and Yu Cheng},
  title = {{INSET: Sentence Infilling with INter-SEntential Transformer}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.226},
}
Figure