Sentence compression is the task of shortening a sentence while retaining its core meaning and ensuring it remains grammatically correct. At first glance, this seems like a simple problem of subtraction: just identify the "unimportant" words and remove them. However, research over the last two decades has revealed a fundamental tension between the simplicity of word deletion and the complexity of human communication.
The Deletion Constraint
The dominant paradigm in early sentence compression was extractive, meaning the model is restricted to deleting existing tokens. While computationally efficient, this creates what researchers call a "deletion constraint." As James Clarke and Mirella Lapata argue in Models for Sentence Compression: A Comparison across Domains, Training Requirements and Evaluation Measures, there is a fundamental mismatch between these models and how humans actually summarize.
When a human compresses a sentence, they don't just cross out words; they paraphrase, reorder clauses, and substitute long phrases with concise synonyms. An extractive model cannot perform these essential linguistic transformations. For example, a human might change "at this point in time" to "now," but a deletion-only model is forced to either keep the whole phrase or delete parts of it, often resulting in "at time" or "in time," which may lose the original nuance.
Maintaining Syntactic Integrity
If you start pulling bricks out of a wall at random, the wall eventually collapses. Sentences are the same. To prevent "word salad," researchers have integrated formal syntax into compression models. Jenine Turner and Eugene Charniak demonstrated in Supervised and Unsupervised Learning for Sentence Compression that using syntactic models allows for high-quality summaries even when training data is scarce.
The goal is to ensure that the remaining words still form a valid dependency tree. Jing Jiang and colleagues explored this in Can Syntax Help? Improving an LSTM-based Sentence Compression Model for New Domains, where they used Integer Linear Programming (ILP) to enforce grammatical rules on top of neural network outputs. By combining the "soft" learning of LSTMs with the "hard" constraints of dependency parsing, they created models that generalize much better across different types of text.
The Gap Between Models and Gold Standards
One of the most striking findings in this field is that our "gold standard" datasets—the human-written examples we use to train models—often contain moves that extractive models are physically incapable of making. Naman Gupta, Carolyn Rose, and Sourish Chaudhuri pointed out in Evaluating the Syntactic Transformations in Gold Standard Corpora for Statistical Sentence Compression that humans frequently perform complex transformations, such as replacing a pronoun with its antecedent noun to maintain clarity.
Because deletion-based models cannot "generate" or "restore" tokens, they can never truly replicate human-level compression. This has led to a push toward abstractive compression. As noted by Shun Hasegawa and Hiroya Takamura in Japanese Sentence Compression with a Large Training Dataset, the transition from extractive to abstractive is the "significant challenge" of the field, requiring models to synthesize entirely new phrasing while preserving the original meaning.
Modern Efficiency and Prompt Compression
The principles of sentence compression have found new life in the era of Large Language Models (LLMs). Instead of compressing for human readability, researchers are now compressing to save money and compute. Hoyoun Jung and Kyung-Joong Kim introduced Discrete Prompt Compression with Reinforcement Learning, which uses a policy network to strip unnecessary tokens from prompts.
Even in this modern context, the extractive vs. abstractive trade-off remains. While their reinforcement learning approach successfully reduces prompt length by 25%, it is still limited by the "removal-only" constraint. The history of this field suggests that the next leap in efficiency will come not from better deletion, but from models that can rewrite the prompt entirely—achieving the same intent with a fraction of the original vocabulary.