Natural language generation (NLG) is often treated as a monolithic task: a model takes an input and produces a string of text. However, research spanning decades suggests that generation is more effective when treated as a modular pipeline. By separating what to say (semantic content selection) from how to say it (linguistic realization), researchers have built systems that are more controllable, easier to adapt across languages, and less prone to the "hallucinations" common in modern neural networks.
The Strategic and Tactical Divide
The core of this research direction is the "separation of concerns." As early as 1977, David D. McDonald argued in Language Generation: The Linguistics Component that a primary system should maintain task-specific logic while a dedicated component manages grammatical complexity. This distinction is often framed as a split between a strategic phase (determining content) and a tactical phase (executing grammar).
In modern contexts, this modularity prevents models from becoming "black boxes." For example, Pandey et al. proposed the Mask & Focus framework, which identifies vital "concepts" as a skeletal blueprint before generating a response. By first establishing this semantic skeleton, the model avoids the "boring chatbot" problem—the tendency to output safe but generic filler phrases—because the final output is grounded in a pre-determined semantic plan.
Content Selection as a Steering Mechanism
One of the primary failure modes of end-to-end neural models is their inability to stay organized. They may be fluent but often include irrelevant information or omit key facts. To solve this, researchers use specialized "selectors" to guide the generation process.
Gehrmann et al. demonstrated this with a "bottom-up" summarization approach. Instead of letting a model decide what is important while it writes, they use a data-efficient content selector to mask the attention of the generator. This forces the model to focus only on relevant phrases, improving accuracy and domain transfer. Similarly, the authors of Step-by-Step: Separating Planning from Realization in Neural Data-to-Text Generation use a symbolic planner to decide the structure of information first, leaving the neural model to handle only the "realization" or the actual writing.
Decoupling Lexicon from Syntax
A deeper level of modularity involves separating the words we choose (the lexicon) from the structures we put them in (the syntax). Tristan Thrush explored this in Compositional Neural Machine Translation by Removing the Lexicon from Syntax, creating an architecture that mimics the human brain's separation of word meanings and sentence structure.
This decoupling is particularly vital for languages with complex morphology or flexible word order. Tsarfaty and Sima'an proposed a Relational-Realizational model that decomposes phrase generation into three stages:
- Projecting abstract grammatical relations.
- Configuring their linear order.
- Realizing specific morphosyntactic forms.
By isolating these steps, the model can identify a word's role (like "Subject") independently of where it appears in the sentence or what suffix it carries.
Benefits for Multilingualism and Few-Shot Learning
Decoupling makes systems remarkably portable. If the "what to say" part is language-neutral, you can swap out the "how to say it" part for different languages. Habash and Dorr showed that a single "Thematic Hierarchy" of semantic roles could generate correct sentences in both English and Spanish from the same deep-semantic representation. Ballesteros et al. furthered this by using language-neutral semantic representations to generate summaries in four different languages using a shared pipeline.
This modularity also aids in data-scarce scenarios. Chen et al. found that by using a pre-trained model for general linguistic fluency and a specialized "switch policy" for content selection, they could achieve high-quality generation with as few as 50 examples. The model doesn't have to learn "how to speak" and "what to say" simultaneously; it leverages its pre-existing knowledge of grammar to focus entirely on the new task of mapping data to text.
Summary of the Modular Pipeline
Across these 85 papers, the consensus is that breaking the generation process into specialized sub-tasks—such as lexical mapping, structural aggregation, and linguistic refinement (Zhou)—leads to more robust AI. Whether it is generating poetry with strict rhythmic constraints (Hopkins and Kiela) or translating formal code specifications into English (William R. Swartout), the separation of semantic intent from linguistic form remains the most effective way to ensure that machines say exactly what they mean.
Key Foundational Work
Language Generation: The Linguistics Component (1977) — contributed 2 concept elements to this direction.