Abstract: The computer maxim garbage in, garbage out is especially true of language generation. When a generator slavishly follows its input topics, it usually produces bad text. In order to find more appropriate forms of expression, generators must be given the ability to interpret their input topics. Often, newly formed interpretations can help generators achieve their pragmatic goals with respect to the hearer. Since interpretation requires inference, generators must exercise some control over the inference process. Some general strategies of control, and some specific techniques geared toward achieving pragmatic goals, are described here.
Introduction
In the early development of natural language generation (NLG), systems often functioned as mere "transliterators." If provided with a list of raw facts, they would produce a corresponding list of sentences, often resulting in repetitive and unengaging prose. Eduard Hovy’s 1987 paper, "Interpretation in Generation," argues that this "garbage in, garbage out" approach is fundamentally flawed. To produce text that is useful, concise, and pragmatically effective, a generator must be able to interpret its input through active inference.
The Problem: The Limits of Literalism
Hovy identifies a fundamental gap in straightforward generation: it lacks the nuance of human communication. A human reporter does not simply list every vote tally in an election; they interpret the data to say "Kennedy narrowed the lead." Without the ability to perform inference on its own input, a generator cannot adapt to a listener's level of knowledge, cannot summarize complex events, and cannot achieve pragmatic goals like "slanting" a story to reflect a specific point of view.
The PAULINE System
To address these issues, Hovy developed PAULINE (Planning And Uttering Language In Natural Environments). Consisting of over 12,000 lines of LISP, PAULINE moves beyond fixed templates by utilizing two primary methods of interpretation:
- Bottom-Up Interpretation: The system uses its internal memory network to find higher-level concepts that "contain" the input facts. For example, if the input shows Candidate A has more votes than Candidate B, the system follows links in its memory to identify this as an instance of the concept "BEAT."
- Top-Down Interpretation: This is driven by the generator's pragmatic goals. If the goal is to be "unsympathetic" toward an actor, PAULINE looks for specific "configurations"—patterns of triplets (type, variable, pattern)—that allow it to characterize an action as a "confrontation" rather than a neutral event.
Pragmatic Constraints
PAULINE decides how much to interpret based on three critical factors:
- Hearer's Knowledge: The system avoids explaining what the audience already knows while ensuring it isn't so vague that the reader becomes lost.
- Slant: The system chooses interpretations that communicate a specific opinion or bias.
- Time: A "real-time" tradeoff where more inference leads to better text but requires more computation.
Experimental Domains
As a foundational 1987 study, the paper utilizes structured input from specialized expert systems rather than modern web-scale datasets:
- JUDGE: A program modeling the sentencing behavior of judges, providing raw representations of actions and resulting states in physical altercations.
- Hypothetical Primary Election: A representation of the 1980 Carter vs. Kennedy race consisting of approximately 80 conceptual elements.
- Yale Shantytown Protest: A representation of a 1986 anti-apartheid protest at Yale, involving approximately 120 representation elements.
Key Insights and Ideas
Generation as Inference
Hovy posits that generating language is an act of directed inference. A generator should not just "forget" the interpretations it makes; it should integrate them back into its memory, effectively "understanding" the topic better through the act of describing it.
The Economic Tradeoff
There is a literal cost to good prose. High-quality, interpreted text requires more time and effort. Hovy argues that a generator must decide if the pragmatic payoff—such as successfully convincing a listener—is worth the computational expense.
The "Middle Ground" of Detail
Following Grice’s Maxim of Quantity, Hovy notes that being too specific makes a generator unintelligible, while being too general makes it seem uncooperative. Interpretation is the tool used to find the "just right" level of detail for a given context.
Slanting via Concept Choice
One of the paper's most powerful insights is that slanting a text is achieved not just through adjectives, but through the choice of concept. Calling a protest a "confrontation" versus a "demonstration" changes the entire narrative before a single biased descriptor is even used.
Results and Qualitative Analysis
The paper provides qualitative comparisons of PAULINE’s output across different settings:
- Efficiency: By interpreting multiple input topics into a single high-level concept (like "FIGHT"), the generator produces text that is significantly more readable than the raw input.
- Neutral vs. Interpreted: In election examples, PAULINE moves from dry vote counts to concise summaries: "Kennedy beat Carter by 335 votes."
- Pragmatic Slanting: In the Yale protest example, PAULINE can generate an "anti-protester" version that interprets student actions as "taking over" a plaza and "forcing" university action, rather than simply "building a shantytown."
Conclusion and Limitations
While PAULINE was a landmark in NLG, Hovy acknowledged several limitations, most notably the "knowledge bottleneck." The system relied on pre-defined configurations in its memory network; without these links, the generator could not make the leap to higher-level concepts. Hovy suggested that future generators should be integrated with more robust, general-purpose inference engines. Despite these early limitations, the paper remains a seminal work in defining generation as a strategic, goal-oriented process of interpretation.
Summary updated