Neural sentence generation from formal semantics

Kana Manome, Masashi Yoshikawa, Hitomi Yanaka, Pascual Martínez-Gómez, Koji Mineshima, Daisuke Bekki

Introduction

While semantic parsing—the conversion of natural language into logical formulas—has matured significantly, the inverse task of generating natural language sentences from formal semantics remains a challenging frontier. Formal logic, particularly higher-order logic based on Neo-Davidsonian event semantics, offers a precise way to capture complex linguistic phenomena such as negation, quantification, and coordination. However, neural models often struggle to bridge the gap between these rigid, nested structures and the fluid nature of human language. This paper explores a neural sequence-to-sequence (seq2seq) approach to this problem, introducing techniques to maintain semantic fidelity and proposing a novel evaluation framework based on textual entailment.

Methodology

The core of the proposed system is a seq2seq model with an attention mechanism. To effectively process logical formulas, the authors investigated two linearization strategies: a token-based approach, treating formulas as simple sequences of operators and predicates, and a graph-based approach, which preserves variable bindings and scope through a tree-like structure.

Predicate Masking

A significant challenge in neural generation is "hallucination," where the model generates words not present in the input logic. To mitigate this, the authors introduced predicate masking. This technique uses a masking vector to constrain the decoder's output at each timestep. The mask ensures that the model only selects from words related to the predicates in the input formula (e.g., mapping walk(e) to "walk," "walked," or "walking") or from a predefined set of common function words.

The predicate masking mechanism constrains the output vocabulary based on input predicates and function words.

Data Preparation

The model was trained using the Stanford Natural Language Inference (SNLI) corpus. To create the necessary logic-sentence pairs, the researchers employed the ccg2lambda pipeline, which uses a Combinatory Categorial Grammar (CCG) parser to automatically generate ground-truth logical formulas from natural language text. The dataset consisted of 50,000 hypothesis sentences, split into training, development, and test sets.

Evaluation via Recognizing Textual Entailment (RTE)

Standard metrics like BLEU often fail to capture semantic accuracy, as they focus on surface-level word overlap. To address this, the authors proposed using Recognizing Textual Entailment (RTE) as a semantic evaluation metric. By using a semantic parser to check if the generated sentence ($S_2$) logically entails the original sentence ($S_1$) and vice versa ($S_1 \iff S_2$), they could determine if the core meaning was preserved, regardless of the specific phrasing used by the model.

Results and Discussion

The experimental results demonstrate that the inclusion of predicate masking is critical for high-quality generation. The masked token-based model achieved a BLEU score of 60.0, a substantial improvement over the 43.0 score of the unmasked version and the 38.3 score of a rule-based baseline.

Performance comparison across different models and linearization techniques using BLEU and RTE metrics.

In terms of semantic accuracy, the masked models achieved approximately 90% accuracy on the bidirectional entailment test ($S_1 \iff S_2$). This suggests that the neural generator successfully preserves the underlying logic in the vast majority of cases. While the token-based model performed slightly better on BLEU, the graph-based model showed competitive performance in semantic evaluation, highlighting the value of structural information.

Conclusion and Future Work

This research demonstrates that neural models, when properly constrained by predicate masking, can effectively generate natural language from complex formal semantics. However, some limitations remain. To simplify the reasoning process, the authors normalized formulas by removing tense and plurality, which occasionally led to minor discrepancies (e.g., generating "the men" instead of "a man"). Future work will focus on incorporating these richer semantic features back into the generation process and scaling the model to handle longer, multi-clause sentences.

Summary updated

Abstract

Sequence-to-sequence models have shown strong performance in a wide range of NLP tasks, yet their applications to sentence generation from logical representations are underdeveloped. In this paper, we present a sequence-to-sequence model for generating sentences from logical meaning representations based on event semantics. We use a semantic parsing system based on Combinatory Categorial Grammar (CCG) to obtain data annotated with logical formulas. We augment our sequence-to-sequence model with masking for predicates to constrain output sentences. We also propose a novel evaluation method for generation using Recognizing Textual Entailment (RTE). Combining parsing and generation, we test whether or not the output sentence entails the original text and vice versa. Experiments showed that our model outperformed a baseline with respect to both BLEU scores and accuracies in RTE.
BibTeX
@inproceedings{manome2018neural,
  author = {Kana Manome and Masashi Yoshikawa and Hitomi Yanaka and Pascual Martínez-Gómez and Koji Mineshima and Daisuke Bekki},
  title = {{Neural sentence generation from formal semantics}},
  booktitle = {Inlg},
  year = {2018},
  doi = {10.18653/v1/w18-6549},
}
Figure