Improving Logical-Level Natural Language Generation with Topic-Conditioned Data Augmentation and Logical Form Generation

Ao Liu, Congjian Luo, Naoaki OkazakiJ. Inf. Process 2021 · OpenReview

TL;DR

This paper tackles the challenge of generating natural language descriptions that are logically entailed by structured data, a task hindered by the scarcity of high-quality, logically annotated data. The authors introduce a novel approach combining topic-conditioned data augmentation to generate more training examples and a dual-task learning framework for logical form generation and text generation, significantly improving performance, especially in low-resource settings.

What Problem Does This Paper Address?

Generating natural language descriptions from structured data (like tables) is a well-studied problem. However, ensuring these descriptions are not just fluent but also logically sound and directly derivable from the data (logical-level NLG) is difficult. Traditional methods often struggle with "low fidelity" – generating statements that aren't strictly supported by the data or miss crucial logical connections. This is largely due to the labor-intensive process of creating datasets with both structured data, logical forms (which represent the underlying reasoning), and corresponding natural language text. The existing "Logic2text" dataset, while a step forward, is relatively small, limiting the potential of even powerful pre-trained models. This paper aims to overcome this data scarcity and improve logical-level NLG by generating more training data and jointly training models for both text generation and logical form generation.

Methods

The core of the proposed solution lies in three key stages:

  1. Topic-Conditioned Data Augmentation (TopicDA): The authors leverage pre-trained language models (GPT-2) to generate synthetic training data. Unlike previous methods that might augment existing text, TopicDA focuses on generating unpaired logical forms (LFs) and natural language texts directly from tables. The novelty here is the use of "logic types" (e.g., superlative, comparative) as "topics" to guide the generation process. This ensures that the augmented data covers a diverse range of logical operations and expressions, even from tables that might have limited examples in the original dataset.
  2. Round-trip Data Weighting: Since augmented data can be noisy, a weighting strategy is employed. This method uses a "round-trip" process: a logical form is translated to text, and then that text is translated back to a logical form. The similarity between the original and reconstructed logical form (measured by BERTScore) is used to assign a weight to the augmented data point. This allows the model to prioritize higher-quality augmented examples during training.
  3. Joint Training of Logic2text and LG: The paper introduces "Logical Form Generation" (LG) as a dual task to Logic2text. LG aims to generate a valid logical form given a table and a text description. By jointly training Logic2text and LG models, the authors enable a form of "back-translation" where each model can generate pseudo-parallel data for the other. This is further enhanced by a "self-training" mechanism and a "curriculum learning" approach, where data is processed in order of its assigned weight, starting with cleaner examples.

Datasets

The primary dataset used is the Logic2text dataset (Chen et al., 2020b), which contains tables, logical forms, and natural language descriptions. The authors also leverage the LogicNLG dataset (Chen et al., 2020a) to demonstrate how their Logical Form Generation task can aid in annotating new logical-level NLG benchmarks. No new dataset is explicitly introduced, but the augmentation process effectively expands the training data derived from existing sources.

Key Insights and Ideas

  • Logic Types as Topics: Using predefined logic types (like superlative, comparative) as "topics" for data augmentation is a clever way to steer generation and ensure diversity, even with limited original examples. This moves beyond simple text augmentation to structured, semantically diverse data generation.
  • Round-trip BERTScore for Quality Control: The round-trip translation and BERTScore weighting is an elegant solution to the noise inherent in generated data. It provides a nuanced way to assess data quality beyond simple string matching, focusing on semantic consistency.
  • Duality of Logic2text and LG: Framing Logical Form Generation as the inverse task of Logic2text unlocks powerful semi-supervised learning techniques like back-translation. This joint training allows the models to learn from each other, creating a virtuous cycle of improvement.
  • Effectiveness in Low-Resource Settings: The paper strongly emphasizes that their approach yields significant gains when only a small subset of the original data is available (few-shot setting), highlighting its practical utility in scenarios where data annotation is a bottleneck.

Key Results

  • Improved Logic2text Performance: The proposed full model (with TopicDA and joint training) outperforms supervised baselines by a substantial margin, achieving higher BLEU and ROUGE scores on the Logic2text task. For example, in the full data setting, their model achieved 32.68 BLEU-4 and 65.74 ROUGE-1, compared to GPT-2 (full) at 31.69 and 64.88 respectively.
  • Strong Few-Shot Performance: In the few-shot setting (1k training samples), their method significantly outperforms the GPT-2 baseline, demonstrating the effectiveness of data augmentation and joint training in data-scarce environments. Ours (1k) achieved 26.34 BLEU-4 and 59.79 ROUGE-1, while GPT-2 (1k) achieved 23.38 and 56.65.
  • Effective LG Performance: The LG model also shows strong performance, with the full model achieving 67.95 LF Acc. and 89.01 Exec. Acc.
  • Data Augmentation Quality: Analysis shows that augmented data is generally topically consistent (98.52% for LFs, 94.30% for texts) and that the round-trip weighting correctly assigns higher scores to factually correct augmented data.

Limitations and Future Work

  • Factual Correctness of Augmented Data: While the TopicDA method is effective, the paper acknowledges that augmented data can still be noisy and factually incorrect (e.g., Figure 6 shows an incorrect date). The round-trip weighting helps mitigate this, but it's not a perfect filter.
  • LG Evaluation Metric: The paper notes that Execution Accuracy for LG might not perfectly reflect semantic correctness, suggesting a need for better evaluation metrics for this task.
  • Future Work: The authors suggest applying their method to annotate new logical-level NLG benchmarks, indicating a path towards creating more comprehensive datasets for this challenging task. They also mention the potential for adapting the TopicDA method to other domains and topics.

Summary updated

Abstract

Logical Natural Language Generation, i.e., generating textual descriptions that can be logically entailed by a structured table, has been a challenge due to the low fidelity of the generation. \citet{chen2020logic2text} have addressed this problem by annotating interim logical programs to control the generation contents and semantics, and presented the task of table-aware logical form to text (Logic2text) generation. However, although table instances are abundant in the real world, logical forms paired with textual descriptions require costly human annotation work, which limits the performance of neural models. To mitigate this, we propose topic-conditioned data augmentation (TopicDA), which utilizes GPT-2 to generate unpaired logical forms and textual descriptions directly from tables. We further introduce logical form generation (LG), a dual task of Logic2text that requires generating a valid logical form based on a text description of a table. We also propose a semi-supervised learning approach to jointly train a Logic2text and an LG model with both labeled and augmented data. The two models benefit from each other by providing extra supervision signals through back-translation. Experimental results on the Logic2text dataset and the LG task demonstrate that our approach can effectively utilize the augmented data and outperform supervised baselines by a substantial margin.
BibTeX
@article{liu2021improving,
  author = {Ao Liu and Congjian Luo and Naoaki Okazaki},
  title = {{Improving Logical-Level Natural Language Generation with Topic-Conditioned Data Augmentation and Logical Form Generation}},
  journal = {J. Inf. Process.},
  year = {2021},
  url = {https://openreview.net/forum?id=Mi3woPcl3P9},
}
Figure