TL;DR
This paper tackles the challenge of creating diverse and coherent data inputs for data-to-text generation systems. They introduce a novel method using Integer Linear Programming (ILP) and category-specific bigram models of RDF properties to automatically extract relevant subgraphs from DBpedia, aiming to build a benchmark for NLG research.
What Problem Does This Paper Address?
Natural Language Generation (NLG) systems that convert structured data into human-readable text often struggle with a lack of standardized benchmarks for training and evaluation. This makes it difficult to compare different data-to-text generators. A key bottleneck is the creation of suitable input data – structured representations of information that are both relevant to the entity being described and linguistically coherent. Existing methods for selecting this data often produce limited variety or lack sufficient coherence. This paper aims to bridge this gap by automatically generating a large, diverse, and coherent repertoire of data units from RDF knowledge bases like DBpedia.
Methods
The core of the proposed method involves using Integer Linear Programming (ILP) to select subgraphs from DBpedia. The novelty lies in how they formulate the objective function for the ILP, which aims to maximize both relevance and coherence.
Here's a breakdown:
- Category-Specific Bigram Models: They learn two types of bigram models for RDF properties, tailored to specific entity categories (e.g., Astronauts, Universities).
- s-Model (Sibling Bigrams): Captures topic-based coherence by looking at properties that co-occur with the same subject.
- c-Model (Chained Bigrams): Captures discourse-based coherence by examining properties where the object of one triple is the subject of another.
- M-Model (Interpolated): Combines both s- and c-models with a weighting parameter to balance topic and discourse coherence.
- ILP Formulation:
- Representing Triples: Binary variables are introduced for each RDF triple to indicate if it's selected.
- Representing Bigrams: Binary variables are used for pairs of triples that form s- or c-bigrams.
- Objective Function: The ILP maximizes a score based on the probabilities learned from the bigram models, aiming to select triples that are likely to co-occur in a coherent manner within a given category.
- Constraints: The ILP includes constraints to ensure the selected triples form a tree structure (to facilitate text generation) and to restrict the size of the output subgraph.
This approach differs from prior work by leveraging category-specific knowledge and explicitly modeling both topic and discourse coherence through distinct bigram types within an ILP framework.
Datasets
- DBpedia: The primary source of structured data. The authors extract entity graphs from DBpedia for three diverse categories: Astronaut, University, and Monument.
- No new benchmark dataset is introduced. Instead, the method aims to create such a resource by generating varied data units from existing DBpedia data.
Key Insights and Ideas
- Category Matters: Tailoring coherence models (bigrams) to specific entity categories significantly improves the selection of relevant and coherent data. What's coherent for an astronaut might not be for a university.
- Topic vs. Discourse Coherence: Explicitly modeling both "sibling" (topic) and "chained" (discourse) relationships between properties provides a more nuanced approach to coherence than relying on a single metric.
- ILP as a Powerful Selector: Integer Linear Programming is a flexible and powerful tool for optimizing complex selection criteria, balancing multiple objectives (relevance, topic coherence, discourse coherence, size) simultaneously.
- Tree Structure is Key for Generation: Constraining the output to tree structures is a practical step towards making the selected data directly usable by text generation systems, as it maps well to syntactic structures.
Key Results
- Diversity: The different ILP models (s, c, M) produce solutions with low overlap, indicating good diversity. The average overlap between solutions from different models was low (Min: 0.18, Max: 0.24), suggesting that even with the same input, different models yield distinct content.
- Coherence:
- Quantitative Metrics: The baseline method (random selection) showed a higher number of irrelevant triples compared to the ILP-based methods. The M-Model performed best for depth 2.
- Human Evaluation: Annotators rated the coherence of data units produced by the ILP models significantly higher than the baseline. The s-Model showed the highest coherence, particularly for depth 1 graphs, while the c-Model and M-Model also achieved strong results.
- Qualitative Analysis: Example outputs show that the baseline often includes generic "source" properties and lacks cohesion. The s-Model tends to group topically related properties, while the c-Model focuses on related entities. The M-Model strikes a balance, including both related entities and topic-grouped facts.
Limitations and Future Work
- Limitations:
- The paper acknowledges that the s-Model, when selecting triples based solely on sibling bigrams, might fail to eliminate indirectly related triples that are irrelevant to the entity.
- For the c-Model, the number of irrelevant properties can increase with solution size, as more bigrams are selected, potentially leading to lower probability selections.
- The evaluation of the s-Model for depth 2 and the c-Model for depth 1 was disregarded in some overlap calculations due to their design focus.
- Future Work:
- The authors plan to use this content selection procedure to semi-automatically construct a large data-to-text resource for training and testing RDF verbalizers.
- They intend to explore methods for associating RDF subtrees with verbalizations, including automatic generation using symbolic generators, manual/semi-automatic validation, and human annotation via crowdsourcing.
- They suggest investigating how to integrate relatedness metrics from other works (like Cheng et al., 2015) into their ILP formulation.
- Extracting deeper graphs would require higher-order n-gram models, which could be limited by data sparsity.
Summary updated