TL;DR
This paper details NUIG-DSI's submission to the GEM benchmark for data-to-text generation. They leverage the pre-trained T5-base model, experimenting with various masking strategies during pre-training on monolingual data to improve performance on datasets like DART, WebNLG, E2E, and CommonGen. Their findings suggest random masking offers the best performance, though differences across strategies are not always statistically significant.
What Problem Does This Paper Address?
The field of Natural Language Generation (NLG) aims to automatically produce human-readable text from various data sources. While neural models have achieved state-of-the-art results, they sometimes struggle with data coverage, relevance, and correctness compared to traditional rule-based systems. The GEM benchmark was established to provide a standardized evaluation framework for diverse NLG tasks, including data-to-text generation. This paper addresses the challenge of effectively pre-training large language models for data-to-text generation tasks within the GEM benchmark, exploring how different pre-training strategies impact performance across various datasets with distinct characteristics.
Methods
The core of NUIG-DSI's approach is the use of the pre-trained T5-base model, a powerful encoder-decoder architecture. They employ a two-stage process:
- Pre-training on Monolingual Data: Before fine-tuning on specific data-to-text datasets, the T5 model is further pre-trained on large monolingual corpora. This pre-training utilizes a masked language modeling objective. The novelty lies in the exploration of different masking strategies:
- Random Masking: A standard approach where a percentage of tokens are randomly masked.
- Targeted Masking: Specifically masking entities, predicates, concepts, or meaning representation fields, depending on the dataset. This aims to inject domain-specific knowledge or structure into the pre-training process.
- Additional Tagging: For some experiments, they incorporate special tags (e.g.,
<SUB>,<PRED>,<OBJ>) to explicitly mark structural elements within the input triples, and also experiment with entity type tags from DBpedia or NER tags from spaCy. - Fine-tuning on Task-Specific Datasets: After pre-training, the model is fine-tuned on the specific data-to-text datasets (DART, WebNLG, E2E, CommonGen) to generate text from their respective input formats (triples, meaning representations).
Datasets
The submission focuses on four datasets within the GEM benchmark:
- DART (Nan et al., 2021): Open-domain data records structured as triples, paired with crowd-sourced textual annotations. It merges data from tables, WikiSQL, WebNLG, and E2E.
- WebNLG (Gardent et al., 2017): Generates text from RDF triples extracted from DBpedia.
- E2E (Novikova et al., 2017): Generates text from meaning representations in the restaurant domain.
- CommonGen (Lin et al., 2020): Generates coherent sentences describing everyday scenarios from a set of concepts, testing commonsense reasoning.
For pre-training, they utilize:
- DBpedia abstracts: Used for DART and WebNLG.
- Monolingual target-side references: Used for E2E and CommonGen.
Key Insights and Ideas
- Random Masking is Surprisingly Effective: Despite the intuition that targeted masking (entities, predicates, concepts) might be more beneficial for structured data-to-text generation, the experiments consistently show that random masking often performs best or comparably across automatic evaluation metrics. This suggests that the T5 model, with its inherent capabilities, can effectively learn from general linguistic patterns even without explicit structural guidance during pre-training.
- Pre-training Can Hurt Performance: For the CommonGen dataset, additional pre-training on monolingual data actually appeared to hurt performance, a finding that contradicts the general trend of pre-training benefits. This might indicate that the specific nature of commonsense reasoning in CommonGen requires a different pre-training approach or that the chosen monolingual data was not sufficiently aligned.
- Tagging Improves Performance (Slightly): Explicitly tagging input elements like subject, predicate, and object (
<SUB>,<PRED>,<OBJ>) for the DART dataset led to the best results compared to using no tags or more complex tags like DBpedia types or spaCy NER tags. This highlights the importance of providing clear structural cues to the model. - Dataset Specificity Matters: The performance gains and losses vary significantly across datasets, underscoring the need for dataset-specific tuning and potentially different pre-training strategies for different NLG tasks.
Key Results
- DART: Random masking achieved the best scores for BLEU, METEOR, and ROUGE-L on the validation set compared to entity, predicate, or combined masking, though differences were not statistically significant. Adding
<SUB>,<PRED>,<OBJ>tags to the T5-base model resulted in the best performance for their final submission. - WebNLG: The same model and approach used for DART were applied, leveraging pre-training on DBpedia abstracts and fine-tuning on DART, which yielded good results.
- E2E: Random masking on monolingual target references for pre-training outperformed specific masking of meaning representation fields, though again, differences were not statistically significant.
- CommonGen: Random masking and concept masking during pre-training showed mixed results, with random masking performing better on some metrics but concept masking on others. Notably, additional pre-training seemed to hurt performance overall.
- Overall GEM Metrics: NUIG-DSI observed higher scores on CommonGen and DART compared to baselines but lower scores on cleaned E2E and WebNLG (en), particularly on test and challenge subsets.
Limitations and Future Work
- Statistical Significance: A recurring theme is that while random masking often performed best, the differences between masking strategies were frequently not statistically significant. This suggests that further research with more extensive experiments or different evaluation metrics might be needed to definitively establish superiority.
- CommonGen Pre-training: The negative impact of pre-training on CommonGen performance is a significant limitation and an area for future investigation. Understanding why this occurs could lead to more effective pre-training strategies for commonsense reasoning tasks.
- Dataset Coverage: The authors note that at the time of writing, they did not have access to all references for DART and CommonGen test sets, limiting the scope of their reported results.
- Future Work: The paper implicitly suggests exploring more sophisticated pre-training objectives or data augmentation techniques tailored to specific NLG tasks. Investigating the interplay between different tagging schemes and masking strategies could also yield further improvements.
Summary updated