TL;DR
This paper proposes a new shared task for multilingual surface realization, aiming to generate grammatically correct and natural-sounding sentences from Universal Dependencies (UD) tree structures. By leveraging recent advancements in UD treebanks and deep learning, this task seeks to push the boundaries of multilingual Natural Language Generation (NLG).
What Problem Does This Paper Address?
Natural Language Generation (NLG) systems are crucial for converting structured data or meaning representations into human-readable text. While there's been significant progress in NLG, much of it has focused on English. Multilingual generation, especially from complex linguistic structures, remains a significant challenge. Existing shared tasks in NLG often rely on specific meaning representations like AMRs or RDF triples, which are generated by separate analysis systems. This paper addresses the need for a multilingual NLG task that uses a more standardized linguistic representation – Universal Dependencies (UD) – and aims to bridge the gap between linguistic structure and natural language generation across multiple languages.
Methods
The proposed shared task features two tracks:
- Shallow Track: This track takes UD tree structures where word order information has been removed and words are lemmatized. The task is to determine the correct word order and inflect the words to form a coherent sentence.
- Deep Track: This track further simplifies the input by removing functional words (like auxiliaries, prepositions, conjunctions) and some morphological information. Participants must not only determine word order and inflection but also reintroduce these missing elements, making the task more challenging and closer to real-world NLG inputs.
The input data will be provided in CoNLL-U and Human-Friendly Graph (HFG) formats. The paper also details specific preprocessing steps for the UD structures to create the Shallow and Deep track inputs, including scrambling word order, lemmatizing words, removing functional words, generalizing edge labels, and removing certain morphological information.
Datasets
The task will utilize a subset of existing Universal Dependencies (UD) treebanks, focusing on approximately 10 languages with high-quality annotations. A subset of at least 4 treebanks will be used for the Deep Track. The selection criteria include organizer expertise, availability of native speakers for evaluation, treebank size, ease of format conversion, and the variety of linguistic features captured. The paper references the CoNLL-U format and the Human-Friendly Graph (HFG) format used in a previous Surface Realization Shared Task (SR'11).
Key Insights and Ideas
- Leveraging Universal Dependencies: The availability of multilingual UD treebanks is a significant enabler for a large-scale multilingual NLG task. UD provides a consistent linguistic framework across many languages, simplifying cross-lingual system development.
- Deep Learning's Potential: The paper highlights the transformative impact of deep learning on NLP and suggests it can be a "game changer" for NLG, opening up new avenues for system design and performance improvements.
- Bridging the Gap to Real-World NLG: The Deep Track aims to create inputs that are more representative of real-world NLG scenarios, where syntactic and language-specific information might be less explicit, pushing systems to infer more from the given structure.
- Standardized Evaluation: The proposal emphasizes both automatic (BLEU, NIST, METEOR) and manual evaluation (clarity, fluency, meaning similarity) to ensure robust assessment of system performance.
Key Results
Not discussed in the paper. This paper is a proposal for a future shared task, not a report on completed experiments or results.
Limitations and Future Work
- Limited Scope of Deep Track Input: While the Deep Track aims to be closer to real-life inputs, it still retains some linguistic information (e.g., semantic information like nominal number and verbal tense). Future work could explore even more abstract or underspecified inputs.
- Feasibility of Connecting UD to Predicate Arguments: A secondary objective is to assess the feasibility of connecting UD representations to predicate argument structures commonly used in deep NLG systems. The success of this connection will be a key outcome.
- Workshop Collocation: The organizers plan to hold a workshop collocated with major NLP conferences (ACL '18, COLING '18, or EMNLP '18) to present the results of the shared task.
- Funding and Sponsorship: The organizers will contribute research funds, and Google sponsorship will be solicited to ensure a smooth setup and evaluation.
Summary updated