Evaluating Grammaticality in Seq2seq Models with a Broad Coverage HPSG Grammar: A Case Study on Machine Translation

Johnny Wei, Khiem Pham, Brendan O’Connor, Brian Dillon

TL;DR

This paper investigates whether neural machine translation (NMT) models, specifically seq2seq architectures, learn to generate syntactically correct English. By using the comprehensive English Resource Grammar (ERG), they found that over 93% of NMT outputs are parseable, suggesting a degree of grammatical learning. However, the study also highlights that NMT models struggle with the distribution of rarer syntactic rules, leading to differences compared to human-generated translations.

What Problem Does This Paper Address?

Sequence-to-sequence (seq2seq) models have become the workhorse for tasks involving natural language generation, such as machine translation, dialogue systems, and summarization. While these models excel at producing fluent-sounding text, a critical question remains: do they truly understand and adhere to the underlying grammatical structure of the target language? Current seq2seq models are primarily designed for sequential data and are not explicitly built with linguistic grammar in mind. This lack of explicit grammatical grounding makes it difficult to assess the quality and correctness of their generated output beyond surface-level fluency. This paper aims to bridge this gap by evaluating the syntactic properties of NMT output using a formal, broad-coverage grammar.

Methods

The core of this research lies in using the English Resource Grammar (ERG), a highly detailed Head-Phrase Structure Grammar (HPSG) of English, as an evaluation tool. The ERG is known for its linguistic precision and broad coverage of English phenomena. The authors trained a French-to-English seq2seq NMT model on a large parallel corpus. Crucially, they filtered the training and testing data to ensure that the human-generated reference translations were parseable by the ERG.

The evaluation process involved parsing the output of the NMT model using the ERG. This allowed them to determine if the generated sentences could be assigned a valid grammatical structure according to the ERG. Beyond simple parseability, they also analyzed the types of grammatical rules used in both the NMT output and the reference translations. To further investigate ungrammatical outputs, they conducted manual annotation of a subset of sentences that the ERG could not parse. They also employed a discriminative analysis using logistic regression to identify specific syntactic rules that differentiate between reference and NMT translations.

Datasets

The primary dataset used was the Europarl v7 parallel corpus, containing French-to-English sentence pairs. From this corpus, 1.6 million sentence pairs were selected where the English reference sentence was parseable by the ERG. This subset was then divided into training (1.4 million pairs), validation (5,000 pairs), and analysis (200,000 pairs) sets.

Key Insights and Ideas

  • Parseability as a Proxy for Grammaticality: The paper proposes that parseability by a broad-coverage, linguistically precise grammar like the ERG can serve as a strong indicator of grammatical correctness in NMT output.
  • NMT Models Learn Grammatical Structure: The high parseability rate (over 93%) suggests that seq2seq models, even without explicit grammatical constraints, do learn to generate output that conforms to a grammar.
  • Struggles with Rare Constructions: While overall parseability is high, NMT models exhibit difficulty in learning the correct distribution and usage of rarer syntactic rules, leading to deviations from human reference translations.
  • Literal Translation Tendency: Qualitative analysis suggests that the NMT model often prefers more literal translations, mirroring the French source sentence's structure and punctuation, sometimes at the expense of more natural English phrasing.
  • ERG as a Powerful Evaluation Tool: The ERG's detailed annotations provide rich insights into the specific linguistic constructions that NMT models either learn well or struggle with.

Key Results

  • High Parseability: Approximately 93.2% of the NMT translations were parseable by the ERG.
  • Unparseable Cases: Of the unparseable sentences, 3.2% were due to resource limitations (time/memory), 0.5% due to parser errors, and 3.1% due to the parser exhausting the search space.
  • Manual Annotation Findings: For sentences deemed unparseable by the ERG due to exhaustive search space, manual annotation revealed that 60% were considered ungrammatical by humans. Correcting agreement attachment errors could fix 18.3% of these ungrammatical sentences.
  • Rule Usage Analysis: Figure 2 shows that the usage of the top 10 most frequent rules in NMT translations closely matches that of reference translations. However, Figure 3 reveals that the ratio of rule usage between NMT and reference translations becomes more variable for rarer rules, indicating NMT's difficulty with these.
  • Discriminative Rules: A logistic regression model identified 47 rules that discriminate towards reference translations and 24 rules that discriminate towards NMT translations, highlighting specific areas of divergence.

Limitations and Future Work

  • Grammar Coverage Limitations: The authors acknowledge that the ERG, despite its breadth, has limitations in its coverage. They note that some sentences deemed ungrammatical by humans might not be parseable by the ERG, suggesting potential grammar gaps.
  • Semantic Analysis: The current study focuses solely on syntactic properties. Future work could explore the ERG's semantic representation (Minimal Recursion Semantics - MRS) to evaluate the semantic correctness of NMT output.
  • Human Ratings and MT Quality Estimation: The authors suggest using human ratings and machine translation quality estimation datasets to further understand which syntactic biases are preferable for NMT systems.
  • Evaluating Beyond Parseability: Exploring how to evaluate models despite the biased coverage of the grammar is a direction for future research.
  • Broader Applications: The paper hopes to inspire further research into HPSG-based or language-like grammar evaluations of neural networks across various NLP tasks.

Summary updated

BibTeX
@inproceedings{wei2018evaluating,
  author = {Johnny Wei and Khiem Pham and Brendan O’Connor and Brian Dillon},
  title = {{Evaluating Grammaticality in Seq2seq Models with a Broad Coverage HPSG Grammar: A Case Study on Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure