Evaluating Syntactic Properties of Seq2seq Output with a Broad Coverage HPSG: A Case Study on Machine Translation

Johnny Tian-Zheng Wei, Khiem Pham, Brian Dillon, Brendan O'Connor

Introduction

Neural Machine Translation (NMT) systems, primarily based on sequence-to-sequence (seq2seq) architectures, have achieved remarkable fluency. However, a persistent question remains: do these models internalize the underlying structural rules of language, or are they merely sophisticated pattern matchers? This study investigates the grammaticality of NMT outputs by leveraging the English Resource Grammar (ERG), a broad-coverage, hand-coded formal grammar based on Head-driven Phrase Structure Grammar (HPSG).

Methodology

The researchers employed a "closed-loop" evaluation strategy to quantify the syntactic soundness of a French-to-English NMT model.

The English Resource Grammar (ERG)

The ERG serves as a linguistically precise judge. If a sentence adheres to formal English rules, the ERG can generate a derivation tree; otherwise, it rejects the input. To ensure a controlled experiment, the training data (Europarl v7) was filtered to include only the 1.6 million sentence pairs where the English reference was successfully parseable by the ERG.

Evaluation Framework

After training a standard NMT model on this "clean" dataset, the authors parsed the resulting translations. By comparing the machine-generated rule trees against human references, they could identify specific syntactic deviations and "fingerprints" of machine output.

Comparison of HPSG parse trees for a reference translation and an NMT output fragment.

Key Results and Insights

High Parseability vs. Structural Fragility

The NMT model demonstrated a functional grasp of grammar, producing parseable English 93.2% of the time. However, the remaining ~7% revealed significant failures. Manual audits showed that while some "unparseable" outputs were due to grammar gaps in the ERG, 60% were genuinely ungrammatical. Furthermore, a strong negative correlation (-0.320) was found between sentence length and parseability, indicating that syntactic coherence degrades as complexity increases.

The "Long Tail" of Syntax

While the model effectively mimicked common subject-verb-object structures, its performance plummeted on rarer grammatical constructions. This "long tail" phenomenon suggests that NMT models struggle to generalize beyond high-frequency patterns. As shown in the statistical analysis of rule distributions, the model's usage of rare rules deviates significantly from human references.

Ratio of rule counts (NMT vs. Reference) plotted against rule rank, illustrating the 'long tail' of syntax.

Syntactic Literalism and Fragments

The analysis highlighted a tendency toward "syntactic literalism," where the model adopts structures from the source language (French) that are awkward or incorrect in the target language (English). For instance, the model frequently produced "fragments"—noun or prepositional phrases that fail to form a complete sentence—and struggled with English-specific requirements like expletive pronouns (e.g., the "it" in "it is raining").

Conclusion

This research demonstrates that while seq2seq models are adept at inducing functional grammar from data, they lack the robust structural understanding required for rare or complex constructions. The study suggests that current NMT evaluation should move beyond simple overlap metrics like BLEU to incorporate deeper syntactic analysis.

Summary updated

BibTeX
@inproceedings{wei2018evaluating,
  author = {Johnny Tian-Zheng Wei and Khiem Pham and Brian Dillon and Brendan O'Connor},
  title = {{Evaluating Syntactic Properties of Seq2seq Output with a Broad Coverage HPSG: A Case Study on Machine Translation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure