Introduction
Neural Machine Translation (NMT) systems, primarily based on sequence-to-sequence (seq2seq) architectures, have achieved remarkable fluency. However, a persistent question remains: do these models internalize the underlying structural rules of language, or are they merely sophisticated pattern matchers? This study investigates the grammaticality of NMT outputs by leveraging the English Resource Grammar (ERG), a broad-coverage, hand-coded formal grammar based on Head-driven Phrase Structure Grammar (HPSG).
Methodology
The researchers employed a "closed-loop" evaluation strategy to quantify the syntactic soundness of a French-to-English NMT model.
The English Resource Grammar (ERG)
The ERG serves as a linguistically precise judge. If a sentence adheres to formal English rules, the ERG can generate a derivation tree; otherwise, it rejects the input. To ensure a controlled experiment, the training data (Europarl v7) was filtered to include only the 1.6 million sentence pairs where the English reference was successfully parseable by the ERG.
Evaluation Framework
After training a standard NMT model on this "clean" dataset, the authors parsed the resulting translations. By comparing the machine-generated rule trees against human references, they could identify specific syntactic deviations and "fingerprints" of machine output.
Key Results and Insights
High Parseability vs. Structural Fragility
The NMT model demonstrated a functional grasp of grammar, producing parseable English 93.2% of the time. However, the remaining ~7% revealed significant failures. Manual audits showed that while some "unparseable" outputs were due to grammar gaps in the ERG, 60% were genuinely ungrammatical. Furthermore, a strong negative correlation (-0.320) was found between sentence length and parseability, indicating that syntactic coherence degrades as complexity increases.
The "Long Tail" of Syntax
While the model effectively mimicked common subject-verb-object structures, its performance plummeted on rarer grammatical constructions. This "long tail" phenomenon suggests that NMT models struggle to generalize beyond high-frequency patterns. As shown in the statistical analysis of rule distributions, the model's usage of rare rules deviates significantly from human references.
Syntactic Literalism and Fragments
The analysis highlighted a tendency toward "syntactic literalism," where the model adopts structures from the source language (French) that are awkward or incorrect in the target language (English). For instance, the model frequently produced "fragments"—noun or prepositional phrases that fail to form a complete sentence—and struggled with English-specific requirements like expletive pronouns (e.g., the "it" in "it is raining").
Conclusion
This research demonstrates that while seq2seq models are adept at inducing functional grammar from data, they lack the robust structural understanding required for rare or complex constructions. The study suggests that current NMT evaluation should move beyond simple overlap metrics like BLEU to incorporate deeper syntactic analysis.
Summary updated