Introduction
Standard sequence-to-sequence (seq2seq) models, such as the Transformer, have become the de facto standard for sequence modeling tasks like machine translation. However, these models often suffer from a "compositionality" problem. Because they model sequences autoregressively word-by-word and rely heavily on surface-level correlations, they frequently fail on benchmarks designed to test compositional generalization.
For instance, if a model learns the concepts "jump" and "run twice" during training, it should intuitively understand how to process "jump twice," even if it has never encountered that specific combination. Standard neural models often fail this test because they lack an explicit understanding of the hierarchical structure of language. To address this, this paper introduces Latent Neural Grammars, an alternative hierarchical approach that bridges the gap between classical, structure-heavy grammar models and modern, flexible neural networks.
Methodology: Latent Neural Grammars
The authors propose a neural parameterization of Quasi-Synchronous Context-Free Grammars (QCFG). This approach treats both source and target sentences as having underlying tree structures. By inducing these structures as latent variables during training, the model gains a strong inductive bias for hierarchical composition.
Key technical components include:
- Latent Tree Induction: Unlike previous grammar-based methods that require a pre-existing parser to provide "gold" trees, this model treats the tree structures of both the source and target sequences as fully latent. It learns to "discover" the most useful hierarchical structure for the task during training.
- Neural Rule Probabilities: Instead of relying on a massive table of fixed, manually engineered rules, the model utilizes neural embeddings. It represents each target non-terminal and source tree node as a vector (using a TreeLSTM for the source nodes). Rule probabilities are calculated via feedforward networks acting on these embeddings, allowing the model to share parameters across combinatorial structures.
- The "Inside" Algorithm: To train the model while marginalizing out the unknown latent trees, the authors employ a vectorized version of the "inside algorithm."
- Phrase-level Copying: A unique extension of this grammar-based approach is the ability to copy entire phrases from the source to the target in a single step. This operation is much more natural in a tree-based model than in a standard word-based autoregressive model.
By forcing the model to generate the target through a tree-transduction process—where target spans are explicitly aligned to source spans—the model is nudged toward learning the underlying logic of the task rather than just memorizing word sequences. This also provides interpretability by design: the model produces a "provenance" map, allowing users to see exactly which part of the input command caused a specific part of the output sequence.
Experiments and Key Insights
The model was evaluated across three distinct domains to test its generalization and flexibility:
- SCAN: A diagnostic dataset for compositional generalization (e.g., translating "walk twice after jump" into a sequence of actions). It includes difficult splits like the jump split, where the word "jump" is only seen in isolation during training.
- StylePTB: A dataset for text style transfer, such as converting active voice to passive voice.
- Small-scale Machine Translation: An English-French translation task using the "daxy" split to test how well the model handles novel words embedded in known syntactic structures.
Key Results
Compositional Generalization (SCAN): The Neural QCFG achieved near-perfect accuracy (ranging from 95.7% to 98.7%) on the most difficult SCAN splits (Jump, Around Right, Length). In stark contrast, standard RNNs and Transformers often drop to near 0% or 50% on these exact same splits, highlighting the power of the structural inductive bias.
Style Transfer: On tasks like "Active to Passive" and "Emphasis," the Neural QCFG (specifically when utilizing phrase copying) outperformed several strong baselines, including a finetuned GPT-2 model, in terms of BLEU and ROUGE scores. The phrase copying mechanism mimics human style transfer by keeping core phrases intact while altering the surrounding syntactic "wrapper."
Machine Translation: The results here revealed a trade-off. While the model handled the compositional "daxy" test perfectly (100% accuracy), it was outperformed by a well-tuned Transformer on general translation quality (BLEU). This suggests that while a strict grammar bias is excellent for logical compositionality, it may be too restrictive for the inherent "messiness" and flexibility of real-world natural language translation.
Limitations and Future Work
While Latent Neural Grammars offer massive improvements in compositional generalization, they come with significant computational costs. The primary bottleneck is the $O(S^3 T^3)$ complexity of the inside algorithm (where $S$ and $T$ are the lengths of the source and target sequences). This makes the model slow and memory-intensive for long sequences. Additionally, the model is more sensitive to hyperparameters and random seeds than standard seq2seq models.
Future work may explore "soft" versions of the grammar that scale linearly rather than cubically, potentially allowing these powerful structural biases to be applied to longer texts or even other modalities like images and audio for grounded language learning.
Summary updated