Natural language is fundamentally hierarchical, yet most neural networks process it as a flat sequence of tokens. The research direction of unsupervised grammar induction explores how models can discover these hidden tree structures—such as phrases, dependencies, and discourse relations—without any human-annotated treebanks. By treating syntax as a latent variable or an emergent property of language modeling, these approaches allow machines to "parse" language using only raw text as a guide.
Syntax as a Latent Variable
The core intuition behind this field is that hierarchical structure is a necessary tool for solving complex tasks. Rather than forcing a model to follow human-defined grammar rules, researchers treat the sentence structure as a latent variable that the model must optimize to improve its performance on a downstream objective.
For example, in machine translation, Neubig et al. demonstrated that a model could induce a "reordering parser" by treating sentence structure as a hidden variable optimized for word alignment. Similarly, in grounded question answering, Gupta et al. used a differentiable parse chart to evaluate all possible binary trees, learning to weight the most semantically compatible structures.
This "task-specific syntax" often proves more effective for machines than traditional linguistics. Brychcín showed that latent trees induced to optimize word sequence prediction can actually outperform human-annotated syntactic trees in reducing perplexity. The model discovers a hierarchy that serves the data, even if it doesn't perfectly align with a linguist's textbook.
Emergence from Sequential Objectives
Perhaps the most surprising finding in this area is that models do not always need explicit structural components to learn grammar. Standard sequence models, like LSTMs and Transformers, often develop a "hierarchical inductive bias" simply by trying to predict the next word or a masked token.
- Recurrent Models: Gulordava et al. proved that LSTMs trained on generic language modeling could track complex dependencies in "nonce" sentences (grammatically correct but nonsensical), suggesting they internalize abstract syntax. Goldberg further argued that these sequential models capture hierarchical rules like subject-verb agreement through indirect statistical supervision.
- Attention-Based Models: In large pre-trained models like BERT, syntax is often "baked in." Wu et al. developed "Perturbed Masking," a technique to extract syntax trees directly from BERT's internal representations without any additional training. They found that the model’s internal natural syntax is robust enough to reconstruct accurate trees across different languages and even source code (Wan et al.).
Structural Inductive Biases
While syntax can emerge from flat sequences, many researchers argue for building "soft" structural constraints into the architecture itself. These constraints guide the model toward hierarchical organization without requiring explicit labels.
One approach is the Tree Transformer, proposed by Wang et al., which uses a "Constituent Attention" module to group words into phrases. By optimizing a masked language modeling objective, the model learns to identify phrase boundaries and structural anchors. Similarly, Luo et al. induced phrases by assigning "syntactic heights" to tokens, allowing the model to predict entire future phrases rather than just single words.
Other researchers look at specialized memory structures. Merrill et al. found that the push-and-pop operations of differentiable stacks naturally align with the boundaries of linguistic constituents like verb phrases. By providing a stack-based bias, the model is incentivized to organize information hierarchically to manage its internal memory.
Why It Works: The Bottleneck Principle
The reason unsupervised induction works is that hierarchy is an efficient solution to the information bottleneck of sequential data. To predict a distant verb based on a subject, a model must "remember" the subject across many intervening words.
As Cheng et al. noted, internal attention mechanisms allow models to bypass the linear sequence and link related words directly. This creates a "soft" structural bias. Whether through recursive merging (Socher et al.), latent subtrees (Xia et al.), or gated convolutions (Cho et al.), the model is essentially discovering the most compressed, predictive representation of the input. In the world of natural language, that representation is almost always a tree.