Abstract
Natural language is hierarchically structured: smaller units (e.g., phrases) are nested within larger units (e.g., clauses). When a larger constituent ends, all of the smaller constituents that are nested within it must also be closed. While the standard LSTM architecture allows different neurons to track information at different time scales, it does not have an explicit bias towards modeling a hierarchy of constituents. This paper proposes to add such an inductive bias by ordering the neurons; a vector of master input and forget gates ensures that when a given neuron is updated, all the neurons that follow it in the ordering are also updated. Our novel recurrent architecture, ordered neurons LSTM (ON-LSTM), achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.
Introduction
Natural language is inherently hierarchical, with words forming phrases, which in turn form clauses and sentences. A fundamental property of this hierarchy is nesting: when a high-level constituent (like a sentence) concludes, all subordinate units nested within it must also close. Standard Recurrent Neural Networks (RNNs) and LSTMs typically process data as linear sequences. Although LSTMs utilize gating mechanisms to track information across varying time scales, they lack an explicit structural bias to represent these nested tree structures. Consequently, these models often struggle with long-term dependencies and fail to capture the underlying syntax essential for human-like language understanding.
The Ordered Neurons LSTM (ON-LSTM) addresses this by introducing an inductive bias that forces the network to allocate internal memory in a way that mirrors linguistic hierarchy. By enforcing an ordering on neurons, the model can naturally learn hierarchical structures from sequential data.
Methodology: Ordered Neurons
The core innovation of this work is the Ordered Neurons inductive bias, which ranks neurons based on their "life cycle." High-ranking neurons are designated to store global, long-term information (e.g., sentence subjects), while low-ranking neurons track local, short-term information (e.g., part-of-speech tags).
The cumax() Activation and Master Gates
To enforce this ranking, the authors introduce cumax(), a cumulative version of the softmax function. This produces a vector that transitions between zeros and ones, acting as a "master gate" to define a split point within the neuron vector. Two master gates are integrated into the LSTM architecture:
- Master Forget Gate ($\tilde{f}_t$): Controls the erasure of information. If a high-ranking neuron is erased, all neurons lower in the hierarchy are also erased, mimicking the closure of a large linguistic constituent.
- Master Input Gate ($\tilde{i}_t$): Controls the writing of new information. Updating a high-ranking neuron implies the start of a new high-level structure, necessitating updates to lower-level neurons as well.
Experimental Evaluation
The ON-LSTM was evaluated across four distinct tasks to test its linguistic and logical capabilities:
- Language Modeling: Evaluated on the Penn Treebank (PTB) for next-word prediction.
- Unsupervised Constituency Parsing: Tested on the Wall Street Journal (WSJ) portion of PTB to determine if internal gates align with human-annotated parse trees.
- Targeted Syntactic Evaluation: Assessed using minimally different sentence pairs to test specific phenomena like subject-verb agreement.
- Logical Inference: Evaluated on synthetic logical expressions (e.g.,
and,or,not) to test recursive structure handling.
Key Results
- Language Modeling: ON-LSTM achieved a test perplexity of 56.17 on PTB, outperforming the standard AWD-LSTM (57.3) with a comparable parameter count.
- Unsupervised Parsing: The model established a new state-of-the-art for unsupervised parsing on the WSJ test set. The second layer achieved an F1 score of 47.7, significantly outperforming previous models like PRPN, particularly on longer sentences.
- Logical Inference: ON-LSTM demonstrated superior generalization to longer sequences compared to standard LSTMs. When trained on sequences of length $\leq 6$, it maintained high accuracy on sequences up to length 12.
Discussion and Insights
A primary finding is that ON-LSTM induces tree structures purely from a language modeling objective, without exposure to gold-standard parse trees. The master gates effectively implement "stack-like" behavior, where erasing a high-level neuron automatically clears the sub-stack of lower-level neurons.
Furthermore, the authors observed layer differentiation: middle layers typically provide the most accurate syntactic trees, while the first and last layers focus on local word-level features.
Conclusion and Future Work
While ON-LSTM successfully integrates hierarchical bias, it currently assumes a fixed linear ordering of neurons. Future research could explore dynamic ordering or multidimensional hierarchies. Additionally, while effective for English, the impact of this bias may vary in morphologically rich languages. Despite some computational overhead from the cumax operation, the model's ability to capture syntax from raw text marks a significant step in bridging the gap between connectionist models and symbolic linguistic structure.
Summary updated