Abstract
Recursive Neural Networks (RecNNs) are powerful tools for modeling the hierarchical structure of natural language. However, a significant limitation of standard RecNNs is their reliance on a single, shared weight matrix for all composition operations, regardless of the grammatical categories involved. This paper introduces Tag-Guided Hyper-Recursive Neural Networks (TG-HRecNN) and TG-HTreeLSTM, which leverage Part-of-Speech (POS) tags to dynamically generate composition parameters. By using a secondary hypernetwork to guide the main composition network, our approach achieves specialized linguistic modeling with high parameter efficiency. We demonstrate state-of-the-art results on the Stanford Sentiment Treebank (SST) and consistent improvements across multiple NLP benchmarks.
Introduction
In Natural Language Processing, representing a sentence as a fixed-size vector is a core task. Recursive Neural Networks achieve this by following a sentence's syntax tree, recursively combining child nodes into parent nodes. A persistent challenge in this architecture is the "one-size-fits-all" nature of the composition function. Intuitively, the semantic fusion of an adjective and a noun (e.g., "very good") should differ from that of a verb and its object (e.g., "ate pizza").
Existing solutions either use a unique matrix for every possible tag combination—leading to a "parameter explosion"—or ignore explicit syntactic rules entirely. The proposed Tag-Guided models bridge this gap by using POS tags to dynamically modulate the weights of the composition function.
Methodology: Tag-Guided Hyper-Networks
The core innovation is the separation of the model into two interacting components: a Main Network and a Hypernetwork.
- Main Network: This is a standard RecNN or TreeLSTM that performs the actual semantic composition of word and phrase vectors.
- Hypernetwork: A smaller, auxiliary network that takes POS tags and "head-lexicalized" information as input. Instead of computing phrase meanings, it generates scaling factors ($z$ vectors) that modify the weight matrices of the Main Network at each specific node.
This "Aha!" mechanism allows the model to effectively learn specialized composition rules for different grammatical structures. An Information Fusion Layer ensures that the hypernetwork combines discrete POS tag embeddings with semantic context before adjusting the main weights.
Experimental Results
The authors evaluated the proposed models on five major benchmarks: SST (Sentiment), MR (Movie Reviews), SUBJ (Subjectivity), TREC (Question Classification), and SICK (Semantic Entailment).
As shown in the results table, the TG-HTreeLSTM models consistently outperform their vanilla counterparts. Notably, on the fine-grained SST-1 task, the model achieved 53.7% accuracy, matching or exceeding much more complex state-of-the-art architectures. Furthermore, the TG-HRecNN showed a remarkable 6.4% improvement over the standard RecNN on the same dataset.
Beyond accuracy, the model is highly parameter-efficient. The TG-HTreeLSTM achieves these results with only 0.84M parameters, significantly fewer than other high-performing variants like the BiTreeLSTM (1.30M).
Conclusion
By incorporating syntactic guidance through POS tags, Tag-Guided Hyper-networks provide a flexible and efficient way to model the diverse ways words combine in natural language. This approach avoids the pitfalls of parameter-heavy "untied" models while capturing the nuanced linguistic rules that standard RecNNs miss. Future work may explore applying these encoders to document-level tasks or integrating them into parallelizable architectures like Transformers to mitigate the speed limitations of recursive processing.
Summary updated