Learning to Compose over Tree Structures via POS Tags

Gehui Shen, Zhi-Hong Deng, Ting Huang, Xi Chen

Abstract

Recursive Neural Networks (RecNNs) are powerful tools for modeling the hierarchical structure of natural language. However, a significant limitation of standard RecNNs is their reliance on a single, shared weight matrix for all composition operations, regardless of the grammatical categories involved. This paper introduces Tag-Guided Hyper-Recursive Neural Networks (TG-HRecNN) and TG-HTreeLSTM, which leverage Part-of-Speech (POS) tags to dynamically generate composition parameters. By using a secondary hypernetwork to guide the main composition network, our approach achieves specialized linguistic modeling with high parameter efficiency. We demonstrate state-of-the-art results on the Stanford Sentiment Treebank (SST) and consistent improvements across multiple NLP benchmarks.

Introduction

In Natural Language Processing, representing a sentence as a fixed-size vector is a core task. Recursive Neural Networks achieve this by following a sentence's syntax tree, recursively combining child nodes into parent nodes. A persistent challenge in this architecture is the "one-size-fits-all" nature of the composition function. Intuitively, the semantic fusion of an adjective and a noun (e.g., "very good") should differ from that of a verb and its object (e.g., "ate pizza").

Existing solutions either use a unique matrix for every possible tag combination—leading to a "parameter explosion"—or ignore explicit syntactic rules entirely. The proposed Tag-Guided models bridge this gap by using POS tags to dynamically modulate the weights of the composition function.

Methodology: Tag-Guided Hyper-Networks

The core innovation is the separation of the model into two interacting components: a Main Network and a Hypernetwork.

Figure 2: The architecture of the proposed Tag-Guided Hyper-Recursive Neural Network. The hypernetwork (blue) processes POS tags and lexical information to generate dynamic weights for the main composition network (black).
  1. Main Network: This is a standard RecNN or TreeLSTM that performs the actual semantic composition of word and phrase vectors.
  2. Hypernetwork: A smaller, auxiliary network that takes POS tags and "head-lexicalized" information as input. Instead of computing phrase meanings, it generates scaling factors ($z$ vectors) that modify the weight matrices of the Main Network at each specific node.

This "Aha!" mechanism allows the model to effectively learn specialized composition rules for different grammatical structures. An Information Fusion Layer ensures that the hypernetwork combines discrete POS tag embeddings with semantic context before adjusting the main weights.

Experimental Results

The authors evaluated the proposed models on five major benchmarks: SST (Sentiment), MR (Movie Reviews), SUBJ (Subjectivity), TREC (Question Classification), and SICK (Semantic Entailment).

Figure 6: Performance comparison of the proposed TG-H models against various baselines across six datasets. Bold values indicate the best performance in each category.

As shown in the results table, the TG-HTreeLSTM models consistently outperform their vanilla counterparts. Notably, on the fine-grained SST-1 task, the model achieved 53.7% accuracy, matching or exceeding much more complex state-of-the-art architectures. Furthermore, the TG-HRecNN showed a remarkable 6.4% improvement over the standard RecNN on the same dataset.

Beyond accuracy, the model is highly parameter-efficient. The TG-HTreeLSTM achieves these results with only 0.84M parameters, significantly fewer than other high-performing variants like the BiTreeLSTM (1.30M).

Conclusion

By incorporating syntactic guidance through POS tags, Tag-Guided Hyper-networks provide a flexible and efficient way to model the diverse ways words combine in natural language. This approach avoids the pitfalls of parameter-heavy "untied" models while capturing the nuanced linguistic rules that standard RecNNs miss. Future work may explore applying these encoders to document-level tasks or integrating them into parallelizable architectures like Transformers to mitigate the speed limitations of recursive processing.

Summary updated

BibTeX
@article{shen2020learning,
  author = {Gehui Shen and Zhi-Hong Deng and Ting Huang and Xi Chen},
  title = {{Learning to Compose over Tree Structures via POS Tags}},
  journal = {Expert Systems with Applications},
  year = {2020},
  doi = {10.1016/j.eswa.2019.112917},
}
Figure