Abstract
Recursive Neural Networks (RecNNs) are effective for processing hierarchical text structures but often suffer from signal dilution, where critical keywords are overshadowed by less relevant tokens. This paper introduces Weighted Neural Networks (WNN) and Binary-Expectation Neural Networks (BENN), two architectures that incorporate gating mechanisms to dynamically tune feature weights. By learning to "mute" noise and amplify salient signals, these models achieve state-of-the-art results in sentiment analysis and document coherence tasks.
Introduction
Recursive Neural Networks (RecNNs) process text according to its inherent hierarchical structure, such as a parse tree. While powerful, standard RecNNs treat every node in the tree with roughly equal importance during the composition process. This leads to a specific form of the gradient vanishing problem: in a sentence like "The movie is wonderful," the sentiment-bearing signal from "wonderful" can be diluted as it is combined with less relevant tokens like "the" and "is" at each layer. If a keyword is buried deep in a large parse tree, its influence may fail to reach the final sentence representation at the root.
Methods
To address signal dilution, the authors propose two novel architectures that borrow the concept of "weight tuning" from traditional models like SVMs and "gates" from LSTMs:
- Weighted Neural Network (WNN): Each node $C$ in the parse tree is assigned a learned weight $M_C \in [0, 1]$ based on its own representation. The output passed to the parent node is the element-wise product $M_C * h_C$, allowing the model to "soft-mute" uninformative nodes.
- Binary-Expectation Neural Network (BENN): This probabilistic approach uses a binary variable $B_C$ to indicate whether information should be passed up. To maintain differentiability, the model uses the expectation of the embedding, scaling the representation by the probability that a node is "important."
Key Results and Insights
The proposed models were evaluated on standard benchmarks including the Stanford Sentiment Treebank (SST), the IMDB dataset, and coherence prediction corpora. WNN and BENN consistently outperformed standard recursive models:
- Sentiment Analysis (SST): BENN achieved 87.0% accuracy on binary classification, compared to 82.4% for the standard Recursive model.
- Document Analysis: BENN reached a precision of 0.910 on the IMDB dataset, surpassing both recursive-recurrent hybrids and bigram SVMs.
- Coherence: The BENN-recursive model set a new high mark of 93.6% accuracy in sentence ordering tasks.
A key insight is the model's ability to perform automatic feature selection. Unlike standard RecNNs, WNN/BENN learn to ignore "stop words" automatically through the objective function, inducing a sparsity similar to that of Bag-of-Words SVMs.
Limitations and Future Work
While WNN/BENN represent a significant advancement for recursive models, they still slightly underperform the Paragraph Vector approach on certain coarse-grained sentiment tasks. Future work may explore the computational complexity of calculating weights for every node in very large trees and extending this gating logic to non-recursive deep learning architectures.
Summary updated