Feature Weight Tuning for Recursive Neural Networks

Jiwei LiarXiv 2014 · arXiv

Abstract

Recursive Neural Networks (RecNNs) are effective for processing hierarchical text structures but often suffer from signal dilution, where critical keywords are overshadowed by less relevant tokens. This paper introduces Weighted Neural Networks (WNN) and Binary-Expectation Neural Networks (BENN), two architectures that incorporate gating mechanisms to dynamically tune feature weights. By learning to "mute" noise and amplify salient signals, these models achieve state-of-the-art results in sentiment analysis and document coherence tasks.

Introduction

Recursive Neural Networks (RecNNs) process text according to its inherent hierarchical structure, such as a parse tree. While powerful, standard RecNNs treat every node in the tree with roughly equal importance during the composition process. This leads to a specific form of the gradient vanishing problem: in a sentence like "The movie is wonderful," the sentiment-bearing signal from "wonderful" can be diluted as it is combined with less relevant tokens like "the" and "is" at each layer. If a keyword is buried deep in a large parse tree, its influence may fail to reach the final sentence representation at the root.

Methods

To address signal dilution, the authors propose two novel architectures that borrow the concept of "weight tuning" from traditional models like SVMs and "gates" from LSTMs:

  1. Weighted Neural Network (WNN): Each node $C$ in the parse tree is assigned a learned weight $M_C \in [0, 1]$ based on its own representation. The output passed to the parent node is the element-wise product $M_C * h_C$, allowing the model to "soft-mute" uninformative nodes.
  2. Binary-Expectation Neural Network (BENN): This probabilistic approach uses a binary variable $B_C$ to indicate whether information should be passed up. To maintain differentiability, the model uses the expectation of the embedding, scaling the representation by the probability that a node is "important."
The proposed gating mechanism allows the network to dynamically scale the influence of specific branches in the parse tree.

Key Results and Insights

The proposed models were evaluated on standard benchmarks including the Stanford Sentiment Treebank (SST), the IMDB dataset, and coherence prediction corpora. WNN and BENN consistently outperformed standard recursive models:

  • Sentiment Analysis (SST): BENN achieved 87.0% accuracy on binary classification, compared to 82.4% for the standard Recursive model.
  • Document Analysis: BENN reached a precision of 0.910 on the IMDB dataset, surpassing both recursive-recurrent hybrids and bigram SVMs.
  • Coherence: The BENN-recursive model set a new high mark of 93.6% accuracy in sentence ordering tasks.

A key insight is the model's ability to perform automatic feature selection. Unlike standard RecNNs, WNN/BENN learn to ignore "stop words" automatically through the objective function, inducing a sparsity similar to that of Bag-of-Words SVMs.

Visualization of learned weights: darker nodes indicate higher importance. The model successfully mutes stop words like 'the' and 'is' while highlighting sentiment-rich words like 'wonderful' and 'tedious'.

Limitations and Future Work

While WNN/BENN represent a significant advancement for recursive models, they still slightly underperform the Paragraph Vector approach on certain coarse-grained sentiment tasks. Future work may explore the computational complexity of calculating weights for every node in very large trees and extending this gating logic to non-recursive deep learning architectures.

Summary updated

BibTeX
@misc{li2014feature,
  author = {Jiwei Li},
  title = {{Feature Weight Tuning for Recursive Neural Networks}},
  year = {2014},
  eprint = {1412.3714},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1412.3714},
}
Figure