Global Belief Recursive Neural Networks

Romain Paulus, Richard Socher, Christopher D. Manning

Introduction

Standard Recursive Neural Networks (RNNs) often struggle with words that change meaning based on their surrounding context. This paper introduces Global Belief Recursive Neural Networks (GB-RNNs), an architecture that adds a "feedbackward" pass to the traditional bottom-up approach. This innovation allows the global context of a sentence to flow backward, refining the sentiment of individual words and phrases.

The Problem: Composition vs. Contextualization

In natural language processing, the meaning of a word often depends entirely on its surroundings. Consider the sentence "Android beats iOS." In this context, the word "Android" carries a positive sentiment. However, a standard feedforward RNN—which builds meaning strictly from the bottom up—would likely label "Android" as neutral because it evaluates the word before it "sees" the rest of the sentence.

Existing models excel at composition (combining words into larger phrases) but fall short at contextualization (using the whole sentence to re-evaluate the parts). The authors aim to bridge this gap, specifically for aspect-specific sentiment analysis where users care about the sentiment directed toward a specific entity within a complex sentence.

The GB-RNN Approach

The authors propose a two-step inference process to capture both local composition and global context:

  1. The Feedforward Pass (Standard RNN): The model parses a sentence into a binary tree. It starts at the leaves (words) and moves up to the root, combining vectors to represent increasingly larger phrases.
  2. The Feedbackward Pass (The Innovation): Once the model reaches the root (representing the "global belief" of the entire sentence), it performs a second pass back down the tree. This top-down recursion allows information from the entire sentence to flow back to individual nodes, updating their representations.
Feedforward vs. Feedbackward passes in GB-RNN

Additional Architectural Improvements

  • Hybrid Word Vectors: Instead of choosing between pre-trained unsupervised vectors (like GloVe or Word2Vec) and task-specific supervised vectors, the authors concatenate them. They use a 100-dimensional fixed unsupervised vector and a 30-dimensional supervised vector that is updated during training.
  • Dropout for RNNs: The paper demonstrates that applying dropout (randomly setting units to zero during training) to the hidden layers of an RNN significantly improves performance and prevents overfitting, especially when using high-dimensional word vectors.

Datasets

The model was rigorously evaluated on the SemEval 2013 Task 2 dataset, which includes:

  • Twitter Dataset: 7,862 training phrases and a test set of over 10,000 examples.
  • SMS Dataset: Short text messages used to test the model's ability to generalize to different types of informal text.
  • External Data: The authors also utilized the NRC-Canada sentiment lexicon and a self-collected dataset of 176,311 "noisily labeled" tweets (using emoticons as proxies for sentiment) for pre-training.

Key Insights

  • Context is a Two-Way Street: Meaning doesn't just flow from words to sentences; it flows from sentences back to words. By adding the backward pass, the model "re-thinks" its initial assessment of a word once it understands the full context.
  • Disentangling Representations: The feedbackward pass helps "disentangle" children nodes from their parents. By knowing what the whole sentence means, the model can better determine which specific part of a phrase contributed to a specific sentiment.
  • The Power of Hybridity: Combining fixed, general-purpose word embeddings with flexible, task-specific ones provides a "best of both worlds" scenario—retaining broad linguistic knowledge while fine-tuning for the nuances of sentiment.

Key Results

The GB-RNN achieved State-of-the-Art Performance, outperforming the winners of the SemEval 2013 challenge. It reached an F1 score of 89.41 on the Twitter dataset and 88.40 on the SMS dataset.

Performance comparison of GB-RNN against SVM baselines

Crucially, the feedbackward step provided a 3% to 5% improvement in F1 score over the standard feedforward RNN. As demonstrated in the "Android vs. iOS" example, the model successfully flipped the sentiment of neutral words to positive or negative based on the global context—a feat previous recursive models couldn't reliably achieve.

Limitations and Future Work

  • Dependency on Parsers: The model relies on the Stanford Parser to create the initial binary tree. If the parser fails on messy social media text (e.g., heavy slang or poor grammar), the GB-RNN's performance may suffer.
  • Computational Cost: Adding a second pass through the tree doubles the inference steps compared to a standard RNN, though the authors argue the accuracy gain justifies the cost.
  • Broader Applications: While this paper focuses on sentiment, the authors suggest that "global belief" propagation could be highly effective for other NLP challenges, such as word-sense disambiguation (e.g., deciding which meaning of the word "bank" is intended based on the sentence).

Summary updated

Abstract

Recursive Neural Networks have recently obtained state of the art performance on several natural language processing tasks. However, because of their feedforward architecture they cannot correctly predict phrase or word labels that are determined by context. This is a problem in tasks such as aspect-specific sentiment classification which tries to, for instance, predict that the word Android is positive in the sentence Android beats iOS. We introduce global belief recursive neural networks (GB-RNNs) which are based on the idea of extending purely feedforward neural networks to include one feedbackward step during inference. This allows phrase level predictions and representations to give feedback to words. We show the effectiveness of this model on the task of contextual sentiment analysis. We also show that dropout can improve RNN training and that a combination of unsupervised and supervised word vector representations performs better than either alone. The feedbackward step improves F1 performance by 3% over the standard RNN on this task, obtains state-of-the-art performance on the SemEval 2013 challenge and can accurately predict the sentiment of specific entities.
BibTeX
@inproceedings{paulus2014global,
  author = {Romain Paulus and Richard Socher and Christopher D. Manning},
  title = {{Global Belief Recursive Neural Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2014},
}
Figure