A New Method of Region Embedding for Text Classification

chao qiao, bo huang, guocheng niu, daren li, daxiang dong, wei he, dianhai yu, hua wuICLR 2018 · OpenReview

TL;DR

This paper introduces Region Embeddings, a novel way to represent text by learning how individual words interact with their local neighbors. By assigning each word a "local context unit" (a weighting matrix) in addition to a standard embedding, the model learns task-specific, phrase-like representations that outperform state-of-the-art methods like FastText and VDCNN on major text classification benchmarks.

What Problem Does This Paper Address?

Text classification often relies on two extremes: Bag-of-Words (BoW), which ignores word order entirely, or n-grams, which capture local order but suffer from "data sparsity." As the length of an n-gram ($n$) increases, the number of possible combinations explodes, making it impossible to see every phrase during training.

Deep learning approaches like CNNs try to fix this using convolutional kernels to "slide" over text. However, standard CNN kernels are word-independent—the same filter is applied regardless of which word is in the center. The authors argue that a word's meaning is inextricably linked to its specific influence on its neighbors (e.g., the word "not" fundamentally changes the meaning of the word following it). They sought a method that captures these word-specific interactions without the memory explosion of traditional n-grams.

Methods

The core innovation is the Local Context Unit. In a standard model, a word $w$ is just a vector $e_w$. In this model, every word $w$ also has a matrix $K_w$.

When looking at a "region" (a window of words), the model calculates a Region Embedding using one of two proposed architectures:

Word-Context and Context-Word Region Embedding architectures
  1. Word-Context (W.C.): The matrix of the middle word in a window interacts with the embedding vectors of its neighbors. This captures how a central word "filters" or emphasizes its surrounding context.
  2. Context-Word (C.W.): The matrices of the neighboring words interact with the embedding vector of the middle word. This captures how the surrounding context "shapes" the meaning of the central word.

The interaction is performed via element-wise multiplication, followed by max-pooling across the window to extract the most salient features. Finally, all region embeddings in a document are summed and passed to a simple linear classifier. This makes the upper architecture much simpler and more robust than "very deep" models.

Datasets

The authors evaluated their models on eight well-known benchmark datasets for text classification, ranging from binary classification to 14-class problems:

  • Sentiment Analysis: Yelp Review (Polarity & Full), Amazon Review (Polarity & Full).
  • News Classification: AG’s News, Sogou News.
  • Question Answering: Yahoo! Answers.
  • Ontology Extraction: DBPedia.

Key Insights and Ideas

  • Words as Operators: Instead of treating words as static points in a vector space, this model treats them as operators. A word’s matrix $K_w$ acts as a custom "lens" through which it views its neighbors.
  • Task-Specific Phrases: Unlike previous "region embedding" methods that were unsupervised (like word2vec), these are learned directly for the classification task. This allows the model to learn that "not" is a critical "reversal" operator in sentiment analysis.
  • Overfitting Resilience: Despite having a large number of parameters (since every word has a matrix), the model proved surprisingly robust to overfitting.
Accuracy across region sizes and embedding dimensions compared to baselines

As shown in the experimental results, increasing embedding dimensions didn't hurt performance as much as it did for FastText or CNNs. The authors found that their model maintained high accuracy even as the embedding dimension scaled up to 1024.

  • Simplicity at the Top: By doing the "heavy lifting" at the word-interaction level, the rest of the network can be incredibly simple (just a sum and a linear layer), making it faster and more stable than complex architectures like LSTMs or deep CNNs.

Key Results

  • State-of-the-Art Performance: The Word-Context (W.C.) model beat or matched the best-published results on 6 out of 8 datasets.
  • Accuracy Gains: It outperformed the "Very Deep CNN" (VDCNN) on most tasks and significantly bested FastText. For example, on the Yelp Polarity dataset, it achieved 96.4% accuracy.
  • Interpretability: Heatmap visualizations showed the model correctly identifying that for words like "however" or "but," the right-hand context is more important for determining sentiment, while for "very," the immediate neighbor is key.

Limitations and Future Work

  • Parameter Count: Because every word in the vocabulary requires a matrix, the model's memory footprint is significantly larger than standard word embedding models. The authors mitigated this in experiments by using a relatively small embedding dimension (128).
  • Fixed Window Size: The model uses a fixed "region size" (e.g., 7 words). While they experimented with combining multiple sizes (3, 5, and 7), it still lacks the long-range dependency tracking of Transformers or RNNs.
  • Future Directions: The authors expressed interest in applying these local context units to unsupervised learning and more complex natural language processing tasks beyond simple classification.

Summary updated

Abstract

To represent a text as a bag of properly identified “phrases” and use the representation for processing the text is proved to be useful. The key question here is how to identify the phrases and represent them. The traditional method of utilizing n-grams can be regarded as an approximation of the approach. Such a method can suffer from data sparsity, however, particularly when the length of n-gram is large. In this paper, we propose a new method of learning and utilizing task-specific distributed representations of n-grams, referred to as “region embeddings”. Without loss of generality we address text classification. We specifically propose two models for region embeddings. In our models, the representation of a word has two parts, the embedding of the word itself, and a weighting matrix to interact with the local context, referred to as local context unit. The region embeddings are learned and used in the classification task, as parameters of the neural network classifier. Experimental results show that our proposed method outperforms existing methods in text classification on several benchmark datasets. The results also indicate that our method can indeed capture the salient phrasal expressions in the texts.
BibTeX
@inproceedings{qiao2018new,
  author = {chao qiao and bo huang and guocheng niu and daren li and daxiang dong and wei he and dianhai yu and hua wu},
  title = {{A New Method of Region Embedding for Text Classification}},
  booktitle = {International Conference on Learning Representations},
  year = {2018},
  url = {https://openreview.net/forum?id=BkSDMA36Z},
}
Figure