Neural Networks Leverage Corpus-wide Information for Part-of-speech Tagging

Yuta TsuboiEMNLP 2014 · DOI

Introduction

Part-of-speech (POS) tagging—assigning labels like "noun" or "verb" to words—is a cornerstone of natural language processing. Historically, the most effective taggers relied on sparse discrete features, such as checking if a word is preceded by "the". While successful, these models struggle to incorporate "corpus-wide" information: global statistics about word behavior across millions of sentences.

Global information, such as word embeddings or general tag probabilities, is typically represented as dense continuous features (vectors). In standard linear models, manually engineering the non-linear interactions between these features leads to a combinatorial explosion of parameters. This paper introduces a hybrid approach that leverages neural networks to automatically learn these complex interactions.

The Hybrid Architecture

The researchers proposed a hybrid model that integrates a feed-forward neural network into a conventional transition-based tagger. This design allows the system to benefit from both traditional feature engineering and modern deep learning.

The hybrid neural network architecture combines sparse discrete inputs with dense continuous corpus-wide features.

As illustrated in the architecture above, the model consists of:

  • A Linear Component: Handles traditional, manually engineered binary features (N-grams of words and tags).
  • A Neural Component: A feed-forward network that takes dense, corpus-wide features as input. It utilizes four types of global information: word embeddings (word2vec and GloVe), POS tag distributions, supertag distributions, and context word distributions.
  • Advanced Activations: Instead of standard ReLU functions, the model employs Maxout and $L_p$-pooling layers to better capture feature interactions and reduce variance in hidden activations.

Key Insights and Results

A major finding of this work is that the choice of activation function is critical. Maxout and $L_p$-pooling outperformed standard Rectified Linear Units (ReLUs), suggesting that "pooling" multiple feature combinations is particularly effective for linguistic data. Furthermore, PCA visualizations revealed that the network's hidden layers naturally clustered words by their POS tags, even without explicit instruction to do so.

The hybrid model established new state-of-the-art records for English POS tagging at the time of its publication.

Comparison of the hybrid model ($L_p, p=2$) against previous benchmarks on the Penn Treebank and CoNLL 2009 datasets.

As shown in the results tables, the model achieved:

  • 97.51% accuracy on the Penn Treebank (PTB), surpassing Manning (2011).
  • 98.02% accuracy on the CoNLL 2009 dataset, outperforming complex joint models.

Notably, the model showed significant gains in tagging "unknown" words (those not seen during training). This is attributed to the global word embeddings, which provide context about a word's behavior based on its usage in large, unlabeled corpora.

Conclusion and Future Work

The success of this hybrid approach demonstrates that combining linear models for local context with neural networks for global context provides a powerful framework for NLP tasks. While the model is computationally more intensive than pure linear baselines—taking roughly seven times longer to train—the accuracy gains, particularly for unknown words, are substantial. Future research may explore the use of Recurrent Neural Networks (RNNs) to better capture long-distance dependencies and the history of the tagging sequence.

Summary updated

BibTeX
@inproceedings{tsuboi2014neural,
  author = {Yuta Tsuboi},
  title = {{Neural Networks Leverage Corpus-wide Information for Part-of-speech Tagging}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2014},
  doi = {10.3115/v1/d14-1101},
}
Figure