Spanish NER with Word Representations and Conditional Random Fields

Jenny Copara, José Ochoa, Camilo Thorne, Goran GlavasNEWS@ACM 2016 · OpenReview

Abstract

Authors: Jenny Linet Copara Zea, Jose Eduardo Ochoa Luna, Camilo Thorne, Goran Glavaš.
Source: Proceedings of the Sixth Named Entity Workshop, 2016.

Introduction

Named Entity Recognition (NER)—the task of identifying and categorizing entities like people, locations, and organizations in text—usually requires massive amounts of manually labeled data. For languages like Spanish, such "gold standard" data is often scarce. While Deep Learning (DL) has recently pushed the boundaries of NER performance, these models can be computationally expensive and often function as "black boxes."

This paper investigates whether more transparent, traditional probabilistic models, specifically linear chain Conditional Random Fields (CRFs), can be "boosted" using word representations learned from massive amounts of unlabeled text. By leveraging large unlabeled corpora—including English datasets—the authors aim to achieve state-of-the-art performance without the overhead of a full Deep Learning pipeline.

Methodology

The researchers used a linear chain CRF as their base classifier. They began with a set of standard baseline features, such as POS tags, capitalization patterns, and prefixes/suffixes. They then systematically added four types of unsupervised word representations to enhance the model's feature space:

  1. Brown Clustering: A hierarchical clustering method that groups words based on the contexts in which they appear.
  2. Clustering Embeddings: Applying k-means clustering to continuous word vectors to create discrete features.
  3. Binarized Embeddings: Converting continuous word vectors into discrete bit-strings based on mean value thresholds.
  4. Distributional Prototypes: Identifying words most strongly associated with specific entity types (e.g., "Maria" for PERSON) using Normalized Pointwise Mutual Information (NPMI) and mapping them to embeddings.
Example of Brown Clusters and their associated words, showing semantic grouping of entities and related terms.

A core innovation of this study is the cross-lingual approach: using Brown clusters generated from English Wikipedia to improve a Spanish NER model. This is supported by the fact that many named entities are identical across languages.

Key Insights and Discussion

  • The Power of Cross-Lingual Features: Surprisingly, the best-performing model used Brown clusters derived from English text. This works because many named entities (like "Facebook," "Twitter," or "Hamburg") are identical across languages, and the context in which they appear is often structurally similar.
  • Binarization is a Double-Edged Sword: While binarizing embeddings helps in English NER, it actually decreased performance in Spanish when used alone. The authors suggest this is because the process throws away too much relevant information for the Spanish language's specific nuances.
  • Capitalization is Critical: The researchers found that using lowercased prototypes led to a significant drop in F-score (0.78%). In Spanish NER, the "real context" provided by casing is vital for mapping words to the correct vector space.
  • Feature Synergy: While individual representations like "Prototypes" or "Binarization" performed poorly on their own, they provided a significant boost when combined with Brown clusters and k-means clustering.

Results

The experimental results, evaluated on the CoNLL-2002 Spanish Corpus, demonstrate that smart feature engineering can indeed compete with modern neural approaches.

F1 performance of various CRF configurations compared to state-of-the-art models.

As shown in the results table, the Baseline F-score was 80.02%. The best-performing configuration (Brown + Clustering + Prototypes) achieved an F1-score of 82.44%*. This approach outperformed the long-standing 2002 baseline (81.39%) and even beat some early Deep Learning models, such as dos Santos and Guimarães (2015) at 82.21%. While it falls short of the absolute current state-of-the-art (which sits around 85.77% using Recurrent Neural Networks), it offers a much more interpretable and computationally efficient alternative.

Conclusion

This research confirms that traditional probabilistic models like CRFs remain highly relevant for NER tasks when augmented with rich, unsupervised word representations. The success of cross-lingual features suggests promising avenues for low-resource language processing by leveraging data from high-resource languages. Future work will focus on exploring even larger-scale cross-lingual datasets and investigating how these word representations might be better integrated into the CRF framework to close the gap with Deep Learning.

Summary updated

BibTeX
@inproceedings{copara2016spanish,
  author = {Jenny Copara and José Ochoa and Camilo Thorne and Goran Glavas},
  title = {{Spanish NER with Word Representations and Conditional Random Fields}},
  booktitle = {NEWS@ACM},
  year = {2016},
  url = {https://openreview.net/forum?id=MZ8HEoRjgr},
}
Figure