Revisiting Embedding Features for Simple Semi-supervised Learning

Jiang Guo, Wanxiang Che, Haifeng Wang, Ting Liu

Introduction

In the landscape of 2014 NLP, word embeddings like Word2Vec were rapidly becoming the standard for leveraging unlabeled data. However, a fundamental architectural mismatch existed: state-of-the-art systems primarily relied on linear models, such as Conditional Random Fields (CRFs). These models are designed for high-dimensional, sparse, discrete feature spaces, whereas word embeddings are low-dimensional, dense, and continuous.

This paper addresses the "Dense/Sparse Battle" by investigating how to transform continuous embeddings into a format that linear models can fully exploit. The authors demonstrate that simply concatenating dense vectors often underperforms compared to traditional methods like Brown clustering, and propose discretization as the solution.

Bridging the Gap: Discretization Strategies

The authors explore three primary methods to adapt continuous embeddings for linear frameworks:

  1. Binarization: Continuous values in embedding dimensions are converted into binary features. A feature "fires" if a value is significantly positive or negative relative to the mean, allowing the model to ignore noise and focus on strong signals.
  2. Clustering: By performing k-means clustering on embeddings, the model uses cluster IDs as discrete features. This captures semantic groupings similar to Brown clustering but utilizes the richer information present in modern embeddings.
  3. Distributional Prototypes (Novel): This task-specific method identifies "prototypes" for each label (e.g., "Michael" for the PER tag) using Normalized Pointwise Mutual Information (NPMI) on the training set.
Examples of automatically identified prototypes for various Named Entity types.

For any word in the test set, the system calculates its cosine similarity to these prototypes. If the similarity exceeds a tuned threshold, a task-specific feature is triggered. This effectively propagates label information from known entities to semantically similar words in the embedding space.

Integration of distributional prototype features into a Conditional Random Field (CRF) model.

Experimental Setup and Results

The methods were evaluated on the CoNLL-2003 Named Entity Recognition (NER) task using embeddings trained on a 1.52-billion-word English Wikipedia dump. The results highlight the superiority of discrete transformations over raw dense features:

  • Baseline (No embeddings): 83.43 F1
  • Direct Dense Embeddings: 86.21 F1
  • Binarized Embeddings: 86.75 F1
  • Distributional Prototype: 87.44 F1
  • Best Combination (Cluster + Prototype): 88.11 F1

The combined discrete approach not only outperformed direct embedding usage but also surpassed the long-standing gold standard of Brown clustering (87.49 F1). Beyond accuracy, these sparse binary features improved computational efficiency, reducing tagging time per sentence by 2x to 4x compared to dense vector math.

Conclusion and Future Directions

The study proves that for linear NLP models, the representation of features is as important as the information they contain. By discretizing embeddings—particularly through the novel distributional prototype method—researchers can achieve significant accuracy gains and faster inference. While the method requires tuning hyperparameters like the similarity threshold, it provides a powerful, simple baseline for semi-supervised learning. Future work may look toward "embedding kernels" to integrate continuous similarity more natively into linear frameworks.

Summary updated

Abstract

Recent work has shown success in using continuous word embeddings learned from unlabeled data as features to improve supervised NLP systems, which is regarded as a simple semi-supervised learning mechanism. However, fundamental problems on effectively incorporating the word embedding features within the framework of linear models remain. In this study, we investigate and analyze three different approaches, including a new proposed distributional prototype approach, for utilizing the embedding features. The presented approaches can be integrated into most of the classical linear models in NLP. Experiments on the task of named entity recognition show that each of the proposed approaches can better utilize the word embedding features, among which the distributional prototype approach performs the best. Moreover, the combination of the approaches provides additive improvements, outperforming the dense and continuous embedding features by nearly 2 points of F1 score.
BibTeX
@inproceedings{guo2014revisiting,
  author = {Jiang Guo and Wanxiang Che and Haifeng Wang and Ting Liu},
  title = {{Revisiting Embedding Features for Simple Semi-supervised Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2014},
  doi = {10.3115/v1/d14-1012},
}
Figure