Keep Your Bearings: Lightly-Supervised Information Extraction with Ladder Networks That Avoids Semantic Drift

Ajay Nagesh, Mihai SurdeanuNAACL 2018 · OpenReview

Introduction

Named Entity Classification (NEC) is a fundamental task in Information Extraction (IE) that involves categorizing text mentions—such as "Apple" or "Washington"—into predefined types like Person, Organization, or Location. While supervised learning achieves high performance, the cost of producing high-quality labeled data is often prohibitive. Consequently, researchers frequently turn to semi-supervised learning, particularly bootstrapping.

Bootstrapping starts with a small set of "seed" examples and iteratively expands the labeled set by identifying new patterns and entities. However, this iterative process is prone to semantic drift. Because errors compound at each step, a system initially searching for female names might eventually drift into learning flower names after encountering "Daisy," losing its original semantic grounding.

The Ladder Network Framework

To address these challenges, this paper proposes a non-iterative approach using Ladder Networks (LNs). A Ladder Network is a deep denoising auto-encoder characterized by lateral skip connections between the encoder and decoder at every layer.

The Ladder Network architecture (left) and the input representation for entity mentions and contexts (right).

The architecture consists of:

  • Encoder: Processes the input through multiple layers, adding noise at each step to create a "corrupted" path alongside a "clean" one.
  • Decoder: Reconstructs the clean data from the noisy representation.
  • Skip Connections: These connections allow the network to pass low-level details directly to the decoder, freeing higher layers to focus on abstract features relevant to classification.

The model is trained using a joint loss function that minimizes classification error on a small labeled set while simultaneously minimizing reconstruction error on a large unlabeled corpus. By processing all data in a single training phase, the LN avoids the compounding errors inherent in iterative promotion.

Input Representation

The input to the network represents an entity mention by concatenating two distinct embeddings:

  1. Entity Embedding: The average of pre-trained word embeddings for the words within the mention (e.g., "Ann Stephens").
  2. Context Embedding: The average of embeddings for $n$-gram patterns surrounding the mention, capturing the linguistic environment (e.g., "said @ENTITY, president").

Experimental Evaluation

The approach was evaluated on two standard benchmarks: CoNLL 2003 (4 entity types) and Ontonotes (11 entity types). To simulate a "lightly-supervised" scenario, the authors used extremely small labeled sets: only 0.3% for CoNLL and 0.6% for Ontonotes.

Results and Discussion

The Ladder Network significantly outperformed traditional bootstrapping (EPB) and Label Propagation (LP) methods. On CoNLL, the LN achieved 66.11% accuracy compared to 40.74% for the baseline. On the more complex Ontonotes dataset, the performance gap was even wider: 63.12% for LN versus 21.06% for the baseline.

Precision vs. Throughput on CoNLL and Ontonotes, showing how Ladder Networks maintain purity while baselines suffer from semantic drift.

Crucially, the LN maintained high "pool purity"—the precision of predicted categories—as more entities were processed. In contrast, the precision of bootstrapping methods plummeted, a clear indicator of semantic drift. Furthermore, the LN approach proved to be 7x faster to train than iterative baselines, offering a significant practical advantage for large-scale applications.

Conclusion

By replacing iterative bootstrapping with a joint optimization framework, Ladder Networks effectively eliminate semantic drift in NEC. This work demonstrates that deep denoising auto-encoders with skip connections can achieve state-of-the-art results with minimal supervision. Future research may explore more complex encoders, such as CNNs or RNNs, and extend this framework to other IE tasks like Relation Extraction or Sentiment Analysis.

Summary updated

BibTeX
@inproceedings{nagesh2018keep,
  author = {Ajay Nagesh and Mihai Surdeanu},
  title = {{Keep Your Bearings: Lightly-Supervised Information Extraction with Ladder Networks That Avoids Semantic Drift}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=H1Vtx7WuZH},
}
Figure