Learning Better Internal Structure of Words for Sequence Labeling

Yingwei Xin, Ethan Hart, Vibhuti Mahajan, Jean-David RuviniEMNLP 2018 · DOI

Abstract

Sequence labeling tasks like Named Entity Recognition (NER) and Part-of-Speech (POS) tagging often rely on word-level representations. While sentence-level modeling has advanced significantly with deep architectures, word-level models—specifically those capturing character-level internal structures—have remained relatively shallow. This paper introduces IntNet, a deep, wide, and "funnel-shaped" convolutional architecture designed to capture morphological features like prefixes and suffixes without down-sampling. By preserving character order and utilizing dense connections, IntNet achieves state-of-the-art results on multiple benchmarks without external resources, demonstrating particular strength in morphologically rich languages and out-of-vocabulary word handling.

Introduction

In Natural Language Processing (NLP), sequence labeling is a fundamental task for identifying entities and syntactic roles. Despite the success of deep learning in sentence-level representations, a "sophistication gap" exists at the word level. Most models use shallow character-level encoders, such as a single convolutional layer or a basic RNN. However, words possess a rigid internal logic (morphology) that simple stacking of layers often fails to capture, sometimes even leading to decreased accuracy. The challenge lies in developing an architecture that respects the unique spatial properties of characters while leveraging the representational power of deep networks.

Methods: The IntNet Architecture

The core innovation, IntNet, replaces standard character-level encoders with a design guided by three principles: a funnel-shaped wide architecture, the absence of down-sampling, and dense/alternate connections.

Figure 1: The IntNet architecture featuring parallel convolutional kernels and alternate layer connections.
  1. Funnel-Shaped Wide Architecture: Inspired by Inception networks, IntNet employs multiple parallel convolutional kernels of varying sizes (3, 4, and 5) within each block. This allows the model to simultaneously detect different character n-grams, such as potential prefixes, suffixes, or roots.
  2. No Down-Sampling: Unlike standard CNNs, IntNet avoids max-pooling or striding. This preservation of sequential character order is vital for recognizing word structure, as down-sampling can destroy the precise positioning necessary for morphological understanding.
  3. Dense/Alternate Connections: To facilitate gradient flow and feature reuse, each layer has access to the feature maps of previous alternate layers. This enables the learning of increasingly complex patterns of word shape.

The resulting character-to-word representation is integrated into a standard Bi-LSTM-CRF framework for sequence labeling.

Experimental Setup and Results

The authors evaluated IntNet on six benchmarks: NER (Spanish, Dutch, English, German), Syntactic Chunking (CoNLL-2000), and POS Tagging (Penn Treebank). The model was tested in a "pure" setting, devoid of gazetteers or external knowledge bases.

Figure 5: F1 scores over training epochs for English, German, Spanish, and Dutch, comparing IntNet variants against char-LSTM and char-CNN baselines.

Key Findings

  • Performance Gains: IntNet consistently outperformed char-CNN and char-LSTM baselines. Significant improvements (over 2% F1) were noted in morphologically rich languages like German and Dutch.
  • State-of-the-Art Results: The model achieved new SOTA benchmarks for English NER (85.34 F1) and PTB POS tagging (97.23) among models not using external resources.
  • Out-of-Vocabulary (OOV) Robustness: IntNet excelled at labeling words not seen during training or in pre-trained embeddings, outperforming other models by more than 3% in some categories. This suggests the model successfully learns the "logic" of word construction.
  • Preservation of Position: By removing down-sampling, the model effectively distinguishes between identical character sequences appearing in different positions (e.g., "un-" as a prefix vs. "-un-" in the middle of a word).

Conclusion

IntNet demonstrates that depth in character-level modeling is beneficial when implemented through a wide, funnel-shaped architecture that preserves spatial information. While training is approximately 20% slower than shallow alternatives, inference time remains comparable, and the gains in accuracy—especially for OOV words and complex morphologies—are substantial. Future work may explore IntNet's utility in other sub-word tasks like machine translation or semantic similarity.

Summary updated

BibTeX
@inproceedings{xin2018learning,
  author = {Yingwei Xin and Ethan Hart and Vibhuti Mahajan and Jean-David Ruvini},
  title = {{Learning Better Internal Structure of Words for Sequence Labeling}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1279},
}
Figure