Abstract
Sequence labeling tasks like Named Entity Recognition (NER) and Part-of-Speech (POS) tagging often rely on word-level representations. While sentence-level modeling has advanced significantly with deep architectures, word-level models—specifically those capturing character-level internal structures—have remained relatively shallow. This paper introduces IntNet, a deep, wide, and "funnel-shaped" convolutional architecture designed to capture morphological features like prefixes and suffixes without down-sampling. By preserving character order and utilizing dense connections, IntNet achieves state-of-the-art results on multiple benchmarks without external resources, demonstrating particular strength in morphologically rich languages and out-of-vocabulary word handling.
Introduction
In Natural Language Processing (NLP), sequence labeling is a fundamental task for identifying entities and syntactic roles. Despite the success of deep learning in sentence-level representations, a "sophistication gap" exists at the word level. Most models use shallow character-level encoders, such as a single convolutional layer or a basic RNN. However, words possess a rigid internal logic (morphology) that simple stacking of layers often fails to capture, sometimes even leading to decreased accuracy. The challenge lies in developing an architecture that respects the unique spatial properties of characters while leveraging the representational power of deep networks.
Methods: The IntNet Architecture
The core innovation, IntNet, replaces standard character-level encoders with a design guided by three principles: a funnel-shaped wide architecture, the absence of down-sampling, and dense/alternate connections.
- Funnel-Shaped Wide Architecture: Inspired by Inception networks, IntNet employs multiple parallel convolutional kernels of varying sizes (3, 4, and 5) within each block. This allows the model to simultaneously detect different character n-grams, such as potential prefixes, suffixes, or roots.
- No Down-Sampling: Unlike standard CNNs, IntNet avoids max-pooling or striding. This preservation of sequential character order is vital for recognizing word structure, as down-sampling can destroy the precise positioning necessary for morphological understanding.
- Dense/Alternate Connections: To facilitate gradient flow and feature reuse, each layer has access to the feature maps of previous alternate layers. This enables the learning of increasingly complex patterns of word shape.
The resulting character-to-word representation is integrated into a standard Bi-LSTM-CRF framework for sequence labeling.
Experimental Setup and Results
The authors evaluated IntNet on six benchmarks: NER (Spanish, Dutch, English, German), Syntactic Chunking (CoNLL-2000), and POS Tagging (Penn Treebank). The model was tested in a "pure" setting, devoid of gazetteers or external knowledge bases.
Key Findings
- Performance Gains: IntNet consistently outperformed char-CNN and char-LSTM baselines. Significant improvements (over 2% F1) were noted in morphologically rich languages like German and Dutch.
- State-of-the-Art Results: The model achieved new SOTA benchmarks for English NER (85.34 F1) and PTB POS tagging (97.23) among models not using external resources.
- Out-of-Vocabulary (OOV) Robustness: IntNet excelled at labeling words not seen during training or in pre-trained embeddings, outperforming other models by more than 3% in some categories. This suggests the model successfully learns the "logic" of word construction.
- Preservation of Position: By removing down-sampling, the model effectively distinguishes between identical character sequences appearing in different positions (e.g., "un-" as a prefix vs. "-un-" in the middle of a word).
Conclusion
IntNet demonstrates that depth in character-level modeling is beneficial when implemented through a wide, funnel-shaped architecture that preserves spatial information. While training is approximately 20% slower than shallow alternatives, inference time remains comparable, and the gains in accuracy—especially for OOV words and complex morphologies—are substantial. Future work may explore IntNet's utility in other sub-word tasks like machine translation or semantic similarity.
Summary updated