Quadratic Features and Deep Architectures for Chunking

Joseph P. Turian, James Bergstra, Yoshua Bengio

Introduction

Syntactic chunking—the process of identifying and labeling segments of text such as noun phrases or verb phrases—is a fundamental task in Natural Language Processing (NLP). Traditionally, performance gains in chunking have been driven by three main avenues: intensive manual feature engineering, sophisticated inference techniques (like Conditional Random Fields), or increasing model complexity.

This research explores a different path, suggesting that many NLP models suffer from "underfitting" linguistic data. By borrowing architectural concepts from computer vision—specifically deep hierarchies and quadratic feature extractors—the authors demonstrate that increasing a model's internal capacity can significantly reduce the need for laborious hand-crafted features, even when making independent label predictions.

The I-T-W-O Architecture

The core of the proposed approach is a neural architecture termed I-T-W-O (Input-Token-Window-Output). This model processes text through a hierarchical structure designed to capture increasingly abstract representations of the input.

The I-T-W-O architecture diagram showing the flow from input features to token and window layers, ending in a 23-class output.

As shown in the architecture diagram, the model utilizes a sliding window of seven tokens to predict the label of the central token. The process begins with an input layer (204 dimensions) consisting of word embeddings, POS tags, and orthographic features. This feeds into a "token layer" (150 units) and subsequently a "window layer" (400 units) before reaching the final output layer. This depth allows the model to learn a hierarchy of features, moving from individual token representations to wider contextual patterns.

Quadratic Filters: Bio-Inspired Feature Extraction

The "secret sauce" of this architecture is the inclusion of quadratic filters. While standard neural networks typically use linear filters followed by a non-linear activation function ($f(x) = \sigma(Wx + b)$), the authors introduce a quadratic term: $f(x) = \sigma(Wx + b + \sum(V_j x)^2)$.

These filters are inspired by "complex cells" in the V1 area of the visual cortex, which are known to respond to specific patterns regardless of exact position. In an NLP context, these filters allow the model to automatically learn complex interactions between features—such as the coincidence of a specific POS tag and a particular word embedding—without requiring the researcher to define these interactions manually.

Performance comparison showing how increasing the number of quadratic filters improves F1 scores across different model depths.

As illustrated in the performance plot, adding even a small number of quadratic filters leads to a sharp increase in accuracy across various architectures. Notably, a simple logistic regression model (I-O) enhanced with 16 quadratic filters was found to be nearly as effective as the authors' entire suite of manual feature engineering, highlighting the power of automated feature interaction learning.

Experimental Results and Insights

The models were evaluated using the standard CoNLL-2000 shared task dataset. Key findings include:

  • Depth and Capacity: Deeper architectures consistently outperformed shallower ones. The best-performing deep model (I-T-W-W-O) achieved a competitive F1 score of 93.81 on the test set.
  • Competitive Simplicity: Despite predicting each label independently (without a CRF or Viterbi decoding to model label dependencies), the high-capacity feature extractor allowed the model to remain competitive with more complex sequence-modeling systems.
  • Feature Importance: Ablation studies revealed that Part-of-Speech (POS) tags and pre-trained word embeddings were the most critical inputs. Removing these features resulted in significant performance drops, whereas orthographic features had a more marginal impact.

Conclusion

This work demonstrates that architectural innovations from computer vision can be successfully adapted to NLP. By utilizing deep hierarchies and quadratic filters, researchers can build models that "learn to see" patterns in text more effectively. While the current model treats labels independently, future work combining these powerful feature extractors with sequence-aware inference techniques like CRFs holds the potential for even greater state-of-the-art performance in syntactic labeling tasks.

Summary updated

BibTeX
@inproceedings{turian2009quadratic,
  author = {Joseph P. Turian and James Bergstra and Yoshua Bengio},
  title = {{Quadratic Features and Deep Architectures for Chunking}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
}
Figure