Representing the Unification of Text Featurization using a Context-Free Grammar

Doruk Kilitçioglu, Serdar KadiogluAAAI 2021 · DOI

Abstract

In modern machine learning, text featurization is often an ad-hoc process involving fragmented libraries and complex "glue code." This paper presents TextWiser, an open-source library that unifies text featurization through a formal Context-Free Grammar (CFG). By defining featurization pipelines as "sentences" in a formal language, TextWiser allows practitioners to combine disparate methods—from TF-IDF to BERT—into complex, reproducible, and fine-tunable pipelines via a simple JSON schema. This framework bridges the gap between research experimentation and production deployment by providing a formal contract for text representations.

Introduction

The challenge of turning text into numerical vectors (featurization) is compounded by the lack of a "silver bullet" method. Data scientists must navigate a fragmented ecosystem of libraries like NLTK, Gensim, and HuggingFace, each with its own API and data structures. This leads to significant engineering overhead, maintenance difficulties, and reproducibility issues when moving from research notebooks to production environments. TextWiser addresses these challenges by providing a unified framework for pipeline construction, treating embedding pipelines as structured objects governed by a formal grammar rather than ad-hoc scripts.

The Grammar of Embeddings

The core innovation of TextWiser is the Grammar of Embeddings. The authors define a CFG where terminals represent specific featurization methods and productions define how these methods are transformed or concatenated. This formalization allows for the creation of "Compound Embeddings" that can be easily shared and reproduced.

Supported embeddings in TextWiser, highlighting pre-trained availability and fine-tuning support.

As shown in the table of supported embeddings, the library integrates a vast array of models, ranging from traditional Bag of Words to state-of-the-art Transformer architectures. The grammar enables two primary operations:

  • Transformations: Chaining an embedding with secondary processes like SVD or NMF.
  • Concatenations: Merging multiple embedding types into a single feature vector.
Transformations supported by TextWiser and their compatibility with gradient-based fine-tuning.

Crucially, many of these transformations and embeddings support gradients, as detailed in the figures above. This allows for end-to-end fine-tuning of the entire featurization pipeline for specific downstream tasks, ensuring that the representations are optimized for the problem at hand. By representing a pipeline as a JSON schema, TextWiser provides a formal "contract" that guarantees identical featurization across different environments.

Implementation and Industrial Application

TextWiser implements this grammar using a PyTorch-based backend, providing a Scikit-learn-style fit_transform interface regardless of the underlying model's complexity. The authors demonstrated the framework's utility through EaSe (Embeddings-as-a-Service), an internal deployment at Fidelity Investments. This service provides pre-computed and on-demand embeddings for various business units. In production tests on a V100 GPU, the system achieved a throughput of 766 sequences per second using a BERT-base model, demonstrating that the framework is performant enough for industrial-scale applications.

Conclusion and Future Work

By treating featurization as a formal language, TextWiser moves NLP pipeline design from ad-hoc scripting to systematic engineering. This approach not only improves reproducibility but also paves the way for automated search (AutoML) to discover optimal text representations. Future work aims to leverage this grammar to automatically build embeddings from components, effectively performing architecture search for text features. The library's ability to unify over 25 embedding types and 100+ pre-trained models makes it a powerful tool for both research and industrial NLP applications.

Summary updated

BibTeX
@inproceedings{kilitcioglu2021representing,
  author = {Doruk Kilitçioglu and Serdar Kadioglu},
  title = {{Representing the Unification of Text Featurization using a Context-Free Grammar}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i17.17814},
}
Figure