CFO: A Framework for Building Production NLP Systems

Rishav Chakravarti, Cezar Pendus, Andrzej Sakrajda, Anthony Ferritto, Lin Pan, Michael R. Glass, Vittorio Castelli, J. William Murdock, Radu Florian, Salim Roukos, Avirup Sil

Abstract: Rishav Chakravarti, Cezar Pendus, Andrzej Sakrajda, Anthony Ferritto, Lin Pan, Michael Glass, Vittorio Castelli, J. William Murdock, Radu Florian, Salim Roukos, Avi Sil. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations.

2019.

Introduction

Moving an NLP model from a research environment like a Jupyter notebook to a production-grade deployment is a notoriously difficult engineering challenge. Real-world applications rarely rely on a single isolated model; instead, they require a complex "flow" of multiple components, including document retrieval, span extraction, de-duplication, and ranking.

Existing toolkits often suffer from language lock-in (typically Python-centric) or brittle orchestration logic that requires manual, hard-coded data transformations. The Computation Flow Orchestrator (CFO) is designed to bridge this gap, providing a language-agnostic, scalable framework to manage complex computation flow graphs.

The CFO Framework and GAAMA

CFO utilizes Google Protocol Buffers (protobuf) to define declarative interfaces for every node in a system. This approach ensures compile-time safety, preventing the type mismatches common in JSON-based systems. By defining these interfaces, the framework can auto-generate the extensive "glue code"—nearly 3,000 lines of C++ in the authors' demonstration—required for connectivity, serialization, and error handling.

Architecturally, each component in the flow is treated as a containerized gRPC microservice. This allows disparate technologies to work together seamlessly; for example, a BERT model running in PyTorch can communicate with an Elasticsearch node or a custom C++ post-processing script.

To demonstrate the framework's utility, the authors developed GAAMA (Go Ahead, Ask Me Anything), an end-to-end Question Answering (QA) system. As shown in the architecture below, GAAMA orchestrates the flow from initial question to final answer snippet.

The GAAMA question-answering system architecture built using the CFO framework.

Technical Innovations: U-MRC

A key technical highlight of the system is the U-MRC (Unsupervised Machine Reading Comprehension) pre-training strategy. Rather than relying solely on standard BERT pre-training, the model is further trained on "cloze-style" queries by masking entities in Wikipedia sentences. This aligns the pre-training task more closely with the ultimate goal of answer extraction, proving more effective than generic Masked Language Modeling.

Evaluation and Performance

The system was evaluated using landmark benchmarks including SQuAD 1.1 & 2.0 and Natural Questions (NQ). The authors emphasize NQ because it consists of real Google search queries, avoiding the observational bias found in crowd-sourced datasets.

Key results include:

  • State-of-the-Art Accuracy: The BERT-based MRC model achieved a 54.5 F1 score on the Natural Questions dev set, outperforming contemporary systems.
  • Automatic Parallelization: The CFO orchestrator automatically handles asynchronous calls, allowing multiple candidate documents to be processed by the MRC node in parallel without custom multi-threading code.

Production Realities: Latency vs. Accuracy

While BERT-Large provides a significant boost in accuracy over BERT-Base (an 8.3% F1 increase), the authors provide a sobering look at the associated computational costs. In a production environment, BERT-Large's latency on a CPU can exceed 6 seconds for 95th percentile queries, making GPU acceleration a practical requirement for real-time deployment.

Performance and latency trade-offs between BERT-Base and BERT-Large models.

Conclusion

CFO simplifies the transition from research to production by automating the orchestration of microservices. While it does not inherently reduce the computational cost of large models, it provides the infrastructure necessary to manage them at scale. Future work will focus on model compression techniques like knowledge distillation and improving the initial retrieval phase to provide better context for the MRC nodes.

Summary updated

BibTeX
@inproceedings{chakravarti2019cfo,
  author = {Rishav Chakravarti and Cezar Pendus and Andrzej Sakrajda and Anthony Ferritto and Lin Pan and Michael R. Glass and Vittorio Castelli and J. William Murdock and Radu Florian and Salim Roukos and Avirup Sil},
  title = {{CFO: A Framework for Building Production NLP Systems}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=RmaBs1Fjrv},
}
Figure