CFO: A Framework for Building Production NLP Systems

Rishav Chakravarti, Cezar Pendus, Andrzej Sakrajda, Anthony Ferritto, Lin Pan, Michael R. Glass, Vittorio Castelli, J. William Murdock, Radu Florian, Salim Roukos, Avirup Sil

Abstract

This paper introduces a novel orchestration framework, called CFO (COMPUTATION FLOW ORCHESTRATOR), for building, experimenting with, and deploying interactive NLP (Natural Language Processing) and IR (Information Retrieval) systems to production environments. We then demonstrate a question answering system built using this framework which incorporates state-of-the-art BERT based MRC (Machine Reading Comprehension) with IR components to enable end-to-end answer retrieval. Results from the demo system are shown to be high quality in both academic and industry domain specific settings. Finally, we discuss best practices when (pre-)training BERT based MRC models for production systems.

Introduction

Moving an NLP system from a research notebook to a production environment is notoriously difficult. Researchers often deal with "monolithic" scripts, while production systems require modularity, low latency, and language flexibility. Existing tools often fall into two traps: they are either too rigid (tied to specific libraries) or overly complex.

The Computation Flow Orchestrator (CFO) is a new framework designed to bridge this gap. By using a microservices architecture powered by gRPC and Protocol Buffers, CFO allows developers to "stitch together" disparate components—like an Elasticsearch retriever and a PyTorch-based BERT model—into a scalable, language-agnostic pipeline.

The CFO Framework

The core of the paper is the CFO framework, which treats an NLP pipeline as a directed graph of computation nodes.

  • Interface-First Design: Every node defines its inputs and outputs using Google’s Protocol Buffers (Protobuf). This ensures type safety and high-speed serialization, which is much faster than standard JSON/XML.
  • Microservices Architecture: Each component runs as a containerized gRPC microservice. This allows a Java-based search engine to talk seamlessly to a Python-based deep learning model.
  • Auto-Generated Orchestration: CFO includes a compiler that takes a graph specification and auto-generates the "glue code" (in C++) to manage data flow, logging, and parallelization.

Case Study: The GAAMA System

To prove the framework works, the authors built GAAMA (Go Ahead, Ask Me Anything), an end-to-end Question Answering (QA) system.

The GAAMA QA system architecture built on the CFO framework.

As shown in Figure 1, GAAMA orchestrates the flow from a user's question through document retrieval in Elasticsearch to answer extraction via BERT. Because the interfaces are standardized, developers can swap components—like replacing a traditional retriever with a modern Neural IR retriever—without changing the orchestration code.

Datasets and Insights

The authors evaluated their system using standard benchmarks like SQuAD 1.1 & 2.0 and Natural Questions (NQ). They highlight NQ as superior for production testing because the questions are "naturally" generated by users rather than being written by annotators who have already seen the answer.

A key technical insight is the "U-MRC" boost. By training BERT on an unsupervised auxiliary task (extracting "cloze-style" masked terms from Wikipedia), the authors improved the model's final performance on the NQ task by 1.5%.

Key Results

At the time of writing, the BERT-based MRC model used in GAAMA was the top-performing submission on the Natural Questions public leaderboard.

Performance comparison on the Natural Questions (NQ) dataset.

As seen in Figure 4, combining U-MRC pre-training with data augmentation (using SQuAD 1.1 and 2.0) pushed their NQ F1 score to 54.5, a significant jump over the standard BERT baseline of 50.2.

Production Reality Check: Latency

While BERT-Large offers a massive 8.3% absolute F1 improvement over BERT-Base, it effectively requires a GPU for production. On a CPU, the 95th percentile latency for BERT-Large was 6 seconds—far too slow for an interactive system. However, on a GPU (Nvidia Tesla P100), the system achieved a 95th percentile query latency of less than one second.

Limitations and Future Work

While powerful, the heavy reliance on GPUs for BERT-Large is a cost and infrastructure barrier. The authors suggest that future work should focus on network pruning and knowledge distillation to shrink these models for faster CPU deployment. Additionally, they plan to explore more complex features like query expansion to improve the initial document retrieval phase.

Summary updated

Abstract

This paper introduces a novel orchestration framework, called CFO (COMPUTATION FLOW ORCHESTRATOR), for building, experimenting with, and deploying interactive NLP (Natural Language Processing) and IR (Information Retrieval) systems to production environments. We then demonstrate a question answering system built using this framework which incorporates state-of-the-art BERT based MRC (Machine Reading Comprehension) with IR components to enable end-to-end answer retrieval. Results from the demo system are shown to be high quality in both academic and industry domain specific settings. Finally, we discuss best practices when (pre-)training BERT based MRC models for production systems.
BibTeX
@inproceedings{chakravarti2019cfo,
  author = {Rishav Chakravarti and Cezar Pendus and Andrzej Sakrajda and Anthony Ferritto and Lin Pan and Michael R. Glass and Vittorio Castelli and J. William Murdock and Radu Florian and Salim Roukos and Avirup Sil},
  title = {{CFO: A Framework for Building Production NLP Systems}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=RmaBs1Fjrv},
}
Figure