GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing

Haibin Lin

Introduction: The Reproducibility and Deployment Gap

In the rapidly evolving field of deep learning, researchers and engineers face two persistent challenges: reproducibility and deployment. Published papers often omit "minor refinements"—such as specific data augmentation techniques or hyperparameter schedules—that are critical for achieving state-of-the-art (SOTA) results. Furthermore, while Python is the standard for research, its performance limitations often necessitate a complete rewrite of models for production environments like C++ or mobile backends.

GluonCV and GluonNLP are high-performance, open-source toolkits built on Apache MXNet designed to bridge these gaps. They provide modular building blocks and a "model zoo" of over 200 pre-trained models, accompanied by the exact training scripts and logs used to create them.

The Gluon Architecture: Hybridizing for Performance

The toolkits leverage the Gluon API, which offers an imperative programming style similar to PyTorch, making it intuitive for debugging and research. However, it uniquely features a hybridizing mechanism that allows Python code to be converted into a symbolic graph. This enables high-performance execution and seamless deployment into non-Python environments such as C++, Java, Scala, and Julia.

Key technical features include:

  • Modular APIs: Decoupled components for data processing, model architecture, and loss functions allow for flexible experimentation.
  • Efficient Data Pipelines: Utilities like "fixed bucketing" group similar input shapes (e.g., sentence lengths) into mini-batches, minimizing wasted computation on padding.
  • Quantization Support: The toolkits support int8 and float16 inference, significantly speeding up models on hardware like Intel CPUs with negligible accuracy loss.

Modular Design and the "Bag of Tricks"

A standout feature of these toolkits is the inclusion of a "Bag of Tricks"—documented training refinements that often allow Gluon implementations to outperform the original authors' results. For instance, the Gluon implementation of ResNet-50 achieves 79.2% accuracy on ImageNet, nearly 4% higher than the original Caffe version. By documenting these refinements, the authors provide a roadmap for others to achieve similar performance gains.

Empirical Evaluation and the Model Zoo

The authors evaluated the toolkits across a wide range of benchmarks, including ImageNet, COCO, and Kinetics400 for vision, and SQuAD, MNLI, and MRPC for NLP. The results demonstrate that Gluon implementations consistently match or exceed other open-source implementations (OOSI).

Performance comparison of GluonCV/NLP against other open-source implementations (OOSI) across various CV and NLP tasks.

Beyond raw accuracy, the toolkits focus on efficiency. For example, using int8 quantization reduced BERT-base latency by 59.6% on Intel hardware. The model zoo provides a clear view of the trade-offs between accuracy and throughput, allowing engineers to select the optimal model for their specific hardware constraints.

Accuracy vs. throughput (#Samples/sec) for various image classification models, illustrating the Pareto frontier.

Conclusion and Future Directions

GluonCV and GluonNLP serve as both production-ready tools and educational resources, the latter through their integration with the interactive book Dive into Deep Learning. Future work aims to enrich the API with new SOTA models, improve NumPy compatibility for broader data science integration, and expand support for diverse edge deployment scenarios.

Summary updated

Abstract

We present GluonCV and GluonNLP, the deep learning toolkits for computer vision and natural language processing based on Apache MXNet (incubating). These toolkits provide state-of-the-art pre-trained models, training scripts, and training logs, to facilitate rapid prototyping and promote reproducible research. We also provide modular APIs with flexible building blocks to enable efficient customization. Leveraging the MXNet ecosystem, the deep learning models in GluonCV and GluonNLP can be deployed onto a variety of platforms with different programming languages. The Apache 2.0 license has been adopted by GluonCV and GluonNLP to allow for software distribution, modification, and usage.
BibTeX
@article{lin2020gluoncv,
  author = {Haibin Lin},
  title = {{GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing}},
  journal = {J. Mach. Learn. Res.},
  year = {2020},
  url = {https://openreview.net/forum?id=4xhIX_SD3w},
}
Figure