DeepPavlov 1.0: Your Gateway to Advanced NLP Models Backed by Transformers and Transfer Learning

Maksim Savkin, Anastasia Voznyuk, Fedor Ignatov, Anna Korzanova, Dmitry Karpov, Alexander Popov, Vasily KonovalovEMNLP 2024

TL;DR

DeepPavlov 1.0 is a new version of an open-source NLP framework that simplifies the use of advanced transformer-based models. It offers a no-code, configuration-driven approach to training, inference, and deployment, making state-of-the-art NLP accessible to a wider audience, even those without deep ML expertise.

What Problem Does This Paper Address?

Building and deploying cutting-edge Natural Language Processing (NLP) models, especially those based on complex transformer architectures, can be a significant hurdle. This is particularly true for individuals and organizations without extensive machine learning expertise. The complexity of model training, fine-tuning, and integration into software applications creates a barrier to entry. While many NLP frameworks exist, there's a need for solutions that are not only powerful but also user-friendly, maintainable, and actively supported by a community, addressing the gap in long-term, accessible NLP tools.

Methods

DeepPavlov 1.0 is built on PyTorch and integrates seamlessly with Hugging Face's transformers library, allowing it to leverage a vast array of pre-trained models. Its core innovation lies in its no-code methodology, where all operations—training, inference, and deployment—are managed through configuration files. This approach abstracts away much of the underlying ML complexity. The framework supports advanced transfer learning techniques like Sequential Transfer Learning (STL), Multi-Task Learning (MTL), and Cross-Lingual Transfer Learning (CTL), enabling the development of multilingual models and models that perform well with limited training data. It also offers a modular, configuration-based architecture that allows for easy customization of pipelines.

Datasets

The paper mentions the use of several datasets for training and evaluation:

  • HuggingFace datasets: DeepPavlov's dataset_reader supports datasets from HuggingFace.
  • NER datasets: The framework utilizes datasets like CoNLL-2003 and OntoNotes for Named Entity Recognition (NER) tasks.
  • Massive dataset: A combination of OntoNotes 5.0 and the Massive dataset was used for training the primary DeepPavlov 1.0 NER model.
  • Synthetic data: Synthetic texts were generated using OpenChat-3.5-0106 (7B) to augment NER datasets with fine-grained location-related entities.
  • DeepPavlov Topics dataset: Used for fine-tuning their unique topic classification model.
  • SemEval-2014 Task 4 dataset: Utilized as the base for training the Aspect-Based Sentiment Analysis (ABSA) model.
  • SQuAD 1.1 validation set: Used to evaluate improvements in Reading Comprehension.
  • GLUE benchmark: Used to demonstrate MTL results.

The paper highlights the creation of a custom NER dataset with 32 entities, exceeding the number of entities in many other publicly available NER datasets.

Key Insights and Ideas

  • No-Code NLP is Achievable: The most significant insight is that complex NLP tasks can be managed effectively through configuration files alone, drastically lowering the barrier to entry for non-experts.
  • Transfer Learning is Key: DeepPavlov 1.0 heavily relies on transfer learning, enabling powerful models to be built even with limited domain-specific data.
  • User-Centric Development: The framework's evolution is driven by user feedback and demand, particularly in expanding the NER model with custom entities and focusing on popular tasks like NER and sentiment analysis.
  • Balancing Performance and Usability: The authors consciously aim for models that strike an optimal balance between performance, inference speed, and resource requirements, making them practical for real-world applications rather than solely chasing state-of-the-art benchmarks.

Key Results

  • Versatile Interaction: DeepPavlov 1.0 can be interacted with via a command-line interface (CLI), REST API, or directly through Python code.
  • NER Performance: The DeepPavlov 1.0 NER model, based on DeBERTa-v3-base, achieves competitive performance on OntoNotes and CoNLL datasets, offering a more efficient alternative to larger models like XLM-ROBERTa-large.
  • Reading Comprehension Improvement: The framework improved performance on the SQuAD 1.1 validation set, achieving an exact match score of 81.49% and an F1-score of 88.86%.
  • MTL Capabilities: Multi-task learning models show comparable or even superior performance to single-task models on the GLUE benchmark.
  • Custom NER Entities: The DeepPavlov 1.0 NER demo features 32 entities, a notable increase compared to many other NER datasets.

Limitations and Future Work

Limitations:

  • Limited Model Scope: The framework currently focuses on a curated set of high-demand models, meaning tasks like Part-of-Speech (POS) tagging or Dependency parsing are not yet supported, as they were not prioritized based on user demand.
  • Fine-tuning Expertise: While DeepPavlov 1.0 simplifies model usage, fine-tuning models for specific tasks still requires a level of NLP and programming expertise.
  • Not Always SOTA: For many models, DeepPavlov does not achieve state-of-the-art performance on popular benchmarks, prioritizing usability and efficiency over absolute peak performance.

Future Work:

  • Expand NER Entities: Continue to enhance the NER model by incorporating more custom entities based on user needs.
  • Multilingual ASPE Dataset: Construct a multilingual Aspect-Based Sentiment Analysis (ASPE) dataset to broaden its applicability.
  • Community Growth: Foster the existing community through forums and interactive demos to guide future development.

Summary updated

BibTeX
@inproceedings{savkin2024deeppavlov,
  author = {Maksim Savkin and Anastasia Voznyuk and Fedor Ignatov and Anna Korzanova and Dmitry Karpov and Alexander Popov and Vasily Konovalov},
  title = {{DeepPavlov 1.0: Your Gateway to Advanced NLP Models Backed by Transformers and Transfer Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
}
Figure