SeqGPT: An Out-of-the-Box Large Language Model for Open Domain Sequence Understanding

Tianyu Yu, Chengyue Jiang, Chao Lou, Shen Huang, Xiaobin Wang, Wei Liu, Jiong Cai, Yangning Li, Yinghui Li, Kewei Tu, Hai-Tao Zheng, Ningyu Zhang, Pengjun Xie, Fei Huang, Yong Jiang

Abstract

SeqGPT is an open-source, bilingual (English/Chinese) Large Language Model (LLM) specifically designed to master open-domain sequence understanding. By unifying diverse Natural Language Understanding (NLU) tasks into two simple "atomic" operations—extraction and classification—SeqGPT consistently outperforms much larger models like ChatGPT on zero-shot NLU benchmarks. This paper details the unified approach and the two-stage training strategy that enables such high performance in a moderately sized model.

Introduction

While general-purpose LLMs like GPT-4 are incredibly capable, they are often "too footloose" for structured NLU tasks. Tasks like Named Entity Recognition (NER), Relation Extraction (RE), and Event Extraction (EE) require specific output formats and adherence to predefined label sets. Standard LLMs often struggle with these constraints, requiring extensive and often brittle prompt engineering to produce usable results.

Furthermore, the massive size of state-of-the-art models makes them expensive to run and difficult to customize for specific domains. SeqGPT addresses these challenges by providing an "out-of-the-box" solution: a moderately sized model that is pre-specialized for the rigors of NLU across any domain.

Methods

The core innovation of SeqGPT is its Unified Approach. Instead of treating every NLU task as a unique problem, the authors translate them into two "atomic" tasks:

  1. Extraction (EXT): Identifying relevant spans of text based on a query (e.g., "Extract all person names").
  2. Classification (CLS): Associating the entire input with a subset of labels (e.g., "Classify the sentiment of this text").
Overview of the SeqGPT unified approach, mapping various NLU tasks to EXT and CLS operations.

To train the model, they employed a two-stage training strategy:

  • Pre-training (PT): They used ChatGPT to synthesize an extremely fine-grained dataset. ChatGPT was tasked with "inventing" appropriate labels and identifying answers for text from diverse domains (Wikipedia, news, medicine). This created a massive, albeit noisy, pool of over 800,000 distinct labels to boost the model's generalization.
  • Fine-tuning (FT): The model was then refined using a high-quality collection of 152 real-world NLU datasets, covering 233 different atomic tasks. The models are based on the BLOOMZ family, ranging from 560M to 7.1B parameters.

Key Results and Insights

SeqGPT demonstrates a remarkable ability to handle "unseen" domains and labels, proving its utility as a truly open-domain tool.

  • Beating the Giants: Even the smallest SeqGPT-560M model outperformed ChatGPT by a margin of 27.4 points on average across held-out NLU benchmarks. The SeqGPT-7B1 model achieved a top average score of 65.5, significantly higher than the base BLOOMZ model (6.2) and ChatGPT (38.1).
  • Scaling Success: Performance improves significantly with model size, particularly for held-out datasets, validating the generalization power of the unified approach.
Performance scaling of SeqGPT across different model sizes for both held-in and held-out datasets.
  • Diversity Over Volume: Scaling task diversity is the real driver of performance. Performance improves logarithmically as the number of distinct training tasks increases.
  • Cross-lingual Synergy: Training on both English and Chinese simultaneously improved performance in both languages, suggesting that underlying NLU skills are largely language-agnostic.

Conclusion

By reducing the complexity of NLU to just EXT and CLS, SeqGPT learns a stable "interface" for understanding language, making it much more reliable than a general-purpose chat model. While the 7B scale model still lacks the complex reasoning of much larger models for highly intricate tasks, it provides a powerful, specialized tool for open-domain sequence understanding. Future work will focus on improving synthetic data quality and investigating the diminishing returns of synthetic pre-training as base model size increases.

Summary updated

Abstract

Large language models (LLMs) have shown impressive abilities for open-domain NLP tasks. However, LLMs are sometimes too footloose for natural language understanding (NLU) tasks which always have restricted output and input format. Their performances on NLU tasks are highly related to prompts or demonstrations and are shown to be poor at performing several representative NLU tasks, such as event extraction and entity typing. To this end, we present SeqGPT, a bilingual (i.e., English and Chinese) open-source autoregressive model specially enhanced for open-domain natural language understanding. We express all NLU tasks with two atomic tasks, which define fixed instructions to restrict the input and output format but still ``open'' for arbitrarily varied label sets. The model is first instruction-tuned with extremely fine-grained labeled data synthesized by ChatGPT and then further fine-tuned by 233 different atomic tasks from 152 datasets across various domains. The experimental results show that SeqGPT has decent classification and extraction ability, and is capable of performing language understanding tasks on unseen domains. We also conduct empirical studies on the scaling of data and model size as well as on the transfer across tasks. Our models are accessible at https://github.com/Alibaba-NLP/SeqGPT.
BibTeX
@inproceedings{yu2024seqgpt,
  author = {Tianyu Yu and Chengyue Jiang and Chao Lou and Shen Huang and Xiaobin Wang and Wei Liu and Jiong Cai and Yangning Li and Yinghui Li and Kewei Tu and Hai-Tao Zheng and Ningyu Zhang and Pengjun Xie and Fei Huang and Yong Jiang},
  title = {{SeqGPT: An Out-of-the-Box Large Language Model for Open Domain Sequence Understanding}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i17.29917},
}
Figure