SeqGPT: An Out-of-the-box Large Language Model for Open Domain Sequence Understanding

Tianyu Sun, Chengyue Jiang, Chao Lou, Shen Huang, Xiaobin Wang, Wei Liu, Jiong Cai, Yangning Li, Yinghui Li, Kewei Tu, Hai-Tao Zheng, Ningyu Zhang, Pengjun Xie, Fei Huang, Yong Jiang

Introduction

Natural Language Understanding (NLU) remains a cornerstone of artificial intelligence, encompassing tasks such as Named Entity Recognition (NER), Relation Extraction (RE), and Event Extraction (EE). While general-purpose Large Language Models (LLMs) like GPT-4 have demonstrated remarkable capabilities, they often struggle with the rigid requirements of NLU tasks, which demand adherence to specific label sets and structured output formats. SeqGPT emerges as a specialized, bilingual (English and Chinese) solution designed to bridge this gap. By unifying diverse NLU challenges into two fundamental "atomic" tasks—extraction and classification—SeqGPT offers an out-of-the-box model that significantly outperforms generalist models like ChatGPT on zero-shot NLU benchmarks.

The Challenge of Restricted NLU

Standard LLMs are often "too footloose" for structured NLU. When tasked with extracting specific entities or classifying text into predefined categories, they frequently hallucinate labels outside the provided set or fail to follow complex formatting instructions consistently. Achieving high performance typically requires extensive prompt engineering or the use of massive, computationally expensive models. SeqGPT addresses these issues by providing a moderate-sized model pre-trained to understand the underlying structure of NLU tasks across any domain, eliminating the need for task-specific fine-tuning.

Methodology: Atomic Task Decomposition

The core innovation of SeqGPT lies in its "Unified Approach," which simplifies the complex landscape of NLU into two fundamental operations:

  1. Extraction (EXT): Identifying and isolating relevant spans of text based on a specific query (e.g., "Find all 'Person' entities").
  2. Classification (CLS): Categorizing a segment of text into a subset of provided labels (e.g., determining sentiment or intent).

Complex NLU tasks are decomposed into these atomic units. For instance, Relation Extraction is handled as an initial extraction step to identify entities, followed by a classification step to define their relationship.

The SeqGPT workflow decomposes various NLU tasks into unified Extraction and Classification operations.

Training Strategy

SeqGPT utilizes a two-stage training process to achieve its open-domain capabilities:

  • Pre-training (PT): The model is first exposed to a massive synthetic dataset containing over 800,000 distinct, ultra-fine-grained labels generated by ChatGPT. This stage teaches the model the general concept of open-domain understanding.
  • Fine-tuning (FT): The model is then refined using 152 real-world NLU datasets, translated into 233 atomic tasks. This aligns the model with high-quality, human-annotated data.
  • Negative Labeling: To enhance robustness, the training includes "negative" samples—labels that do not apply to the text—forcing the model to learn precise boundaries for extraction and classification.

Experimental Setup and Datasets

The researchers evaluated SeqGPT using an extensive array of data. The fine-tuning phase covered 11 distinct NLU categories, including NER, RE, EE, Natural Language Inference (NLI), and Machine Reading Comprehension (MRC). To test true zero-shot generalization, performance was measured on 49 "held-out" datasets that the model never encountered during training.

Key Results and Scaling Insights

SeqGPT demonstrates a clear advantage over generalist models in structured understanding. The SeqGPT-7B1 variant achieved an average score of 65.5 across all tasks, compared to ChatGPT’s score of 38.1. Notably, even the smallest 560M parameter version of SeqGPT outperformed the 175B+ parameter ChatGPT on several benchmarks, suggesting that for structured understanding, targeted training is more efficient than raw scale.

A critical finding of the study is the importance of task diversity over raw data volume. Performance gains were found to follow a logarithmic trend relative to the number of distinct training tasks. Furthermore, the model exhibits strong scaling properties; as both the model size and the number of training datasets increase, performance on both held-in and held-out benchmarks improves consistently.

Performance scaling of SeqGPT relative to model size (left) and the number of training datasets (right).

Human evaluations further support these quantitative results. In blind tests, SeqGPT-7B1 was preferred over ChatGPT in 7 out of 10 NLU categories. Annotators specifically noted that SeqGPT's outputs are more concise and easier to integrate into downstream engineering pipelines compared to the often verbose responses of ChatGPT.

Conclusion and Future Directions

SeqGPT proves that targeted training on unified atomic tasks can allow moderate-sized models to outperform massive generalist LLMs in structured NLU. Its bilingual synergy and "out-of-the-box" utility make it a powerful tool for open-domain sequence understanding.

However, limitations remain. While SeqGPT excels at extraction and classification, it still lags behind larger models in tasks requiring multi-step logical reasoning. Future work will focus on improving reasoning capabilities and further expanding task diversity to push the boundaries of zero-shot NLU performance.

Summary updated

Abstract

Large language models (LLMs) have shown impressive ability for open-domain NLP tasks. However, LLMs are sometimes too footloose for natural language understanding (NLU) tasks which always have restricted output and input format. Their performances on NLU tasks are highly related to prompts or demonstrations and are shown to be poor at performing several representative NLU tasks, such as event extraction and entity typing. To this end, we present SeqGPT, a bilingual (i.e., English and Chinese) open-source autoregressive model specially enhanced for open-domain natural language understanding. We express all NLU tasks with two atomic tasks, which define fixed instructions to restrict the input and output format but still ``open'' for arbitrarily varied label sets. The model is first instruction-tuned with extremely fine-grained labeled data synthesized by ChatGPT and then further fine-tuned by 233 different atomic tasks from 152 datasets across various domains. The experimental results show that SeqGPT has decent classification and extraction ability, and is capable of performing language understanding tasks on unseen domains. We also conduct empirical studies on the scaling of data and model size as well as on the transfer across tasks. Our model is accessible at this https URL.
BibTeX
@inproceedings{sun2024seqgpt,
  author = {Tianyu Sun and Chengyue Jiang and Chao Lou and Shen Huang and Xiaobin Wang and Wei Liu and Jiong Cai and Yangning Li and Yinghui Li and Kewei Tu and Hai-Tao Zheng and Ningyu Zhang and Pengjun Xie and Fei Huang and Yong Jiang},
  title = {{SeqGPT: An Out-of-the-box Large Language Model for Open Domain Sequence Understanding}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i17.29917},
}
Figure