Abstract
SeqGPT is an open-source, bilingual (English/Chinese) Large Language Model (LLM) specifically designed to master open-domain sequence understanding. By unifying diverse Natural Language Understanding (NLU) tasks into two simple "atomic" operations—extraction and classification—SeqGPT consistently outperforms much larger models like ChatGPT on zero-shot NLU benchmarks. This paper details the unified approach and the two-stage training strategy that enables such high performance in a moderately sized model.
Introduction
While general-purpose LLMs like GPT-4 are incredibly capable, they are often "too footloose" for structured NLU tasks. Tasks like Named Entity Recognition (NER), Relation Extraction (RE), and Event Extraction (EE) require specific output formats and adherence to predefined label sets. Standard LLMs often struggle with these constraints, requiring extensive and often brittle prompt engineering to produce usable results.
Furthermore, the massive size of state-of-the-art models makes them expensive to run and difficult to customize for specific domains. SeqGPT addresses these challenges by providing an "out-of-the-box" solution: a moderately sized model that is pre-specialized for the rigors of NLU across any domain.
Methods
The core innovation of SeqGPT is its Unified Approach. Instead of treating every NLU task as a unique problem, the authors translate them into two "atomic" tasks:
- Extraction (EXT): Identifying relevant spans of text based on a query (e.g., "Extract all person names").
- Classification (CLS): Associating the entire input with a subset of labels (e.g., "Classify the sentiment of this text").
To train the model, they employed a two-stage training strategy:
- Pre-training (PT): They used ChatGPT to synthesize an extremely fine-grained dataset. ChatGPT was tasked with "inventing" appropriate labels and identifying answers for text from diverse domains (Wikipedia, news, medicine). This created a massive, albeit noisy, pool of over 800,000 distinct labels to boost the model's generalization.
- Fine-tuning (FT): The model was then refined using a high-quality collection of 152 real-world NLU datasets, covering 233 different atomic tasks. The models are based on the BLOOMZ family, ranging from 560M to 7.1B parameters.
Key Results and Insights
SeqGPT demonstrates a remarkable ability to handle "unseen" domains and labels, proving its utility as a truly open-domain tool.
- Beating the Giants: Even the smallest SeqGPT-560M model outperformed ChatGPT by a margin of 27.4 points on average across held-out NLU benchmarks. The SeqGPT-7B1 model achieved a top average score of 65.5, significantly higher than the base BLOOMZ model (6.2) and ChatGPT (38.1).
- Scaling Success: Performance improves significantly with model size, particularly for held-out datasets, validating the generalization power of the unified approach.
- Diversity Over Volume: Scaling task diversity is the real driver of performance. Performance improves logarithmically as the number of distinct training tasks increases.
- Cross-lingual Synergy: Training on both English and Chinese simultaneously improved performance in both languages, suggesting that underlying NLU skills are largely language-agnostic.
Conclusion
By reducing the complexity of NLU to just EXT and CLS, SeqGPT learns a stable "interface" for understanding language, making it much more reliable than a general-purpose chat model. While the 7B scale model still lacks the complex reasoning of much larger models for highly intricate tasks, it provides a powerful, specialized tool for open-domain sequence understanding. Future work will focus on improving synthetic data quality and investigating the diminishing returns of synthetic pre-training as base model size increases.
Summary updated