Introduction
Modern Large Language Models (LLMs) demonstrate remarkable conversational prowess but often struggle with structured Natural Language Understanding (NLU) tasks, such as Named Entity Recognition (NER), relationship extraction, and text classification. While supervised fine-tuning (SFT) is a common remedy, existing NLU datasets frequently suffer from task imbalance, structural rigidity (e.g., over-reliance on JSON), and the "specialization tax"—where fine-tuning on narrow tasks degrades general reasoning and coding capabilities. To address these challenges, researchers from Ant Group developed Hum, a massive synthetic instruction corpus of 2.8 million entries designed to enhance NLU without sacrificing general-purpose utility.
The Hum Synthesis Framework
The core contribution of this work is the Hum framework, a human-LLM collaborative mechanism that moves beyond simple data conversion to "compound synthesis." This approach ensures that models learn underlying semantics rather than just structural patterns.
The synthesis process consists of four primary stages:
- Basic Instruction Synthesis: Converting raw NLU data into standard instruction-schema-input-output formats.
- Guidelines Synthesis: Incorporating semantic descriptions and both positive and negative examples to teach the model the logic behind correct extractions.
- Preference Rules Synthesis: Utilizing GPT-4 to generate fine-grained, conditional rules for the same text (e.g., "extract only the highest degree" vs. "extract all degrees"), forcing the model to follow specific instructions rather than memorizing templates.
- Format Variants Synthesis: Training the model to output results in diverse formats—including JSON, Markdown tables, and plain text—to ensure robustness against structural changes.
Dataset and Evaluation
The resulting Hum dataset comprises over 2.8 million instructions. The distribution is diverse, covering Relation Extraction (29%), NER (23%), SPO extraction (11%), and smaller portions of Event Extraction, Machine Reading Comprehension (MRC), and Text Classification. Crucially, 8% of the dataset consists of "Instruction Generalist" (IG) tasks to preserve the model's conversational capabilities.
Evaluation was conducted across five specialized NLU benchmarks: CrossNER, FewRel, CCF Law, C3, and IMDB. To monitor general capabilities, the researchers also tested the models on 28 general datasets via the OpenCompass framework.
Results and Performance Gains
Fine-tuning various base models (including Qwen2, Llama3, and Baichuan2) on the Hum corpus yielded significant improvements. On average, NLU performance increased by 3.1% across six different LLMs.
Key findings from the results include:
- Superiority over Specialized Models: Hum-tuned models consistently outperformed existing specialized NLU models like OneKE and YAYI-UIE.
- Competitive with GPT-4: In several zero-shot scenarios, Hum-tuned models achieved higher scores than GPT-4, particularly in MRC (C3) and classification (IMDB) tasks.
- Mitigating the Specialization Tax: General benchmark testing confirmed that the Hum-tuned models maintained their performance in coding, math, and general reasoning, successfully avoiding the catastrophic forgetting typically associated with narrow fine-tuning.
Conclusion
The Hum framework demonstrates that large-scale, diverse, and logically complex synthetic data can bridge the gap between general-purpose LLMs and specialized NLU requirements. By prioritizing "logic over layout" through preference rules and format diversification, the researchers have created a path for developing models that are both highly specialized and broadly capable. Future work will focus on expanding the variety of data types to further refine the balance between NLU precision and general utility.
Summary updated