Improving Natural Language Understanding for LLMs via Large-Scale Instruction Synthesis

Lin Yuan, Jun Xu, Honghao Gui, Mengshu Sun, Zhiqiang Zhang, Lei Liang, Jun Zhou

Introduction

Modern Large Language Models (LLMs) demonstrate remarkable conversational prowess but often struggle with structured Natural Language Understanding (NLU) tasks, such as Named Entity Recognition (NER), relationship extraction, and text classification. While supervised fine-tuning (SFT) is a common remedy, existing NLU datasets frequently suffer from task imbalance, structural rigidity (e.g., over-reliance on JSON), and the "specialization tax"—where fine-tuning on narrow tasks degrades general reasoning and coding capabilities. To address these challenges, researchers from Ant Group developed Hum, a massive synthetic instruction corpus of 2.8 million entries designed to enhance NLU without sacrificing general-purpose utility.

The Hum Synthesis Framework

The core contribution of this work is the Hum framework, a human-LLM collaborative mechanism that moves beyond simple data conversion to "compound synthesis." This approach ensures that models learn underlying semantics rather than just structural patterns.

The Hum Instruction Synthesis Framework, illustrating the transition from basic to compound synthesis across four stages.

The synthesis process consists of four primary stages:

  1. Basic Instruction Synthesis: Converting raw NLU data into standard instruction-schema-input-output formats.
  2. Guidelines Synthesis: Incorporating semantic descriptions and both positive and negative examples to teach the model the logic behind correct extractions.
  3. Preference Rules Synthesis: Utilizing GPT-4 to generate fine-grained, conditional rules for the same text (e.g., "extract only the highest degree" vs. "extract all degrees"), forcing the model to follow specific instructions rather than memorizing templates.
  4. Format Variants Synthesis: Training the model to output results in diverse formats—including JSON, Markdown tables, and plain text—to ensure robustness against structural changes.

Dataset and Evaluation

The resulting Hum dataset comprises over 2.8 million instructions. The distribution is diverse, covering Relation Extraction (29%), NER (23%), SPO extraction (11%), and smaller portions of Event Extraction, Machine Reading Comprehension (MRC), and Text Classification. Crucially, 8% of the dataset consists of "Instruction Generalist" (IG) tasks to preserve the model's conversational capabilities.

Evaluation was conducted across five specialized NLU benchmarks: CrossNER, FewRel, CCF Law, C3, and IMDB. To monitor general capabilities, the researchers also tested the models on 28 general datasets via the OpenCompass framework.

Results and Performance Gains

Fine-tuning various base models (including Qwen2, Llama3, and Baichuan2) on the Hum corpus yielded significant improvements. On average, NLU performance increased by 3.1% across six different LLMs.

Performance comparison of base LLMs versus their Hum-tuned counterparts across five NLU benchmarks.

Key findings from the results include:

  • Superiority over Specialized Models: Hum-tuned models consistently outperformed existing specialized NLU models like OneKE and YAYI-UIE.
  • Competitive with GPT-4: In several zero-shot scenarios, Hum-tuned models achieved higher scores than GPT-4, particularly in MRC (C3) and classification (IMDB) tasks.
  • Mitigating the Specialization Tax: General benchmark testing confirmed that the Hum-tuned models maintained their performance in coding, math, and general reasoning, successfully avoiding the catastrophic forgetting typically associated with narrow fine-tuning.

Conclusion

The Hum framework demonstrates that large-scale, diverse, and logically complex synthetic data can bridge the gap between general-purpose LLMs and specialized NLU requirements. By prioritizing "logic over layout" through preference rules and format diversification, the researchers have created a path for developing models that are both highly specialized and broadly capable. Future work will focus on expanding the variety of data types to further refine the balance between NLU precision and general utility.

Summary updated

Abstract

High-quality, large-scale instructions are crucial for aligning large language models (LLMs), however, there is a severe shortage of instruction in the field of natural language understanding (NLU). Previous works on constructing NLU instructions mainly focus on information extraction (IE), neglecting tasks such as machine reading comprehension, question answering, and text classification. Furthermore, the lack of diversity in the data has led to a decreased generalization ability of trained LLMs in other NLU tasks and a noticeable decline in the fundamental model's general capabilities. To address this issue, we propose Hum, a large-scale, high-quality synthetic instruction corpus for NLU tasks, designed to enhance the NLU capabilities of LLMs. Specifically, Hum includes IE (either close IE or open IE), machine reading comprehension, text classification, and instruction generalist tasks, thereby enriching task diversity. Additionally, we introduce a human-LLMs collaborative mechanism to synthesize instructions, which enriches instruction diversity by incorporating guidelines, preference rules, and format variants. We conduct extensive experiments on 5 NLU tasks and 28 general capability evaluation datasets for LLMs. Experimental results show that Hum enhances the NLU capabilities of six LLMs by an average of 3.1\%, with no significant decline observed in other general capabilities.
BibTeX
@inproceedings{yuan2025improving,
  author = {Lin Yuan and Jun Xu and Honghao Gui and Mengshu Sun and Zhiqiang Zhang and Lei Liang and Jun Zhou},
  title = {{Improving Natural Language Understanding for LLMs via Large-Scale Instruction Synthesis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i24.34771},
}
Figure