Abstract
High-quality, large-scale instructions are crucial for aligning large language models (LLMs), however, there is a severe shortage of instruction in the field of natural language understanding (NLU). Previous works on constructing NLU instructions mainly focus on information extraction (IE), neglecting tasks such as machine reading comprehension, question answering, and text classification. Furthermore, the lack of diversity in the data has led to a decreased generalization ability of trained LLMs in other NLU tasks and a noticeable decline in the fundamental model's general capabilities. To address this issue, we propose Hum, a large-scale, high-quality synthetic instruction corpus for NLU tasks, designed to enhance the NLU capabilities of LLMs. Specifically, Hum includes IE (either close IE or open IE), machine reading comprehension, text classification, and instruction generalist tasks, thereby enriching task diversity. Additionally, we introduce a human-LLMs collaborative mechanism to synthesize instructions, which enriches instruction diversity by incorporating guidelines, preference rules, and format variants. We conduct extensive experiments on 5 NLU tasks and 28 general capability evaluation datasets for LLMs. Experimental results show that Hum enhances the NLU capabilities of six LLMs by an average of 3.1%, with no significant decline observed in other general capabilities.
Introduction
While Large Language Models (LLMs) have demonstrated remarkable conversational abilities, their performance on core Natural Language Understanding (NLU) tasks—such as precise information extraction (IE), machine reading comprehension (MRC), and text classification (TC)—often lags behind specialized models. This "NLU gap" stems from two primary issues: the narrow focus of existing instruction datasets on basic IE tasks and the "overfitting trap," where rigid formatting requirements (e.g., strict JSON outputs) cause models to lose their general reasoning and linguistic flexibility. For instance, models like OneKE have shown up to a 14.5% decline in general performance after specialized NLU tuning.
To bridge this gap, we introduce Hum, a comprehensive framework and dataset comprising over 2.8 million synthetic instructions. Hum moves beyond simple task-label pairs by employing a human-LLM collaborative mechanism to generate diverse, high-quality instructions that preserve general intelligence while sharpening specialized NLU skills.
The Hum Framework: Compound Instruction Synthesis
The core of our approach is the Instruction Synthesis Framework, which transforms raw text into structured NLU tasks through four distinct stages of enrichment.
As illustrated in the framework, we utilize three primary strategies to ensure instruction diversity:
- Guidelines Synthesis: Prompts are augmented with semantic explanations, label definitions, and representative positive/negative examples to provide the model with deeper context.
- Preference Rules Synthesis: We leverage GPT-4 to generate novel labeling rules (e.g., entity boundary definitions or numerical constraints), teaching the model the underlying logic of the task rather than just the final answer.
- Format Variants Synthesis: To prevent formatting bias, Hum includes instructions requiring outputs in various styles, including Markdown tables, plain text, and multiple JSON structures.
Dataset Composition and Evaluation
The resulting Hum dataset contains 2,812,832 instructions across a wide array of tasks: Named Entity Recognition (23%), Relation Extraction (29%), Knowledge Graph Extraction (12%), and others including Event Extraction and Text Classification.
We evaluated the impact of Hum by fine-tuning six base models (including Qwen2, Llama3, and Mistral) and testing them on five NLU benchmarks: CrossNER (NER), FewRel (RE), CCF Law (Event Extraction), C3 (Reading Comprehension), and IMDB (Sentiment Classification). To monitor general capability maintenance, we utilized the OpenCompass framework across 28 datasets.
Experimental Results
Our results demonstrate that models fine-tuned on the Hum dataset consistently outperform both their base versions and other specialized NLU models like OneKE and YAYI-UIE.
Key findings include:
- Significant NLU Boost: Hum-trained models achieved an average NLU capability improvement of 3.1% across all tested base models.
- Superiority over Specialized Models: As shown in the results table, Hum-tuned variants (e.g., Hum_Qwen2, Hum_Llama3) frequently outperformed state-of-the-art specialized models like OneKE and Llama3-iepile, particularly in complex tasks like CCF Law.
- Preservation of General Intelligence: Unlike previous NLU-specific tuning methods that often lead to a sharp decline in general reasoning, Hum-tuned models maintained stable performance across general benchmarks like math and coding. Radar chart evaluations confirmed no significant decline in the fundamental model's general capabilities.
Conclusion
The Hum dataset demonstrates that the key to enhancing specialized NLU capabilities without sacrificing general LLM performance lies in instruction diversity. By incorporating semantic guidelines, logical preference rules, and varied output formats, we can create models that are both precise NLU engines and versatile general assistants. Future work will focus on addressing current weaknesses in multiple-choice question-answering and further refining the secondary SFT process to eliminate minor performance decays in specific base models.
Summary updated