Easy Dataset: A Unified and Extensible Framework for Synthesizing LLM Fine-Tuning Data from Unstructured Documents

Ziyang Miao, Qiyu Sun, Jingyuan Wang, Yuchen Gong, Yaowei Zheng, Shiqi Li, Richong Zhang

TL;DR

This paper introduces "Easy Dataset," a unified, GUI-driven framework for automatically generating high-quality fine-tuning data for Large Language Models (LLMs) from unstructured documents. It addresses the challenge of domain adaptation by enabling users to efficiently create specialized datasets, significantly improving LLM performance on domain-specific tasks while retaining general knowledge.

What Problem Does This Paper Address?

Large Language Models (LLMs) are incredibly powerful but often struggle when applied to specialized domains due to a lack of high-quality, domain-specific training data. Manually curating such datasets is time-consuming and expensive. Existing automated data synthesis tools often fall short, facing difficulties in effectively parsing diverse and noisy documents and generating faithful, diverse question-answer (QA) pairs. This gap hinders the effective adaptation of LLMs for real-world, specialized applications.

Methods

Easy Dataset tackles this problem with a two-pronged approach:

  1. Adaptive Document Processing: This component handles the extraction of textual content from various document formats (plain text, Markdown, DOCX, PDF). It employs a suite of text extraction models and a "hybrid chunking" strategy. This strategy combines length-based, structure-based, and manual splitting to segment documents into semantically coherent text chunks, ensuring compatibility with LLM context windows. For complex PDFs, it uses layout analysis and vision-language models (VLMs) to accurately capture content from both textual and visual regions.
  2. Persona-Driven Data Synthesis: This component generates diverse and stylistically controlled QA pairs from the processed text chunks. It offers:
  3. Question Generation: LLMs are prompted to create questions based on text chunks, with control over style, audience, and tone. A stochastic punctuation dropout mechanism is used to prevent models from over-relying on punctuation.
  4. Answer Generation: A "knowledge-enhanced prompting" strategy ensures answers are faithful, factually consistent, and relevant to the source content. Users can configure output styles, and for reasoning LLMs, intermediate reasoning steps (chain-of-thought) are included for transparency.
  5. Persona-Driven QA-Pair Generation: Inspired by the Massive Genre-Audience (MGA) method, this feature automatically generates unique (Genre, Audience) pairs to guide LLMs in producing diverse QA pairs from multiple perspectives. This significantly enhances dataset diversity and scale.

A key feature across the pipeline is human-in-the-loop refinement, allowing users to review and edit intermediate outputs via an intuitive visual interface, ensuring data quality.

Datasets

The paper uses financial reports as a representative domain example for evaluation. They collected five up-to-date financial reports and built a domain-specific evaluation dataset of 100 questions derived from these sources. The framework itself is designed to synthesize fine-tuning datasets from unstructured documents, not relying on pre-existing structured datasets for its core function.

Key Insights and Ideas

  • Unified GUI for Data Synthesis: The most significant contribution is the creation of a single, user-friendly graphical interface that integrates the entire data synthesis pipeline, making it accessible to non-technical users.
  • Hybrid Chunking for Robustness: Combining different chunking strategies (length, structure, manual) is crucial for effectively segmenting diverse document types and ensuring that generated data is contextually relevant.
  • Persona-Driven Synthesis for Diversity: Leveraging "Genre-Audience" pairs to guide LLM generation is a powerful technique for creating stylistically varied and semantically rich QA pairs, which is vital for robust fine-tuning.
  • Human-in-the-Loop is Essential: Integrating human review and refinement at various stages is critical for ensuring the quality and accuracy of the synthesized data, bridging the gap between automated generation and real-world utility.
  • Domain Adaptation without Forgetting: Fine-tuning LLMs with domain-specific data synthesized by Easy Dataset significantly improves performance on target tasks while preserving their general knowledge capabilities.

Key Results

Experiments on a financial question-answering task showed that fine-tuning the Qwen2.5-7B-Instruct model using Easy Dataset's synthesized data led to substantial improvements.

  • The base model scored poorly on domain-specific tasks.
  • Fine-tuning with naive data synthesis achieved a score of 57.0 on the domain-specific evaluation.
  • Fine-tuning with persona-driven data synthesis further improved the score to 59.6.
  • On general-purpose benchmarks like MMLU, the persona-driven approach achieved the best results, demonstrating that domain adaptation did not come at the cost of general capabilities.

Limitations and Future Work

The authors acknowledge that while Easy Dataset is extensible, they plan to further enhance it by:

  • Supporting broader modalities: Including SQL, tables, and multi-modal data.
  • Integrating automatic quality monitoring: To further streamline the quality assurance process.
  • Developing advanced enrichment strategies: To continuously improve data variety and fidelity.

Summary updated

Abstract

Large language models (LLMs) have shown impressive performance on general-purpose tasks, yet adapting them to specific domains remains challenging due to the scarcity of high-quality domain data. Existing data synthesis tools often struggle to extract reliable fine-tuning data from heterogeneous documents effectively. To address this limitation, we propose Easy Dataset, a unified framework for synthesizing fine-tuning data from unstructured documents via an intuitive graphical user interface (GUI). Specifically, Easy Dataset allows users to easily configure text extraction models and chunking strategies to transform raw documents into coherent text chunks. It then leverages a persona-driven prompting approach to generate diverse question-answer pairs using public-available LLMs. Throughout the pipeline, a human-in-the-loop visual interface facilitates the review and refinement of intermediate outputs to ensure data quality. Experiments on a financial question-answering task show that fine-tuning LLMs on the synthesized dataset significantly improves domain-specific performance while preserving general knowledge. The source code and installable package are available at https://github.com/ConardLi/easy-dataset and have garnered over 9,000 GitHub stars.
BibTeX
@inproceedings{miao2025easy,
  author = {Ziyang Miao and Qiyu Sun and Jingyuan Wang and Yuchen Gong and Yaowei Zheng and Shiqi Li and Richong Zhang},
  title = {{Easy Dataset: A Unified and Extensible Framework for Synthesizing LLM Fine-Tuning Data from Unstructured Documents}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure