Easy Dataset: A Unified and Extensible Framework for Synthesizing LLM Fine-Tuning Data from Unstructured Documents

Ziyang Miao, Qiyu Sun, Jingyuan Wang, Yuchen Gong, Yaowei Zheng, Shiqi Li, Richong Zhang

TL;DR

This paper introduces "Easy Dataset," a novel, end-to-end framework designed to simplify the creation of high-quality, domain-specific fine-tuning data for Large Language Models (LLMs) from unstructured documents. By combining adaptive document processing with persona-driven data synthesis and a user-friendly GUI, Easy Dataset addresses the critical challenge of acquiring specialized data, significantly improving LLM performance on domain-specific tasks while retaining general capabilities.

What Problem Does This Paper Address?

Large language models (LLMs) have shown remarkable general capabilities, but their effectiveness in specialized domains is often hampered by a lack of high-quality, domain-specific training data. Manually curating such datasets is a laborious and expensive process. Existing automated data synthesis tools often struggle with the complexity of real-world unstructured documents (e.g., mixing text, tables, figures) and generating diverse, faithful question-answer (QA) pairs that avoid overfitting. This gap necessitates a more robust and accessible solution for generating specialized LLM training data.

Methods

Easy Dataset tackles this problem through a two-pronged approach:

  1. Adaptive Document Processing: This component handles the extraction of textual content from diverse document formats (plain text, Markdown, DOCX, PDF). For complex PDFs, it employs layout analysis to identify content regions and uses vision-language models (VLMs) to accurately extract information from visual elements. The extracted text is then segmented into semantically coherent chunks using a "Hybrid Chunking" strategy, which combines length-based, structure-based, and manual splitting for optimal context window compatibility.
  2. Persona-Driven Data Synthesis: This component generates diverse and stylistically controlled QA pairs. It leverages a "Massive Genre-Audience" (MGA) inspired pipeline:
  3. Persona Synthesis: For each document chunk, it automatically generates (Genre, Audience) pairs to define specific personas for question generation. This allows for varied question styles and knowledge levels.
  4. Persona-Guided QA Generation: LLMs are prompted using these personas, text chunks, and a customizable system prompt to generate diverse questions. Answers are then generated using a "knowledge-enhanced prompting" strategy, ensuring factual consistency with the source content. The framework also supports chain-of-thought (CoT) reasoning for enhanced interpretability.

A key feature is the human-in-the-loop visual interface, allowing users to review and refine intermediate outputs at various stages, ensuring data quality and usability for non-technical users.

Datasets

The paper uses financial reports as a representative domain example for evaluation. They collected five up-to-date financial reports and built a domain-specific evaluation dataset of 100 questions derived from these reports. The framework itself is designed to synthesize fine-tuning datasets from unstructured documents, rather than relying on pre-existing structured datasets for its core functionality.

Key Insights and Ideas

  • Bridging the Domain Gap: The core insight is that specialized LLM performance hinges on specialized data. Easy Dataset provides a practical solution to this bottleneck.
  • Handling Document Complexity: The adaptive parsing, especially the use of VLMs for mixed-content PDFs, is crucial for unlocking information from real-world documents that traditional parsers struggle with.
  • Persona-Driven Diversity: The persona-driven approach is a powerful technique for generating varied QA pairs that go beyond simple question-answer formats, leading to more robust fine-tuning. This is inspired by methods used for pretraining data augmentation but adapted for fine-tuning.
  • Usability is Key: The integrated GUI and end-to-end workflow significantly lower the barrier to entry for domain experts who may not be ML engineers, democratizing LLM fine-tuning.
  • Preserving General Capabilities: Fine-tuning with domain-specific data synthesized by Easy Dataset not only improves performance on target tasks but also maintains the LLM's general language understanding, a critical balance.

Key Results

Experiments on a financial QA task showed that fine-tuning the Qwen2.5-7B-Instruct model using Easy Dataset significantly improved its domain-specific performance.

  • The base model scored 3.2 on domain knowledge.
  • Fine-tuning with naive data synthesis achieved a score of 57.0.
  • Fine-tuning with persona-driven data synthesis further boosted performance to 59.6. On general-purpose benchmarks like MMLU, the persona-driven approach achieved the best results and competitive performance across other tasks, demonstrating its effectiveness in improving generalization.

Limitations and Future Work

The authors acknowledge that the framework is extensible and plan to incorporate several future directions:

  • Broader Modalities: Expanding support to include SQL, tables, and multi-modal data.
  • Automated Quality Monitoring: Developing mechanisms for automatic quality checks during data synthesis.
  • Advanced Enrichment Strategies: Further enhancing data variety and fidelity through more sophisticated techniques.

Summary updated

Abstract

Large language models (LLMs) have shown impressive performance on general-purpose tasks, yet adapting them to specific domains remains challenging due to the scarcity of high-quality domain data. Existing data synthesis tools often struggle to extract reliable fine-tuning data from heterogeneous documents effectively. To address this limitation, we propose Easy Dataset, a unified framework for synthesizing fine-tuning data from unstructured documents via an intuitive graphical user interface (GUI). Specifically, Easy Dataset allows users to easily configure text extraction models and chunking strategies to transform raw documents into coherent text chunks. It then leverages a persona-driven prompting approach to generate diverse question-answer pairs using public-available LLMs. Throughout the pipeline, a human-in-the-loop visual interface facilitates the review and refinement of intermediate outputs to ensure data quality. Experiments on a financial question-answering task show that fine-tuning LLMs on the synthesized dataset significantly improves domain-specific performance while preserving general knowledge. The source code and installable package are available at https://github.com/ConardLi/easy-dataset and have garnered over 9,000 GitHub stars.
BibTeX
@inproceedings{miao2025easy,
  author = {Ziyang Miao and Qiyu Sun and Jingyuan Wang and Yuchen Gong and Yaowei Zheng and Shiqi Li and Richong Zhang},
  title = {{Easy Dataset: A Unified and Extensible Framework for Synthesizing LLM Fine-Tuning Data from Unstructured Documents}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure