QGen Studio: An Adaptive Question-Answer Generation, Training and Evaluation Platform

Movina Moses, Mohab Elkaref, James Barry, Shinnosuke Tanaka, Vishnudev Kuruvanthodi, Nathan Herr, Campbell D. Watson, Geeth De MelAAAI 2025 · DOI

Abstract

QGen Studio is an end-to-end, interactive platform designed to bridge the gap between generic Large Language Models (LLMs) and niche domain expertise. It enables users to generate high-quality synthetic question-answer (QA) datasets from their own documents, fine-tune models locally using efficient techniques like Low-Rank Adaptation (LoRA), and evaluate performance through a side-by-side comparison interface. By providing a transparent, "human-in-the-loop" pipeline, QGen Studio democratizes the process of domain adaptation for specialized research and industrial applications.

Introduction

While Large Language Models (LLMs) demonstrate remarkable general capabilities, they frequently falter when applied to specialized or "niche" domains—such as specific legal frameworks, proprietary technical manuals, or rare medical conditions—due to a lack of exposure to domain-specific data. Creating the necessary training data manually is prohibitively expensive and slow. While synthetic data generation offers a workaround, existing tools often operate as "black boxes" with limited user oversight, leading to inconsistent data quality and difficulties in fine-tuning without massive compute resources. QGen Studio addresses these challenges by providing a unified, transparent platform for dataset creation and model refinement.

The QGen Studio Pipeline

The platform integrates a six-step pipeline designed to take raw data and transform it into a specialized, fine-tuned model. This process is divided into three primary phases: adding reference data, generating synthetic datasets, and training/evaluating models.

The six-step QGen Studio pipeline, encompassing reference data addition, synthetic dataset generation, and model training.
  1. Document Upload: Utilizing Docling, the platform converts PDFs, JSONs, and URLs into structured text, preserving critical metadata like headings and paragraph IDs.
  2. Example Prompts: Users curate specific QA pairs to serve as "few-shot" examples, allowing for granular control over the style and complexity of the generated output.
  3. Dataset Generation: The platform interfaces with major providers (OpenAI, IBM watsonx, HuggingFace) and leverages Apple’s MLX framework for local execution.
  4. Filtering & Metrics: Generated pairs are ranked using computational metrics (Bleu, Rouge, Meteor) and semantic measures (TF-IDF, Cosine similarity) against source context to ensure quality.
  5. Model Training: Users can fine-tune LLMs locally using Low-Rank Adaptation (LoRA) via the MLX library, making training feasible on consumer-grade hardware like Apple Silicon.
  6. Model Explorer: A side-by-side evaluation tool allows users to pose questions to both a base model and the fine-tuned version to visually inspect improvements.

Key Innovations and Insights

QGen Studio introduces several features aimed at improving the reliability of synthetic data workflows:

  • Contextual Transparency: The Dataset Viewer highlights the exact sentence or "span" in the source document from which an answer was derived. This traceability is crucial for identifying and mitigating hallucinations.
  • Local-First Fine-Tuning: By integrating MLX and LoRA, the platform emphasizes that domain adaptation should not require massive server farms. Researchers can curate datasets and train specialized models entirely on high-end laptops.
  • Interactive Refinement: The platform treats dataset creation as an iterative process. Users can immediately adjust example prompts or chunking parameters based on initial results, rather than waiting for lengthy batch processes to conclude.

Conclusion and Future Work

QGen Studio demonstrates a successful end-to-end integration of raw PDF parsing, synthetic generation, and local model inference. By utilizing LoRA, it makes model customization accessible to users with limited infrastructure while maintaining high standards for data quality control.

Future iterations of the platform aim to expand beyond single-document processing to support multi-document querying, enabling more complex cross-reference reasoning. Additionally, the authors plan to introduce support for multi-hop reasoning and multilingual generation to better serve the global research community.

Summary updated

BibTeX
@inproceedings{moses2025qgen,
  author = {Movina Moses and Mohab Elkaref and James Barry and Shinnosuke Tanaka and Vishnudev Kuruvanthodi and Nathan Herr and Campbell D. Watson and Geeth De Mel},
  title = {{QGen Studio: An Adaptive Question-Answer Generation, Training and Evaluation Platform}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i28.35362},
}
Figure