Training and evaluating language models requires high-quality, diverse data. Historically, this meant choosing between two sub-optimal paths: expensive, slow human annotation, or unconstrained language model generation, which often suffers from hallucinations and a lack of structural control.
Programmatic question generation offers a powerful alternative. By using structured templates to bridge the gap between raw data sources and natural language, researchers can procedurally synthesize vast, high-quality datasets. This approach provides deterministic control over the underlying logic and facts while scaling effortlessly.
Grounding Text in Structured Knowledge Bases
The simplest way to understand why template-based generation works is to look at how it handles structured data, such as knowledge graphs or relational databases. A knowledge base contains precise, unambiguous facts (e.g., mapping specific properties or entities to their corresponding values).
Instead of asking humans to write questions about these facts, researchers use predefined sentence templates with placeholders. For instance, Chenghua Lin et al. demonstrated in Sherlock: a Semi-Automatic Quiz Generation System using Linked Data how mapping specific properties from a structured knowledge base to sentence templates allows for the semi-automatic generation of educational quizzes. By swapping out entity labels and attributes programmatically via string replacement, thousands of unique questions can be generated instantly.
This mechanism ensures absolute factual grounding. Because the questions are generated directly from a verified database, they cannot contain hallucinations. For example, in the clinical domain, Preethi Raghavan et al. developed emrKBQA: a Clinical Knowledge-Base Question Answering Dataset using this slot-filling approach to populate templates with electronic health record (EHR) data, creating realistic medical question-answering pairs at scale. Similarly, Petr Babkin et al. showed in BuDDIE: A Business Document Dataset for Multi-task Information Extraction that visual question-answering pairs can be programmatically generated from dense entity annotations on business documents, drastically reducing the manual labor required for enterprise document processing while maintaining strict alignment between extracted entities and question targets.
Reframing Tasks as Question Answering
Templates do more than just extract facts; they can fundamentally transform how a model approaches a task. Natural Language Understanding (NLU) tasks like intent classification or slot filling are traditionally treated as classification problems requiring custom architecture heads and extensive domain-specific data.
However, Alexandros Papangelis et al. showed in Language Model is All You Need: Natural Language Understanding as Question Answering that these tasks can be reframed as question-answering problems by using automated templates to convert structured task definitions into natural language queries. This allows a model pre-trained on massive reading comprehension datasets to solve NLU tasks without any architectural changes, achieving high performance even in low-data regimes.
This concept extends directly to instruction tuning. Unstructured text from the web can be repurposed into high-quality instruction-following data by applying custom templates. In COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning, Wenhao Huang et al. utilized custom-designed templates to convert raw internet text into structured instruction-following pairs. Similarly, Xiao Zhang et al. introduced a template-based instruction synthesis framework in Dólares or Dollars? Unraveling the Bilingual Prowess of Financial LLMs Between Spanish and English to convert raw financial domain data into structured instruction-response pairs. By providing clear context and expected output formats through templates, models learn specific task execution logic—such as summarization or sentiment analysis—without requiring manual prompt engineering for every new instance.
Isolating Variables for Rigorous Benchmarking
When evaluating a model's reasoning capabilities, unconstrained natural language introduces confounding variables. If a model fails a test, it is often unclear whether it lacks the underlying reasoning skill or simply misunderstood a uniquely phrased question.
Procedural template generation solves this by enforcing strict structural consistency. In creating a benchmark for physical reasoning, Cory Paik et al. utilized manually curated templates in PROST: Physical Reasoning about Objects through Space and Time to procedurally generate thousands of multiple-choice questions. Because the syntax remains identical across instances, any variation in model performance can be cleanly isolated and attributed to the specific physical concept being tested rather than linguistic variance.
This structural consistency also enables powerful data augmentation across domains. For example, Xingyu Chen et al. scaled a web-based structural reading comprehension dataset in WebSRC: A Dataset for Web-Based Structural Reading Comprehension by creating abstract question templates for specific document layouts and automatically applying them across different domains. By leveraging the consistent logical organization of structured documents, the authors scaled dataset creation while maintaining a uniform evaluation framework.
Abstracting and Learning the Templates
While early methods relied entirely on static, hand-crafted templates, modern approaches introduce flexibility by using machine learning models to abstract or generate the templates themselves.
For instance, Rungsiman Nararatwong et al. employed template-based abstraction in DBQR-QA: A Question Answering Dataset on a Hybrid of Database Querying and Reasoning to transform specific natural language questions into generalized formats that align with a structured knowledge graph schema. Instead of hardcoding every variation, a sequence-to-sequence model can be trained to perform this transformation.
As demonstrated by Natthawut Kertkeidkachorn et al. in Fin-DBQA Shared-task: Database Querying and Reasoning, models can transform specific data-driven questions into generalized templates, which are then dynamically re-populated with new entities and metrics. This hybrid approach combines the scalability and creativity of generative models with the factual guardrails of template-based synthesis, pointing toward a future of fully automated, controllable data pipelines.
Key Foundational Work
Language Model is All You Need: Natural Language Understanding as Question Answering (2020) — contributed 1 concept elements to this direction.
Sherlock: a Semi-Automatic Quiz Generation System using Linked Data (2014) — contributed 1 concept elements to this direction.
DaNetQA: a yes/no Question Answering Dataset for the Russian Language (2020) — contributed 1 concept elements to this direction.