VeriMath: Human-in-the-Loop Generation and Verification of Curriculum-Aligned Mathematics Assessment Items

Quy P. Nguyen, Duong Q. Nguyen, Dinh P. T. Nguyen, Truong-Giang T. Le, Truong A. Bui, Vi H. Vo, Quang-Thinh Bui, My-Phuong Ngo, Trong N. Do, Thao P. Le, Phuc M. Nguyen, H. Nguyen-XuanarXiv 2026 · arXiv

TL;DR

VeriMath is a human-in-the-loop framework that uses Large Language Models (LLMs) to generate high-quality, curriculum-aligned mathematics assessment items. By combining "difficulty anchoring" with a multi-step automated verification and repair loop, it produces drafts that are mathematically rigorous and ready for expert teacher review, achieving a 94% verifier acceptance rate.

What Problem Does This Paper Address?

While Large Language Models (LLMs) have revolutionized text generation, their application in high-stakes education—specifically mathematics—remains fraught with challenges. A "good" math question isn't just fluent; it must be mathematically 100% correct, strictly aligned with a specific national curriculum, set at the correct cognitive difficulty level (e.g., Bloom’s Taxonomy), and formatted perfectly in LaTeX.

Prior work has struggled to balance these constraints. LLMs often "hallucinate" incorrect solutions, produce items that are too easy or too hard, or generate questions that are nearly identical to their training data. VeriMath tackles this by moving away from "one-shot" generation toward a structured, verifiable authoring workflow.

Methods

The VeriMath framework introduces two primary innovations to the standard LLM prompting pipeline:

  1. Difficulty Anchoring: Instead of relying on the LLM's internal (and often inconsistent) sense of difficulty, the system retrieves $k$ reference problems from a teacher-annotated corpus. These "anchors" match the target schema and cognitive level, providing the model with concrete examples of the expected reasoning depth and style without encouraging direct copying.
  2. Consecutive Verification Mechanism (CVM): This is a stateful, iterative process. A candidate item must pass $N=3$ consecutive verification checks by a "Verifier Agent." If a check fails, the Verifier generates a detailed failure trace (a "bug report"), which a "Corrector Agent" uses to repair the item. This reduces the likelihood of a "lucky" pass from a stochastic model.

The system uses distinct role prompts for the Generator (the item writer), the Verifier (a strict mathematics teacher), and the Corrector (the editor).

Table 1: Positioning of VeriMath Relative to Major Lines of Mathematics Item Generation

Datasets

The authors instantiated VeriMath using a case study of the Vietnamese Grade 12 national curriculum:

  • Reference Corpus: 4,555 mathematics problems collected from national and practice exams, each annotated by teachers for assessment schema and cognitive level.
  • Requirement Set: 50 distinct generation requirements covering three formats: Multiple Choice, True/False clusters, and Short Answer.
  • Backbones: The framework was evaluated using two models: Qwen3-8B and gpt-oss-20b.
Table 2: Assessment Schemas Used in the Vietnamese Case-Study Instantiation

Key Insights and Ideas

  • The "Readiness Gap": One of the paper's most important findings is the delta between "automated acceptance" and "teacher readiness." While the AI verifier accepted 94% of items, human experts only deemed about 56% "ready for use." This highlights that pedagogical quality involves subtle judgments (like clarity and "exam-style" feel) that current automated verifiers cannot yet fully capture.
  • Depth over Content: By using anchors to calibrate reasoning depth rather than just topic content, VeriMath successfully generates novel items that still "feel" like they belong in a specific grade-level exam.
  • Stateful Verification is Essential: The requirement for consecutive passes is a clever way to combat the inherent variance in LLM outputs, ensuring that the final draft isn't just a fluke but a robustly correct mathematical artifact.
Table 3: Teacher-Annotated Cognitive/Difficulty Levels in the Vietnamese Case Study

Key Results

  • Operational Success: The pipeline produced verifier-accepted drafts for 94% (47/50) of requirements.
  • Mathematical Accuracy: In blind expert reviews, items generated by the Qwen3-8B backbone achieved 95.5% mathematical correctness and 95.5% curriculum alignment.
  • Teacher Readiness: Strict evaluations showed that 56.8% of generated items met all criteria for immediate classroom use (correctness, alignment, level match, and clarity).
  • Novelty: Embedding-based novelty scores remained high (~0.8), indicating that the system does not simply paraphrase its anchors.

Limitations and Future Work

Despite the high accuracy, the authors identify several persistent "failure modes":

  • Invalid Problem Statements: The most frequent expert-marked issue (43.8% of flags) involved drafts that lacked a necessary condition or described an impossible geometric/algebraic configuration.
  • Formatting Sensitivity: Even with automated repair, LaTeX rendering and symbol placement remain sensitive to local examination conventions.
  • Self-Grading Bias: Currently, the verifier and generator often come from the same model family. Future work suggests using independent model families for verification to increase reliability.
  • Symbolic Checks: The authors propose integrating symbolic math engines (like SymPy or Mathematica) to provide absolute guarantees on answer uniqueness and statement validity.

Ultimately, VeriMath is positioned not as an autonomous generator, but as a powerful drafting aid that keeps the human teacher "in the loop" to provide the final pedagogical seal of approval.

Summary updated

Abstract

Automatic generation of mathematics assessment items is a growing area in educational artificial intelligence, yet classroom deployment remains challenging. Generated questions must satisfy multiple constraints: mathematical correctness, curriculum alignment, appropriate cognitive difficulty, and educator reviewability. While prior work has advanced text generation, reliable assessment authoring requires granular control over localized examination formats and expert educational judgment. To address this challenge, we present VeriMath, a generalized, human-in-the-loop framework for generating rigorous, curriculum-constrained mathematics assessment drafts. Using annotated reference items, the framework anchors generation to the requested examination section and cognitive reasoning level, then applies sequential automated verification and guided repair before submitting drafts for expert review. We evaluate VeriMath through a national curriculum case study using 4,555 mathematics items from the Vietnamese Grade 12 curriculum and 50 distinct generation requirements covering multiple-choice, true/false, and short-answer formats. Across two model backbones, the framework produces verifier-accepted drafts for 94% (47/50) of requirements per run. Blind expert evaluations confirm high mathematical accuracy and curriculum alignment, with strict teacher-readiness assessments accepting up to 56.8% of generated items. Analysis of rejected drafts reveals persistent domain-agnostic challenges involving formatting, statement validity, and solution-step completeness. These findings demonstrate that VeriMath can bridge generative AI and dependable educational applications by producing highly reviewable assessment drafts within tightly constrained national curriculum frameworks.
BibTeX
@misc{nguyen2026verimath,
  author = {Quy P. Nguyen and Duong Q. Nguyen and Dinh P. T. Nguyen and Truong-Giang T. Le and Truong A. Bui and Vi H. Vo and Quang-Thinh Bui and My-Phuong Ngo and Trong N. Do and Thao P. Le and Phuc M. Nguyen and H. Nguyen-Xuan},
  title = {{VeriMath: Human-in-the-Loop Generation and Verification of Curriculum-Aligned Mathematics Assessment Items}},
  year = {2026},
  eprint = {2611.23456},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23456},
}
Figure