TL;DR
VeriMath is a human-in-the-loop framework that uses Large Language Models (LLMs) to generate high-quality, curriculum-aligned mathematics assessment items. By combining "difficulty anchoring" with a multi-step automated verification and repair loop, it produces drafts that are mathematically rigorous and ready for expert teacher review, achieving a 94% verifier acceptance rate.
What Problem Does This Paper Address?
While Large Language Models (LLMs) have revolutionized text generation, their application in high-stakes education—specifically mathematics—remains fraught with challenges. A "good" math question isn't just fluent; it must be mathematically 100% correct, strictly aligned with a specific national curriculum, set at the correct cognitive difficulty level (e.g., Bloom’s Taxonomy), and formatted perfectly in LaTeX.
Prior work has struggled to balance these constraints. LLMs often "hallucinate" incorrect solutions, produce items that are too easy or too hard, or generate questions that are nearly identical to their training data. VeriMath tackles this by moving away from "one-shot" generation toward a structured, verifiable authoring workflow.
Methods
The VeriMath framework introduces two primary innovations to the standard LLM prompting pipeline:
- Difficulty Anchoring: Instead of relying on the LLM's internal (and often inconsistent) sense of difficulty, the system retrieves $k$ reference problems from a teacher-annotated corpus. These "anchors" match the target schema and cognitive level, providing the model with concrete examples of the expected reasoning depth and style without encouraging direct copying.
- Consecutive Verification Mechanism (CVM): This is a stateful, iterative process. A candidate item must pass $N=3$ consecutive verification checks by a "Verifier Agent." If a check fails, the Verifier generates a detailed failure trace (a "bug report"), which a "Corrector Agent" uses to repair the item. This reduces the likelihood of a "lucky" pass from a stochastic model.
The system uses distinct role prompts for the Generator (the item writer), the Verifier (a strict mathematics teacher), and the Corrector (the editor).
Datasets
The authors instantiated VeriMath using a case study of the Vietnamese Grade 12 national curriculum:
- Reference Corpus: 4,555 mathematics problems collected from national and practice exams, each annotated by teachers for assessment schema and cognitive level.
- Requirement Set: 50 distinct generation requirements covering three formats: Multiple Choice, True/False clusters, and Short Answer.
- Backbones: The framework was evaluated using two models: Qwen3-8B and gpt-oss-20b.
Key Insights and Ideas
- The "Readiness Gap": One of the paper's most important findings is the delta between "automated acceptance" and "teacher readiness." While the AI verifier accepted 94% of items, human experts only deemed about 56% "ready for use." This highlights that pedagogical quality involves subtle judgments (like clarity and "exam-style" feel) that current automated verifiers cannot yet fully capture.
- Depth over Content: By using anchors to calibrate reasoning depth rather than just topic content, VeriMath successfully generates novel items that still "feel" like they belong in a specific grade-level exam.
- Stateful Verification is Essential: The requirement for consecutive passes is a clever way to combat the inherent variance in LLM outputs, ensuring that the final draft isn't just a fluke but a robustly correct mathematical artifact.
Key Results
- Operational Success: The pipeline produced verifier-accepted drafts for 94% (47/50) of requirements.
- Mathematical Accuracy: In blind expert reviews, items generated by the Qwen3-8B backbone achieved 95.5% mathematical correctness and 95.5% curriculum alignment.
- Teacher Readiness: Strict evaluations showed that 56.8% of generated items met all criteria for immediate classroom use (correctness, alignment, level match, and clarity).
- Novelty: Embedding-based novelty scores remained high (~0.8), indicating that the system does not simply paraphrase its anchors.
Limitations and Future Work
Despite the high accuracy, the authors identify several persistent "failure modes":
- Invalid Problem Statements: The most frequent expert-marked issue (43.8% of flags) involved drafts that lacked a necessary condition or described an impossible geometric/algebraic configuration.
- Formatting Sensitivity: Even with automated repair, LaTeX rendering and symbol placement remain sensitive to local examination conventions.
- Self-Grading Bias: Currently, the verifier and generator often come from the same model family. Future work suggests using independent model families for verification to increase reliability.
- Symbolic Checks: The authors propose integrating symbolic math engines (like SymPy or Mathematica) to provide absolute guarantees on answer uniqueness and statement validity.
Ultimately, VeriMath is positioned not as an autonomous generator, but as a powerful drafting aid that keeps the human teacher "in the loop" to provide the final pedagogical seal of approval.
Summary updated