TL;DR
ProofPaths is an AI agent framework designed to verify that students actually understand the work they submit, a critical challenge in the age of LLM-assisted assignments. By combining a multi-agent "peer review" grading system with individualized, interactive interviews grounded in the student's own text, it automates the labor-intensive process of probing student reasoning at scale.
What Problem Does This Paper Address?
As Large Language Models (LLMs) become ubiquitous, educators face a growing dilemma: a correct solution no longer guarantees student comprehension. A student might use an AI to generate a perfect "normal equation" for a least-squares problem without understanding the underlying requirement for linearly independent columns.
While individualized interviews (vivas) are the gold standard for assessing true understanding, they are notoriously difficult to scale. A typical university section might require a ratio of three TAs for every thirty students just to conduct these interviews—a staffing level most institutions find unsustainable. ProofPaths seeks to bridge this gap by using AI agents to perform both the grading and the follow-up questioning.
Methods
The framework operates through two parallel workflows: Multi-Agent Grading and Individualized Interviewing.
1. Multi-Agent Grading with Peer Review
Instead of relying on a single prompt, ProofPaths uses three distinct grading agents, each with a specific persona:
- Correctness: Focuses on the accuracy of the solution.
- Rigor: Evaluates the depth and logic of the reasoning.
- Skeptical: Specifically looks for risks of awarding unwarranted credit.
These agents first perform concurrent initial reviews. In a second round, they read each other's critiques and revise their own scores and feedback. The final automated grade is the median of the three revised scores, a strategy designed to bound the influence of any single numerical outlier or "hallucination."
2. Grounded Interviewing
The interview agent doesn't just ask generic questions. It parses the student's submission into numbered excerpts. For each turn of the interview, the agent selects a specific excerpt (e.g., a specific equation or claim) and asks the student to justify it. The student's response then informs the next follow-up question. This process uses Multidimensional Item Response Theory (MIRT) to estimate student ability and guide the difficulty of subsequent questions.
3. Technical Architecture
The system is built with a Go backend for user management and a Python (FastAPI) service to coordinate the agents. It leverages institutional High-Performance Computing (HPC) resources (NVIDIA A100 GPUs) for inference, using vLLM and KV cache quantization to maintain performance.
Datasets
The paper focuses on framework design and functional validation rather than training on a new massive dataset. To evaluate the system, the authors used:
- Synthetic Probes: Nine deterministic tests using a synthetic algebra solution and scripted agent responses to verify the "controller" logic (e.g., how the system handles invalid excerpt IDs or score perturbations).
- Authored Illustrations: A least-squares problem used to demonstrate how the interview agent can catch overgeneralized explanations.
Key Insights and Ideas
- The Median Rule for Robustness: By selecting the median score of three agents, the system becomes mathematically robust against one "rogue" agent. If two agents agree on a range, the final grade is guaranteed to stay within that range regardless of the third agent's score.
- Explicit Source Linking: Every interview question is linked to a retrieved passage from the submission. This allows instructors to quickly audit the transcript and see exactly what part of the student's work the AI was questioning.
- Agentic Peer Review: Allowing agents to critique each other's grading before finalizing a score mimics the human academic review process, leading to more nuanced and defensible feedback.
- Deterministic Controllers: The authors emphasize that while the LLMs are probabilistic, the logic governing the assessment (score aggregation, excerpt tracking, retry limits) must be deterministic and strictly tested.
Key Results
The functional validation confirmed that the framework's controllers behave exactly as specified across various edge cases.
| Input or Perturbation | Observed Behavior |
|---|---|
| Valid excerpt identifier | Exact excerpt returned |
| Unknown excerpt identifier | Rejected |
| Reuse with unused excerpts | Rejected (encourages broader coverage) |
| Upper score shift (95 to 100) | Median stayed stable at 82 |
| Three invalid outputs | Explicit failure (prevents infinite loops) |
In qualitative testing (Fig. 2), the system successfully identified when a student provided a correct formula but an incorrect justification (claiming every quadratic loss has a unique minimizer), and it correctly pivoted to a follow-up question that tested that specific misunderstanding.
Limitations and Future Work
- Extraction Bias: The current system extracts excerpts in the order they appear in the text. This might lead to "front-loading" the interview, potentially omitting critical reasoning found at the end of long solutions.
- Written vs. Oral: The authors acknowledge that written responses lack the dynamic, real-time flow of oral TA interviews and may present different accessibility challenges.
- Calibration: While the agents provide "understanding" labels (e.g., "partially understood"), these still require extensive calibration against expert human judgment to ensure they align with institutional standards.
- Resource Demand: Running multiple agents and multi-turn interviews per student is computationally expensive, requiring significant GPU capacity for large classes.
Summary updated