Scalable Conversational Assessment: Chat-Based Exams with a Two-Agent LLM Pipeline

Geoffrey Challen, Craig ZillesarXiv 2026 · arXiv

TL;DR

Researchers have developed a scalable, AI-driven alternative to oral exams using a two-agent LLM pipeline that separates interviewing from evaluation. By validating the system against fictional topics to prevent data leakage and using "scaffolding-aware" rubrics, they achieved consistent, high-quality assessments at a cost of roughly $0.27 per student.

What Problem Does This Paper Address?

Oral exams are gold standards for assessment because they allow instructors to probe a student's reasoning and clarify ambiguities through follow-up questions. However, they suffer from two major flaws: cost (examiner time scales linearly with enrollment) and consistency (human graders are notoriously "noisy" and inconsistent).

While Large Language Models (LLMs) offer a potential solution, a naive single-agent chatbot is prone to "leaking" answers to students or being easily manipulated by adversarial prompting. This paper tackles the challenge of building an automated system that captures the benefits of oral exams while remaining secure, consistent, and scalable to large classes.

Methods

The authors propose a two-agent architecture designed to separate concerns and ensure security:

  1. The Interviewer (Conversational): This agent interacts directly with the student. Crucially, it is kept in the dark regarding the specific rubric and reference answers. It only receives high-level guidance on what to ask next.
  2. The Evaluator (Analytical): This hidden agent resides in a "Secret Information Zone." It has full access to the rubrics, reference material, and the ongoing transcript. After every student message, it updates the student's progress and sends instructions to the Interviewer (e.g., "ask the student to elaborate" or "move to the next topic").
Figure 2: The two-agent architecture. After each student mes- sage, the evaluator updates its state and sends the interviewer guidance, never reference content. The interviewer cannot leak answers it does not have.

The system uses scaffolding-aware rubrics. Instead of a binary "correct/incorrect," it grades based on how much help the student needed. A student only earns a "Met" status if they volunteer information unprompted; if the interviewer has to ask a leading question to extract the answer, the student only receives "Partially Met."

Datasets

The study utilized two primary data sources for evaluation:

  • Fictional Topic Validation: To ensure the LLM personas used for testing weren't just reciting training data, the authors invented the "Korvath Procedure"—a fictional method in computational ecology. This allowed them to precisely control what a "Good" vs. "Unsatisfactory" student persona should know.
  • Classroom Deployment: The system was deployed in an AI literacy course ($N=11$ students) at a large public university. This resulted in 41 evaluated attempts across three topics: the Turing test, AI creativity, and student reflections.

Key Insights and Ideas

  • The Fictional-Topic Insight: Validating educational LLMs is tricky because the model might already "know" the subject matter from its pre-training. By testing against a completely fabricated topic, the researchers proved the system could distinguish between a student's actual knowledge and the model's internal biases.
  • Separation of Concerns: Splitting the "talker" from the "grader" prevents the AI from becoming a "helpful assistant" that accidentally gives away the answer. If a student asks the Interviewer for help, the Interviewer literally doesn't have the answer to give.
  • Scaffolding as a Grading Metric: The hardest part of automated grading is "credit assignment." The authors found that the most robust way to grade is to look at the interaction. If the AI provided the "scaffolding" (the hints), the student shouldn't get full credit.
  • Regression Testing with Personas: The authors treated their 11 LLM personas (ranging from "Good Student" to "Prompt Injector") as a regression suite. Any change to the system's prompts had to pass this suite to ensure no "grading drift" occurred.

Key Results

  • Validation Success: In the fictional-topic test, the system correctly graded all 11 personas (5 legitimate, 6 adversarial) across 619 automated checks with zero failures.
  • Adversarial Robustness: Adversarial personas like "Answer Extractors" and "Prompt Injectors" were successfully identified and received "Unsatisfactory" grades because the Interviewer never possessed the reference material to leak.
Table 2: Validation harness: 11 personas tested using the fictional assessment, 619 automated checks, zero failures.
  • Cost-Effectiveness: The median 20-minute assessment cost just $0.27 using GPT-5.2 (via Azure OpenAI), making it viable for large-scale courses.
  • Human-AI Alignment: Post-hoc audits by human judges and a different LLM (Claude) showed that while the deployed AI was slightly more lenient than humans, it was generally as consistent as experienced faculty members.

Limitations and Future Work

  • Scale: The classroom deployment was small ($N=11$). Future work is needed to see how the system handles the "long tail" of student behaviors in courses with hundreds or thousands of enrollments.
  • Leniency: The audit revealed the system was sometimes too lenient, failing to sufficiently discount credit when the Interviewer provided too much scaffolding. The authors suggest a "second read" of the full transcript at the end of the exam to correct this.
  • Cheating: While the system is secure against "answer extraction," it cannot prevent a student from copy-pasting questions into a separate AI window in an unproctored environment. The authors recommend using this for summative credit only in proctored settings.
  • Subjectivity: The current system was tested on conceptual and argumentative topics. Its performance on highly technical or code-heavy subjects remains to be seen.

Summary updated

Abstract

Oral exams are compelling because (1) students can be asked to provide their reasoning for an answer and (2) their interactive nature permits follow-up questions to resolve ambiguous answers. But oral exams scale poorly. In this paper, we present scalable conversational assessment, an LLM-based system that conducts interactive text-based conversations with students to assess their ability to make and support an argument. We use a two-agent architecture to separate concerns and reduce the likelihood that answers are leaked to the student: the interviewer interacts directly with the student, but only receives guidance about the next question to ask the student; the hidden evaluator knows the goals of the whole activity, tracks the student's progress using a rubric, and guides the interviewer on when to follow-up on the current question or proceed to the next question. The system was first tested using a collection of simulated students with LLM personas of a particular ability level or adversarial tactic (e.g., answer-extractors, prompt-injectors, confident fabricators). A novel element of this evaluation was that it required the use of a fictional assessment topic to prevent personas that shouldn't have information from using knowledge present in the LLM's pre-training. The system was also deployed in a small, elective AI literacy course in both formative and summative contexts. Through manual and post-facto AI review of the transcripts we identified opportunities to improve the follow-up question generation and ensure that grading only credited ideas that the student (and not the interviewer) contributed.
BibTeX
@misc{challen2026scalable,
  author = {Geoffrey Challen and Craig Zilles},
  title = {{Scalable Conversational Assessment: Chat-Based Exams with a Two-Agent LLM Pipeline}},
  year = {2026},
  eprint = {2611.17220},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17220},
}
Figure