Towards Scalable Measurement of Durable Skills

Amir Globerson, Amy Keeling, Anisha Choudhury, Anna Iurchenko, Aviad Segal, Avinatan Hassidim, Ayça Çakmakli, Ben Gomes, Benn Witt, Cathy Cheunga, Cristine Legare, Diana Akrong, Eliad Carmi, Elisabeth Bauer, Gal Elidan, Hadas Gelbart, Hairong Mu, Katherine Chou, Lev Borovoi, Nir Kerem, Niv Efron, Noa Kerrem Gilo, Preeti Singh, Rajvi Kapadia, Rena Levitt, Roni Rabin, Ronit Levavi Morad, Rotem Yulzary, Shashank Agarwal, Sophie Allweis, Tracey Lee-Joe, Tzvika Stein, Yael Bar Moshe, Yael Haramaty, Yaniv Carmel, Yishay Mor, Yoav Bar Sinai, Yoav Bergner, Yossi Matias, Yuri LevarXiv 2026 · arXiv

TL;DR

This paper introduces Vantage, an AI-driven framework that uses an "Executive LLM" to orchestrate virtual teammates and dynamically steer group discussions to measure "durable skills" like collaboration, critical thinking, and creativity. By actively eliciting evidence of these complex human skills during naturalistic conversations, the system achieves a high density of assessable behaviors while maintaining automated scoring accuracy on par with human experts.

What Problem Does This Paper Address?

Durable skills (often called 21st-century skills) such as collaboration, creativity, and critical thinking are vital for modern workplace success, yet they remain notoriously difficult to evaluate at scale. Traditional educational assessments face a persistent trade-off between ecological validity (emulating natural, real-world human interaction) and psychometric rigor (standardized, scalable, and reproducible control).

Multiple-choice tests lack validity, while human-to-human group tasks introduce massive, uncontrollable variance—for instance, it is impossible to evaluate a student's conflict resolution skills if their group happens to agree on everything. The authors tackle this challenge by leveraging Large Language Models (LLMs) to bridge the gap between unstructured collaboration and standardized assessment.

Methods

The authors develop Vantage, a virtual assessment environment where a human participant collaborates with AI teammates to complete a joint task (such as a debate or a science experiment design). The architecture consists of two primary components:

  • The Executive LLM: Instead of deploying independent AI agents that might chat aimlessly, a single underlying model (Gemini 2.5 Pro or Gemini 3) acts as a centralized controller for all AI teammates collectively. Armed with pedagogical rubrics, the Executive LLM uses internal reasoning to dynamically steer the conversation toward situations that force the user to demonstrate specific skills—such as intentionally provoking a disagreement to test conflict resolution.
  • AI Evaluator: A separate Gemini model analyzes the resulting chat transcripts. To handle variance, it performs turn-by-turn scoring repeated 20 times per turn (filtering out "No Evidence/NA" cases) and aggregates these into holistic conversation-level scores using a trained regression model calibrated against human expert ratings.
The AI-based assessment protocol in Vantage
An excerpt showing the Executive LLM's internal reasoning and steering behavior

Datasets

The framework was validated using three distinct data sources:

  1. Human Collaboration Study: 188 English-speaking participants (ages 18-25) recruited via Prolific generated 373 multi-turn conversations across two collaborative tasks: a Science Experiment design and a Debate task. Two pedagogical experts from New York University annotated these transcripts turn-by-turn and holistically.
  2. Simulated Sandbox Conversations: To scale testing, the authors used Gemini to simulate human students behaving at predefined skill levels (e.g., Level 3 in Conflict Resolution) across 100 iterations per level to evaluate "recovery error" and evidence rates for creativity and critical thinking.
  3. Real-World Student Creativity Submissions: In partnership with the startup OpenMic, the authors evaluated a dataset of 280 real high school students completing a complex multimedia creativity task (designing a news segment based on a short story). 100 submissions were used for prompt/rubric tuning, and 180 were held out to evaluate the AI autorater against expert human grades.

Key Insights and Ideas

  • Orchestration Over Independence: Independent AI agents fail to generate sufficiently informative interactions for psychometric evaluation. A centralized "Executive" model that treats the conversation as an adaptive test is essential to guarantee that assessable behaviors actually manifest.
  • The Cross-Over Effect: Steering is highly specific. When the Executive LLM focuses on eliciting Conflict Resolution, it successfully drives up evidence for that specific sub-skill while naturally reducing evidence for Project Management, and vice versa.
  • Instruction Alone is Insufficient: Simply telling human participants to "focus on conflict resolution" before a conversation without active AI steering does not significantly increase the amount of skill evidence generated. The environment must dynamically provoke the behavior.
  • Simulation as a Psychometric Sandbox: The paper demonstrates that LLM-simulated human subjects closely mirror real human data distributions, providing a viable, low-cost "sandbox" to refine rubrics and assessment protocols before deploying them to real students.

Key Results

  • Evidence Elicitation: The Executive LLM significantly outperformed independent agents, achieving a conversation-level information rate of 92.4% for Project Management and 85% for Conflict Resolution when matched to the target skill.
  • Rater Agreement: The AI Evaluator's agreement with human experts (measured via Cohen's Kappa) was on par with inter-human rater agreement, falling in the moderate-to-good range of 0.45 to 0.64 for collaboration metrics.
  • Real-World Validation: On OpenMic's real student creativity tasks, the Gemini-based autorater achieved an exceptionally high Pearson correlation of 0.88 compared to expert human scores, demonstrating strong criterion validity for complex, non-conversational multimedia tasks.

Limitations and Future Work

  • Idealized Simulations: Fully simulated conversations are currently "too ideal," with evidence rates approaching 100%, which doesn't fully capture the messy unpredictability of real human interactions.
  • Ongoing Human Validation: While the collaboration skill track was heavily validated with human raters, human validation studies for the critical thinking and creativity conversational tracks are still ongoing.
  • Cultural and Linguistic Bias: The authors acknowledge that durable skills are deeply culturally situated. Future work will focus on testing the framework across diverse cultural settings and languages to ensure equity and inclusivity.

Summary updated

Abstract

Durable skills, such as collaboration, creativity and critical thinking, are instrumental to success in the modern workforce. Yet, measuring these skills remains a persistent challenge. Moreover, because what is not measured is often not taught, these skills are often overlooked in mainstream educational curricula. Designing effective assessments for these skills necessitates balancing two often-conflicting requirements: ecological validity and psychometric rigor. On the one hand, the assessment environment should emulate natural real-world human interaction between humans. On the other hand, it should be scalable, controllable and reproducible. Here we argue that LLMs can be used to better capture both of these aims. Concretely, we develop a framework where the subject converses with AI teammates in a way that resembles human-human interaction for authenticity, while also offering the psychometric control required for informative and robust assessment. Importantly, the AI participants not only act as teammates but also, in an "Executive LLM" setup, steer the conversation towards eliciting a high density of observable evidence for skill proficiency. We complement this with an AI evaluator that can be used to measure skill proficiency in such interactions. We evaluate our assessment protocol based on transcripts of interactions of human participants with our AI framework, for multiple durable skills. For the skill of creativity, we further demonstrate the efficacy of an autorater for evaluating complex tasks performed by real students. Our analysis shows that the use of the Executive LLM significantly increases elicited evidence and that LLM-automated scoring of conversations largely agrees with that of expert annotators. This research demonstrates the utility of orchestrated LLMs approaches for measuring complex social and cognitive constructs in a scalable and controllable manner.
BibTeX
@misc{globerson2026towards,
  author = {Amir Globerson and Amy Keeling and Anisha Choudhury and Anna Iurchenko and Aviad Segal and Avinatan Hassidim and Ayça Çakmakli and Ben Gomes and Benn Witt and Cathy Cheunga and Cristine Legare and Diana Akrong and Eliad Carmi and Elisabeth Bauer and Gal Elidan and Hadas Gelbart and Hairong Mu and Katherine Chou and Lev Borovoi and Nir Kerem and Niv Efron and Noa Kerrem Gilo and Preeti Singh and Rajvi Kapadia and Rena Levitt and Roni Rabin and Ronit Levavi Morad and Rotem Yulzary and Shashank Agarwal and Sophie Allweis and Tracey Lee-Joe and Tzvika Stein and Yael Bar Moshe and Yael Haramaty and Yaniv Carmel and Yishay Mor and Yoav Bar Sinai and Yoav Bergner and Yossi Matias and Yuri Lev},
  title = {{Towards Scalable Measurement of Durable Skills}},
  year = {2026},
  eprint = {2609.15864},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.15864},
}
Figure