Who Is a Better Matchmaker? Human vs. Algorithmic Judge Assignment in a High-Stakes Startup Competition

Yang Xi, Orelia Pi, Miaomiao Zhang, Rebecca Xiong, Joseph M. Lane, Nihar B. ShahAAAI 2026 · DOI

Abstract

There is growing interest in applying artificial intelligence (AI) to automate and support complex decision-making tasks. However, it remains unclear how algorithms compare to human judgment in contexts requiring semantic understanding and domain expertise. We examine this in the context of the judge assignment problem, matching submissions to suitably qualified judges. Specifically, we tackled this problem at the Harvard President’s Innovation Challenge, the university’s premier venture competition awarding over $500,000 to student and alumni startups. This setting represents a real-world environment where high-quality judge assignment is essential. We developed an AI-based judge assignment algorithm, the Hybrid Lexical-Semantic Similarity Ensemble (HLSE), and deployed it at the competition. We then evaluated its performance against human expert assignments using blinded match-quality scores from judges on 309 judge-venture pairs. Using a Mann-Whitney U statistic-based test, we found no statistically significant difference in assignment quality between the two approaches (AUC=0.48, p=0.40); on average, algorithmic matches were rated 3.90 and manual matches 3.94 on a 5-point scale, where 5 indicates an excellent match. Furthermore, manual assignments that previously required a full week could be automated in several hours by the algorithm during deployment. These results demonstrate that HLSE achieves human-expert-level matching quality while offering greater scalability and efficiency, underscoring the potential of AI-driven solutions to support and enhance human decision-making for judge assignment in high-stakes settings.

The Challenge of Startup Matchmaking

In high-stakes environments like startup competitions or grant funding, the "matchmaker" role is critical. Pairing a venture (e.g., a biotech startup) with a judge who has the specific expertise to evaluate it is essential for fairness and quality. While academic peer review has successfully used AI to assign reviewers to papers, those systems rely heavily on citation networks and publication histories—data that simply doesn't exist for most startup founders or industry judges.

Startup expertise is "tacit" and multi-dimensional, spanning both technical hurdles and business viability. Historically, this has forced organizers to perform assignments manually, a process that is slow, prone to knowledge loss due to staff turnover, and nearly impossible to scale as competitions grow.

The HLSE Methodology

To address this, researchers developed the Hybrid Lexical-Semantic Similarity Ensemble (HLSE). Rather than relying on a single algorithm, HLSE uses an ensemble approach that integrates three types of text representations:

  1. Lexical (TF-IDF): Traditional keyword matching, excellent for identifying specific industry terms like "CRISPR" or "fintech."
  2. Semantic (Transformer-based Embeddings): Using models like BERT, RoBERTa, and MPNet to capture the broader meaning and context of judge bios and venture descriptions.
  3. Hybrid: A novel approach that uses TF-IDF scores to "weight" transformer embeddings, highlighting the most informative words within a semantic context.

The team trained the ensemble's weights using a "gold standard" dataset of 105 pairs manually scored by experts. For the final assignment, they utilized the PeerReview4All algorithm, which focuses on "max-min" fairness to ensure that even niche ventures receive high-quality judges.

Experimental Validation

The researchers conducted a blinded head-to-head experiment during the 2025 Harvard President’s Innovation Challenge. The process involved independent assignments from both the HLSE algorithm and human experts, which were then released to judges who were blind to the assignment method.

The four-step experimental workflow used to compare HLSE and human expert assignments.

As shown in the workflow above, the study collected 309 total match quality scores from 59 responding judges. This rigorous setup allowed for a direct statistical comparison of the two approaches.

Key Results: Quality and Efficiency

The results revealed that the algorithm performed at a level indistinguishable from human experts in terms of quality, while providing massive gains in efficiency.

  • Match Quality: On a 5-point scale, human expert matches averaged 3.94, while algorithmic matches averaged 3.90. A Mann-Whitney U test confirmed this difference was statistically insignificant (p = 0.40, AUC = 0.48).
  • Consensus: The highest-rated matches (4.10/5) occurred when the human expert and the algorithm independently chose the same judge, suggesting the AI successfully captured the "tacit" logic used by humans.
  • Efficiency: The most dramatic improvement was in time management.
Comparison of time requirements for algorithmic and manual processes across experimental and non-experimental tracks.

As illustrated in the timeline comparison, manual assignment for the 2025 tracks required 8–12 hours of active work. The AI-driven workflow (including data collection and refinement) reduced this significantly. More importantly, it replaced a process that historically took an entire week of staff time in previous years, allowing organizers to focus on high-level strategy rather than spreadsheet logistics.

Conclusion and Future Work

These results demonstrate that HLSE achieves human-expert-level matching quality while offering greater scalability. However, limitations remain. Future work should investigate potential "gaming of the system" by applicants using SEO-like tactics in their descriptions and explore the optimal balance of AI-generated drafts with human-in-the-loop oversight. Testing HLSE across larger, more diverse datasets will further validate its generalizability for high-stakes decision-making.

Summary updated

Abstract

There is growing interest in applying artificial intelligence (AI) to automate and support complex decision-making tasks. However, it remains unclear how algorithms compare to human judgment in contexts requiring semantic understanding and domain expertise. We examine this in the context of the judge assignment problem, matching submissions to suitably qualified judges. Specifically, we tackled this problem at the Harvard President’s Innovation Challenge, the university’s premier venture competition awarding over $500,000 to student and alumni startups. This setting represents a real-world environment where high-quality judge assignment is essential. We developed an AI-based judge assignment algorithm, the Hybrid Lexical-Semantic Similarity Ensemble (HLSE), and deployed it at the competition. We then evaluated its performance against human expert assignments using blinded match-quality scores from judges on 309 judge-venture pairs. Using a Mann-Whitney U statistic-based test, we found no statistically significant difference in assignment quality between the two approaches (AUC=0.48, p=0.40); on average, algorithmic matches were rated 3.90 and manual matches 3.94 on a 5-point scale, where 5 indicates an excellent match. Furthermore, manual assignments that previously required a full week could be automated in several hours by the algorithm during deployment. These results demonstrate that HLSE achieves human-expert-level matching quality while offering greater scalability and efficiency, underscoring the potential of AI-driven solutions to support and enhance human decision-making for judge assignment in high-stakes settings.
BibTeX
@inproceedings{xi2026who,
  author = {Yang Xi and Orelia Pi and Miaomiao Zhang and Rebecca Xiong and Joseph M. Lane and Nihar B. Shah},
  title = {{Who Is a Better Matchmaker? Human vs. Algorithmic Judge Assignment in a High-Stakes Startup Competition}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i47.41449},
}

Loading related content...

Figure