Abstract
In high-stakes startup competitions, the quality of judge-venture matching is critical for ensuring fairness and identifying high-potential ventures. Traditionally, this process is managed manually by expert administrators—a method that is labor-intensive and difficult to scale. This paper evaluates the Hybrid Lexical–Semantic Similarity Ensemble (HLSE), an AI-driven approach designed to automate this matchmaking. In a blinded head-to-head study conducted during the Harvard President’s Innovation Challenge (PIC), the HLSE model achieved performance parity with human experts while reducing administrative overhead by over 90%.
Introduction
High-stakes entrepreneurial competitions, such as the Harvard President’s Innovation Challenge (PIC), serve as vital catalysts for innovation, distributing significant financial awards and prestige. The integrity of these competitions rests on the "matchmaking" process: pairing startup ventures with judges possessing the relevant domain expertise.
Historically, this task has been performed manually by program administrators. However, as the volume of applications grows, manual assignment becomes a significant bottleneck. Unlike academic peer review, which can leverage citation networks (e.g., the Toronto Paper Matching System), startup competitions rely on unstructured data such as pitch descriptions and short professional bios. This study investigates whether an algorithmic ensemble can replicate the nuanced judgment of human experts in this data-constrained environment.
Methodology
The researchers developed a two-stage pipeline to handle the complexity of startup domain matching.
1. Similarity Computation: The HLSE Model
The Hybrid Lexical–Semantic Similarity Ensemble (HLSE) addresses the limitations of single-model approaches by utilizing an ensemble of 47 base learners categorized into three types:
- Lexical (TF-IDF): These models capture specific, high-impact keywords (e.g., "fintech," "CRISPR") that define industry niches.
- Semantic (Transformers): Utilizing architectures like BERT, RoBERTa, and MPNet, these models capture the broader context and underlying business logic within bios and pitches.
- Hybrid Models: These combine the two approaches by using TF-IDF weights to emphasize critical tokens within dense transformer embeddings.
The ensemble uses a linear regression model trained on a manually curated "ground truth" dataset to determine the optimal weights for each learner, ensuring the system balances specific jargon with general context.
2. Assignment: PeerReview4All
Once similarity scores are generated, the system employs the PeerReview4All algorithm for the final assignment. Unlike standard optimization algorithms that maximize total similarity—often at the expense of niche ventures—PeerReview4All focuses on "max-min" fairness. This ensures that every venture, regardless of how specialized its field, receives high-quality matches.
Data and Experimental Design
The study utilized four years (2021–2024) of historical data from the Harvard PIC, encompassing venture applications, judge profiles (supplemented by LinkedIn data), and over 1,400 historical assignments. To train the HLSE ensemble, the researchers manually scored a subset of 105 judge-venture pairs on a 1–5 scale.
To validate the system, a blinded experiment was conducted. Judges were presented with venture assignments without knowing whether the match was generated by a human expert or the HLSE algorithm. They then rated the quality of these matches on a 5-point scale.
Key Results
Performance Parity
The primary finding of the study was the statistical indistinguishability between human and AI performance. Human-curated matches received an average score of 3.94, while AI-generated matches averaged 3.90. A Mann-Whitney U test (AUC = 0.48, p = 0.40) confirmed that there was no statistically significant difference in quality between the two groups.
Efficiency Gains
While quality remained constant, efficiency improved exponentially. The manual assignment process for a competition of this scale typically requires approximately 80 hours of expert labor. The HLSE-powered approach completed the same task in 5–7 hours, including the time required for data collection and preprocessing.
The "Sweet Spot" of Agreement
The study identified 82 instances where the human expert and the AI independently selected the same match. These overlapping assignments received the highest average rating (4.10), suggesting that the intersection of human intuition and algorithmic processing identifies the highest-quality matches.
Discussion and Conclusion
The HLSE framework demonstrates that ensembling lexical and semantic models provides a robust "safety net" for automated matching. During a post-deployment audit, a minor bug was discovered in the hybrid model implementation; however, the ensemble's overall performance remained stable as the other 40+ models compensated for the error.
While the results are promising, limitations remain regarding data privacy and the potential for applicants to "game" the algorithm via keyword stuffing. Future work will focus on making these models more robust against adversarial manipulation and exploring "AI-assisted" workflows, where algorithms provide a baseline for final human refinement. Ultimately, this research suggests that AI can effectively remove the administrative burden of matchmaking in high-stakes environments without sacrificing the quality of expert judgment.
Summary updated