Promoting Research Collaboration with Open Data Driven Team Recommendation in Response to Call for Proposals

Siva Likitha Valluru, Biplav Srivastava, Sai Teja Paladi, Siwen Yan, Sriraam Natarajan

Abstract

Building teams and promoting collaboration are two very common business activities. An example of these are seen in the TeamingForFunding problem, where research institutions and researchers are interested to identify collaborative opportunities when applying to funding agencies in response to latter's calls for proposals. We describe a novel system to recommend teams using a variety of AI methods, such that (1) each team achieves the highest possible skill coverage that is demanded by the opportunity, and (2) the workload of distributing the opportunities is balanced amongst the candidate members. We address these questions by extracting skills latent in open data of proposal calls (demand) and researcher profiles (supply), normalizing them using taxonomies, and creating efficient algorithms that match demand to supply. We create teams to maximize goodness along a novel metric balancing short- and long-term objectives. We validate the success of our algorithms (1) quantitatively, by evaluating the recommended teams using a goodness score and find that more informed methods lead to recommendations of smaller number of teams but higher goodness, and (2) qualitatively, by conducting a large-scale user study at a college-wide level, and demonstrate that users overall found the tool very useful and relevant. Lastly, we evaluate our system in two diverse settings in US and India (of researchers and proposal calls) to establish generality of our approach, and deploy it at a major US university for routine use.

Introduction

In the high-stakes world of academic funding, responding to a Request for Proposal (RFP) from agencies like the NSF or NIH is a complex "TeamingForFunding" problem. Researchers need to quickly assemble multi-disciplinary teams that cover all technical requirements of a call. However, finding the right collaborators is often hindered by "information silos," time constraints, and the sheer volume of available data. Manual team formation is prone to human bias and often relies on existing social circles rather than optimal skill alignment.

The ULTRA system (University-Lead Team Builder from RFPs and Analysis) addresses this challenge by algorithmically matching the "demand" (skills required by an RFP) with the "supply" (latent skills in researcher profiles) to form teams that are qualified, balanced, and resilient.

Methodology

The ULTRA system utilizes four distinct recommendation methods:

  1. M0 (Random Baseline): Randomly samples researchers to form teams.
  2. M1 (String Matching): Extracts technical terms from RFPs and researcher profiles to find direct overlaps.
  3. M2 (Taxonomical Matching): Uses the ACM Computing Classification System (CCS) to understand semantic relationships between terms.
  4. M3 (Boosted Bandit): Treats team recommendation as a contextual bandit problem, using relational boosted trees to learn complex fit rules from historical patterns.

To evaluate these teams, we introduced a "Goodness Score," a weighted metric balancing coverage (percentage of required skills satisfied), set size (preference for smaller teams), redundancy (preventing single points of failure), and k-robustness (viability if members become unavailable).

Quantitative Evaluation

The system was tested in two primary environments: a US setting with 434 RFPs and 200 faculty members, and an India setting with 100 RFPs and 46 researchers. A key finding was the quality-quantity trade-off: as methods became more informed (M0 to M3), the volume of recommendations decreased while the average quality (Goodness Score) significantly increased.

Performance comparison of the four recommendation methods showing the quality-volume trade-off.

As shown in the results, the Boosted Bandit (M3) method achieved a Goodness Score of 0.5295, nearly six times higher than the Random (M0) baseline of 0.0879. This demonstrates that AI acts as an effective filter, providing high-quality "gold-standard" suggestions rather than a flood of mediocre ones.

User Study and Qualitative Feedback

A large-scale user study involving 212 responses at the University of South Carolina validated the system's practical utility. Users evaluated the recommendations based on relevance and usefulness across three different use cases.

Sankey diagram illustrating user ratings for relevance and usefulness across different use cases and methods.

The study showed that 90.09% of users found the outputs relevant, and 97.64% found the tool useful or somewhat useful. Interestingly, even some "irrelevant" suggestions from the random baseline were deemed useful by researchers for sparking serendipitous connections or introducing colleagues they hadn't met before.

Conclusion and Future Work

The ULTRA system successfully automates the complex process of team formation for research funding. While effective, future work will address "ground truth" lags in grant success data and the potential for data decay in public profiles. Future iterations will also incorporate "connection strength" (prior collaboration history) and fairness constraints to ensure balanced workloads across the research community.

Summary updated

Abstract

Building teams and promoting collaboration are two very common business activities. An example of these are seen in the TeamingForFunding problem, where research institutions and researchers are interested to identify collaborative opportunities when applying to funding agencies in response to latter's calls for proposals. We describe a novel system to recommend teams using a variety of AI methods, such that (1) each team achieves the highest possible skill coverage that is demanded by the opportunity, and (2) the workload of distributing the opportunities is balanced amongst the candidate members. We address these questions by extracting skills latent in open data of proposal calls (demand) and researcher profiles (supply), normalizing them using taxonomies, and creating efficient algorithms that match demand to supply. We create teams to maximize goodness along a novel metric balancing short- and long-term objectives. We validate the success of our algorithms (1) quantitatively, by evaluating the recommended teams using a goodness score and find that more informed methods lead to recommendations of smaller number of teams but higher goodness, and (2) qualitatively, by conducting a large-scale user study at a college-wide level, and demonstrate that users overall found the tool very useful and relevant. Lastly, we evaluate our system in two diverse settings in US and India (of researchers and proposal calls) to establish generality of our approach, and deploy it at a major US university for routine use.
BibTeX
@inproceedings{valluru2024promoting,
  author = {Siva Likitha Valluru and Biplav Srivastava and Sai Teja Paladi and Siwen Yan and Sriraam Natarajan},
  title = {{Promoting Research Collaboration with Open Data Driven Team Recommendation in Response to Call for Proposals}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=UkaO1m6Etk},
}
Figure