How to Data in Datathons

Carlos Mougan, Richard Plant, Clare Teng, Marya Bazzi, Alvaro Cabrejas Egea, Ryan Sze-Yin Chan, David Salvador Jasin, Martin Stoffel, Kirstie J. Whitaker, Jules ManserNeurIPS 2023

Datathons are high-stakes, short-duration sprints where data quality can make or break a project. While data science hackathons are booming, many organizations struggle to provide data that leads to meaningful results. Often, participants spend 90% of their limited time cleaning messy files, navigating privacy red tape, or realizing the data doesn't actually answer the proposed question.

This paper identifies a lack of standardized guidelines for the "pre-hackathon" phase and aims to bridge the gap between simply having data and having data that is fit for purpose in a collaborative, time-constrained environment.

The Data Assessment Matrix

Researchers from The Alan Turing Institute leveraged their experience organizing over 80 datathon challenges—specifically their "Data Study Groups" (DSGs)—to create a qualitative framework for organizers.

Distribution of Challenge Owners and Domains across Turing Data Study Groups

The core of the paper is the Data Assessment Matrix, which categorizes data into four levels—Insufficient, Developing, Functional, and Optimal—across five key dimensions:

  1. Appropriateness: Is the data relevant to the specific research questions?
  2. Readiness: Is the data cleaned, documented, and easy to access?
  3. Reliability: How biased is the data, and are those biases accounted for?
  4. Sensitivity: What are the privacy risks (using a Tier 0-4 classification)?
  5. Sufficiency: Is there enough data to achieve statistical significance or train a model?

Key Insights for Organizers

  • The "Functional" Threshold: For a datathon to succeed, data doesn't need to be "Optimal" (perfectly clean and bias-free). It just needs to reach the "Functional" level—where biases are known and accounted for, and data is unified enough to allow for analysis within a week.
  • The Pre-Hackathon Pivot: One of the most important takeaways is that the framework allows organizers to pivot early. If a dataset scores "Developing" in Appropriateness, organizers should change the challenge questions to match the data rather than forcing a failing project.
  • Sensitivity vs. Friction: High sensitivity (Tier 3 or 4) often introduces "system friction" (e.g., restricted servers, no internet access) that can kill the collaborative spirit of a datathon. The paper suggests anonymization or synthetic data (Tier 2) as a sweet spot for these events.

Results in Practice

The authors mapped 10 recent DSG reports to their matrix to validate the framework. These case studies spanned diverse fields, including plankton image classification (Cefas), NLP for conservation threats (WWF), satellite imagery for seal counting (British Antarctic Survey), accessibility prediction for urban travel (CityMaaS), and AI-supported diagnosis for colorectal cancer (Odin Vision).

Heatmap mapping 10 recent Data Study Group reports to the Data Assessment Matrix

Their analysis showed that Appropriateness was generally high (Functional/Optimal), meaning organizers are typically good at picking relevant data. However, Readiness and Sufficiency were the most common bottlenecks. Even in successful datathons, participants often encountered "Developing" levels of readiness, requiring significant on-the-fly cleaning.

For example, in the CityMaaS case study, a "Developing" score in Sufficiency (due to a lack of routing data) forced participants to scramble for external open-source data to make the project viable. The authors also acknowledge a publication bias in these results: most public reports represent "Optimal" or "Functional" data because failed datathons rarely result in published technical reports.

Limitations and Future Directions

The primary limitation of this framework is that it is qualitative and subjective; it relies heavily on the expertise of the organizers to "score" the data. Additionally, the study is based on the collaborative DSG model and may need adjustments for competitive-style hackathons where teams do not share insights.

Future work involves automating parts of this assessment—perhaps through "data nutrition labels"—and expanding the framework to include ethical dimensions beyond just privacy and bias.

Summary updated

BibTeX
@inproceedings{mougan2023how,
  author = {Carlos Mougan and Richard Plant and Clare Teng and Marya Bazzi and Alvaro Cabrejas Egea and Ryan Sze-Yin Chan and David Salvador Jasin and Martin Stoffel and Kirstie J. Whitaker and Jules Manser},
  title = {{How to Data in Datathons}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
}
Figure