Abstract
Predictive modeling competitions provide a new data mining approach that leverages crowds of data scientists to examine a wide variety of predictive models and build the best performance model. In this paper, we report the results of a study conducted on CrowdSolving, a platform for predictive modeling competitions in Japan. We hosted a competition on a link prediction task and observed that (i) the prediction performance of the winner significantly outperformed that of a state-of-the-art method, (ii) the aggregated model constructed from all submitted models further improved the final performance, and (iii) the performance of the aggregated model built only from early submissions nevertheless overtook the final performance of the winner.
Introduction
In data science, the "No Free Lunch" theorem suggests that no single predictive model is universally superior. Achieving peak performance usually requires a grueling process of feature engineering, model selection, and hyperparameter tuning. While a single data scientist might struggle to explore this vast "model space," crowdsourcing platforms like Kaggle or Japan’s CrowdSolving allow a host to outsource this exploration to hundreds of experts. This study quantifies the "power of the crowd" by evaluating how crowdsourced models compare to specialized algorithms and investigating whether combining multiple submissions yields superior results.
Methodology
The researchers hosted a link prediction competition on the CrowdSolving platform. Participants were tasked with predicting whether a hyperlink exists between two Wikipedia articles based on their attributes. The dataset included 23,269 articles with 39,541-dimensional attributes.
- The Baseline: Participant results were compared against a state-of-the-art (SOTA) link prediction method using nonlinear dimension reduction (NLDR).
- The Aggregation Technique: To combine the crowd's wisdom, the authors used stacking. Predictions from all 134 submitted models served as input features for a "meta-classifier"—specifically, a Gradient Tree Boosting model.
- Evaluation: Performance was measured using the Area Under the ROC Curve (AUC), where 1.0 represents perfect classification.
Key Results and Insights
Heuristics Beat "Pure" SOTA
Within just four days, the crowd's models—often relying on clever feature engineering and simple heuristics—surpassed the sophisticated NLDR baseline (AUC 0.7166). This highlights that domain-specific "human inspiration" often outweighs algorithmic complexity in practical applications.
The Ensemble Advantage
The aggregated model (AGG) consistently outperformed the best individual participant. By the end of the competition, the AGG model reached an AUC of 0.9823, significantly higher than the winner’s individual peak of 0.9459.
The "Expedition" Effect
This is the study's most striking finding regarding efficiency. While the winning participant took 33 days of iterative refinement to reach their peak score, the researchers found that by aggregating the submissions available on Day 8, they created a model that already outperformed the winner's final Day 33 score.
As illustrated in Figure 1, the aggregated model (black line) maintains a dominant lead throughout the timeline. On the very first day, the AGG model (using only 5 submissions) achieved an AUC of 0.8479—nearly matching what the best individual could do after five days of work. For a business or researcher, this means crowdsourcing can drastically shorten the timeline for high-quality model development.
Conclusion
The study demonstrates that the real value of crowdsourcing lies in the diversity of the models produced rather than just the single highest score. By ensembling early submissions, a host can achieve "winner-level" performance in a fraction of the time required for a full competition. Future work may explore incentive designs that further encourage this model diversity to maximize the benefits of aggregation.
Summary updated