Creative software crowdsourcing: from components and algorithm development to project concept formations

Wenjun Wu, Wei-Tek Tsai, Wei LiInt. J. Creative Comput 2013 · OpenReview

Introduction

Software engineering is a complex, creative endeavor that traditionally relies on centralized teams. However, the rise of crowdsourcing platforms like Topcoder and AppStori has introduced a new paradigm. This paper, published in the International Journal of Creative Computing, explores how the "wisdom of the crowd" can be harnessed for software development. By applying game theory, the authors analyze the dynamics of quality and participation, proposing a shift from purely competitive models to collaborative ecosystems.

The Participation Paradox and Quality Assurance

A central challenge in software crowdsourcing is the "participation paradox." While platforms boast hundreds of thousands of members, individual competitions often attract only a few submissions. This is largely due to "winner-takes-all" models that create a high-stakes environment, deterring potential contributors who perceive a low probability of success.

Furthermore, ensuring quality in a distributed, non-liable workforce is difficult. The authors frame quality through a "min-max" game theory lens: "Offense" involves finding bugs in others' work, while "Defense" involves minimizing bugs in one's own. In rigid models like Topcoder, this often results in a "Chicken Game" dynamic where participants may drop out if they face strong competition.

Data and Demographics

The study analyzes a significant dataset from Topcoder to understand the platform's reach and activity levels. Demographically, the platform is global, with a heavy concentration of active coders in China, the United States, and India.

Global distribution of active coders on the Topcoder platform.

The authors also examined the distribution of registrations across 1,524 design tasks and 848 development tasks. The data reveals a significant gap between initial interest (registration) and final execution (submission). For instance, while development tasks might see an average of nearly 24 registrations, the actual number of submissions is typically much lower, often around five.

Distribution of registrations for (a) 1,524 design tasks and (b) 848 development tasks.

Comparative Analysis and Proposed Model

The paper compares the waterfall-like, competitive Topcoder model with the agile, community-driven AppStori model. While Topcoder excels at component-level development through strict specifications, AppStori allows the crowd to participate in concept formation and funding, fostering a more collaborative "Coordination Game" environment.

Drawing from the IBM Cleanroom methodology of the 1980s—which separated specification, coding, and testing—the authors propose an "Updated Software Crowdsourcing Process." This model explicitly divides the crowd into developers (Crowd 1) and testers/evaluators (Crowd 2). This structure transforms "cut-throat" competition into a "win-win" ecosystem where structured cross-validation ensures quality.

Conclusion

The evolution of software crowdsourcing requires moving from simple [K × K] competitions to multi-layered [M × N × L] models involving developers, test-case generators, and test-runners. While challenges remain regarding liability and scaling these dynamics to massive infrastructure projects, the study demonstrates that every phase of the Software Development Life Cycle can be successfully crowdsourced. Future research will explore the optimal overlap between development and testing roles and how to integrate these models into mission-critical systems.

Summary updated

BibTeX
@article{wu2013creative,
  author = {Wenjun Wu and Wei-Tek Tsai and Wei Li},
  title = {{Creative software crowdsourcing: from components and algorithm development to project concept formations}},
  journal = {Int. J. Creative Comput.},
  year = {2013},
  url = {https://openreview.net/forum?id=ulHxbGcmlu},
}
Figure