CrowdAL: Towards a Blockchain-empowered Active Learning System in Crowd Data Labeling

Shaojie Hou, Yuandou Wang, Zhiming Zhao

Abstract

Active Learning (AL) is a machine learning technique where the model selectively queries the most informative data points for labeling by human experts. Integrating AL with crowdsourcing leverages crowd diversity to enhance data labeling but introduces challenges in consensus and privacy. This poster presents CrowdAL, a blockchain-empowered crowd AL system designed to address these challenges. CrowdAL integrates blockchain for transparency and a tamper-proof incentive mechanism, using smart contracts to evaluate crowd workers’ performance and aggregate labeling results, and employs zero-knowledge proofs to protect worker privacy.

Introduction

Active Learning (AL) optimizes model training by querying labels for the most informative data points. While crowdsourcing provides a cost-effective alternative to high-cost experts, it introduces significant challenges:

  1. Trust and Transparency: Ensuring fair payment for workers and high-quality labels for requesters.
  2. Consensus: Aggregating labels from diverse, potentially noisy annotators into a reliable ground truth.
  3. Privacy: Protecting worker identities and work history while still allowing for performance-based rewards.

CrowdAL addresses these issues by combining blockchain technology with Zero-Knowledge Proofs (ZKPs) to create a decentralized, privacy-preserving framework for iterative data labeling.

System Architecture

The CrowdAL architecture consists of a Smart Contract Factory and an AL Server, coordinating the flow of data and rewards.

The CrowdAL system architecture illustrating the interaction between data owners, the blockchain, the AL server, and crowd workers.

Core Components

  • Blockchain & Smart Contracts: Ethereum-based contracts (JobManagement and JobInstance) manage job lifecycles and reward distribution. Label aggregation is performed via majority voting within the contract.
  • Active Learning Loop: The AL server handles the ML lifecycle, including querying unlabeled data and retraining the model based on crowd-provided labels.
  • Zero-Knowledge Proofs (ZKPs): Utilizing the Groth-16 scheme, CrowdAL enables anonymous label submission and performance evaluation. Workers can prove their participation and accuracy without revealing their identity or specific contributions.
  • Incentive Mechanism: Rewards are distributed based on accuracy, determined by comparing worker labels against the model's predictions after training on aggregated data.

Evaluation and Results

The feasibility of CrowdAL was evaluated on a local Ethereum chain, focusing on the computational overhead of its smart contracts.

Average gas cost for various smart contract functions, highlighting the overhead of ZKP-related operations.

The primary metric for evaluation was Gas Usage. While the system successfully deployed six core contracts, the results highlight a significant "privacy tax." ZKP-related functions, such as ZKJobInstance (anonymous submission) and ZKJobScore (anonymous scoring), are considerably more expensive than their standard counterparts. For instance, ZKJobInstance requires approximately 2.5 million gas units, nearly double the 1.4 million required for non-ZKP versions.

Discussion and Conclusion

CrowdAL demonstrates that blockchain can effectively decentralize the "oracle" in Active Learning, removing single points of failure and bias. By using the model itself as an evaluator, the system creates a self-reinforcing quality control loop. However, the high on-chain costs associated with ZKPs remain a barrier to mainnet deployment. Future work will focus on optimizing these cryptographic operations and testing the system's scalability with larger datasets and worker pools.

Summary updated

Abstract

Active Learning (AL) is a machine learning technique where the model selectively queries the most informative data points for labeling by human experts. Integrating AL with crowdsourcing leverages crowd diversity to enhance data labeling but introduces challenges in consensus and privacy. This poster presents CrowdAL, a blockchain-empowered crowd AL system designed to address these challenges. CrowdAL integrates blockchain for transparency and a tamper-proof incentive mechanism, using smart contracts to evaluate crowd workers’ performance and aggregate labeling results, and employs zeroknowledge proofs to protect worker privacy.
BibTeX
@misc{hou2025crowdal,
  author = {Shaojie Hou and Yuandou Wang and Zhiming Zhao},
  title = {{CrowdAL: Towards a Blockchain-empowered Active Learning System in Crowd Data Labeling}},
  year = {2025},
  eprint = {2503.00066},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.00066},
}
Figure