Abstract
Active Learning (AL) is a machine learning technique where the model selectively queries the most informative data points for labeling by human experts. Integrating AL with crowdsourcing leverages crowd diversity to enhance data labeling but introduces challenges in consensus and privacy. This poster presents CrowdAL, a blockchain-empowered crowd AL system designed to address these challenges. CrowdAL integrates blockchain for transparency and a tamper-proof incentive mechanism, using smart contracts to evaluate crowd workers’ performance and aggregate labeling results, and employs zero-knowledge proofs to protect worker privacy.
Introduction
Active Learning (AL) optimizes model training by querying labels for the most informative data points. While crowdsourcing provides a cost-effective alternative to high-cost experts, it introduces significant challenges:
- Trust and Transparency: Ensuring fair payment for workers and high-quality labels for requesters.
- Consensus: Aggregating labels from diverse, potentially noisy annotators into a reliable ground truth.
- Privacy: Protecting worker identities and work history while still allowing for performance-based rewards.
CrowdAL addresses these issues by combining blockchain technology with Zero-Knowledge Proofs (ZKPs) to create a decentralized, privacy-preserving framework for iterative data labeling.
System Architecture
The CrowdAL architecture consists of a Smart Contract Factory and an AL Server, coordinating the flow of data and rewards.
Core Components
- Blockchain & Smart Contracts: Ethereum-based contracts (
JobManagementandJobInstance) manage job lifecycles and reward distribution. Label aggregation is performed via majority voting within the contract. - Active Learning Loop: The AL server handles the ML lifecycle, including querying unlabeled data and retraining the model based on crowd-provided labels.
- Zero-Knowledge Proofs (ZKPs): Utilizing the Groth-16 scheme, CrowdAL enables anonymous label submission and performance evaluation. Workers can prove their participation and accuracy without revealing their identity or specific contributions.
- Incentive Mechanism: Rewards are distributed based on accuracy, determined by comparing worker labels against the model's predictions after training on aggregated data.
Evaluation and Results
The feasibility of CrowdAL was evaluated on a local Ethereum chain, focusing on the computational overhead of its smart contracts.
The primary metric for evaluation was Gas Usage. While the system successfully deployed six core contracts, the results highlight a significant "privacy tax." ZKP-related functions, such as ZKJobInstance (anonymous submission) and ZKJobScore (anonymous scoring), are considerably more expensive than their standard counterparts. For instance, ZKJobInstance requires approximately 2.5 million gas units, nearly double the 1.4 million required for non-ZKP versions.
Discussion and Conclusion
CrowdAL demonstrates that blockchain can effectively decentralize the "oracle" in Active Learning, removing single points of failure and bias. By using the model itself as an evaluator, the system creates a self-reinforcing quality control loop. However, the high on-chain costs associated with ZKPs remain a barrier to mainnet deployment. Future work will focus on optimizing these cryptographic operations and testing the system's scalability with larger datasets and worker pools.
Summary updated