Joint Annotator-and-Spectrum Allocation in Wireless Networks for Crowd Labelling

Xiaoyang Li, Guangxu Zhu, Kaiming Shen, Wei Yu, Yi Gong, Kaibin Huang

Abstract

Modern Artificial Intelligence relies on massive amounts of labelled data, often sourced through crowd labelling. However, performing this over wireless networks introduces significant communication overhead and resource management challenges. This paper introduces a framework for wireless crowd labelling that jointly optimizes data encoding rates, annotator clustering, and spectrum allocation. By balancing the trade-offs between compression distortion and the need for redundant labelling, the proposed Joint Annotator-and-Sub-channel Allocation (JASA) algorithm maximizes labelling throughput—the total number of accurately labelled objects—under strict radio and human resource constraints.

Introduction: The Challenge of Wireless Crowd Labelling

The "fuel" for modern AI is high-quality labelled data, but the labelling process remains a laborious and expensive bottleneck. While outsourcing tasks to a "crowd" of mobile users is a scalable solution, doing so over wireless networks creates a fundamental three-way tradeoff:

  1. Encoding Rate: Compressing data (e.g., images or video) saves bandwidth but introduces distortion, which increases the likelihood of human labelling errors.
  2. Repetition Labelling: To mitigate errors caused by distortion or human fallibility, the same object must be sent to multiple annotators. A "majority vote" then determines the final label.
  3. Resource Allocation: Both wireless spectrum (sub-channels) and the number of available annotators are finite resources that must be managed efficiently.

The central problem is determining how to distribute objects to specific annotators, at what compression level, and on which frequency to maximize throughput without sacrificing accuracy.

System Model and Methodology

The authors model the system as a three-stage operation: source encoding, object multicasting, and crowd labelling.

System operations and network architecture for wireless crowd labelling.

To address the NP-hard nature of this joint optimization, the paper introduces several key strategies:

  • Optimal Sequential Clustering: The authors prove that the most efficient way to group annotators is by their Signal-to-Noise Ratio (SNR) in descending order. This significantly simplifies the search space.
  • JASA Algorithm: A Joint Annotator-and-Sub-channel Allocation (JASA) algorithm is developed using a branch-and-bound tree-search approach to find optimal solutions for fading channels.
  • Truncated Channel Inversion (TCI): To further reduce complexity, the authors propose using power control to equalize the channels of different annotators. This transforms the problem into a 2D Knapsack Problem, solvable via Dynamic Programming in pseudo-polynomial time.

Key Insights: The Compression-Redundancy Tradeoff

A primary contribution of this work is the mathematical link established between data distortion and crowd redundancy. If an image is heavily compressed to save bandwidth, the system must "pay" for that savings by hiring more annotators to label the same image to maintain a target accuracy level.

Furthermore, the optimal strategy "flips" depending on resource scarcity:

  • Bandwidth-Constrained: If spectrum is scarce, the system should maximize compression and use a larger crowd to correct the resulting errors.
  • Annotator-Constrained: If human resources are scarce, the system should transmit high-quality, uncompressed data to ensure each individual label is highly reliable, thereby minimizing the need for redundant votes.

Results and Performance

Using synthetic simulations with i.i.d. Rayleigh fading channels and standard rate-distortion models, the authors demonstrated that:

  • Throughput Gains: Joint allocation (JASA) significantly outperforms decoupled allocation strategies that optimize radio and human resources independently.
  • Efficiency: The Dynamic Programming approach (using TCI) offers a practical alternative to the tree search, providing near-optimal performance with much lower computational overhead as the number of annotators grows.
  • Optimal SNR Targeting: In the TCI model, there exists an "optimal targeted SNR." Setting this target too low wastes bandwidth, while setting it too high excludes too many capable annotators, both of which degrade total throughput.

Limitations and Future Directions

While the framework provides a robust foundation, several areas remain for future exploration. The current model assumes "flat" fading and uniform annotator skill levels. In reality, annotators vary in accuracy regardless of data quality. Additionally, the paper treats annotators as a static resource; future work could incorporate incentive mechanisms (e.g., financial rewards) to ensure participation. Other potential extensions include the use of spatial beamforming (MIMO) and more sophisticated matching algorithms between specific tasks and annotator channel conditions.

Summary updated

Abstract

The massive sensing data generated by Internet-of-Things will provide fuel for ubiquitous artificial intelligence (AI), automating the operations of our society ranging from transportation to healthcare. The implementation of ubiquitous AI, however, entails labelling of an enormous amount of data prior to the training of AI models via supervised learning. To tackle this challenge, we explore a new direction called wireless crowd labelling, which involves downloading data to many imperfect mobile annotators for repetition labelling with an aim of exploiting multicasting in wireless networks. In this cross-disciplinary area, the rate-distortion theory and the principle of repetition labelling for accuracy improvement together give rise to a new tradeoff between radio-and-annotator resources under a constraint on labelling accuracy. Building on the tradeoff and aiming at maximizing the labelling throughput, this work focuses on the joint optimization of encoding rate, annotator clustering, and sub-channel allocation, which results in an NP-hard integer programming problem. To devise an efficient solution approach, we establish an optimal sequential annotator-clustering scheme based on the order of decreasing signal-to-noise ratios, thereby allowing the optimal solution to be found by an efficient tree search. This solution can be further simplified when the channels are symmetric. Alternatively, the optimization problem can be recognized as a knapsack problem, which can be efficiently solved in pseudo-polynomial time by means of dynamic programming. In addition, the optimal polices are derived for the annotator constrained and spectrum constrained cases. Last, simulation results are presented to demonstrate the significant throughput gains based on the optimal solution compared with decoupled allocation of the two types of resources.
BibTeX
@article{li2020joint,
  author = {Xiaoyang Li and Guangxu Zhu and Kaiming Shen and Wei Yu and Yi Gong and Kaibin Huang},
  title = {{Joint Annotator-and-Spectrum Allocation in Wireless Networks for Crowd Labelling}},
  journal = {IEEE Trans. Wireless Communications},
  year = {2020},
  doi = {10.1109/twc.2020.3000241},
}
Figure