Testing Worker Skill Structure Before Routing Bursty Annotation Work

Annotation projects have the same problem as cloud services. Demand arrives in bursts, a small trusted workforce is cheap per good label but slow to grow, and an open crowd can absorb spikes but brings unknown quality. Cloud systems handle this by splitting load into a steady base tier and an elastic burst tier. Crowdsourcing quality control has no equivalent discipline for deciding which incoming worker should take which task during a spike. The usual tool is a single reliability number per worker, which hides the fact that one person can be good at species identification and poor at reading medical images.

We propose to describe workers by a small number of latent skills, found with exploratory factor analysis. Exploratory factor analysis is a statistical method that explains correlations among many observed scores with a few hidden factors. The key step is to verify that skill structure on an independent sample of workers before any routing decision depends on it. Only a structure that replicates is used to route burst-tier workers, scored from a handful of screening items, to the task types their profile predicts they will do well. The central bet is that a replicated low-dimensional skill structure lets a short screening test stand in for long qualification. It also gives an explicit check for whether multidimensional routing is justified at all, a check current latent-factor crowdsourcing methods admit they lack.

Core Claim

The closest existing work is Crowdsourcing Utilizing Subgroup Structure of Latent Factor Modeling. It places workers and tasks in a shared latent space, clusters them, and labels each task category using only the worker subgroup with the strongest agreement. It reports accuracy gains of "3.1% to 29.5% on Bluebird" over MiSC and CBCC. Its "Assumption 1" says that each category has an identifiable stronger-agreement subgroup. The authors state that this assumption cannot be statistically tested, and the method learns and applies the structure inside one optimization loop.

Our claim is narrower and checkable. Suppose worker skill structure is discovered on one set of workers and then locked. If its loadings predict held-out workers' per-domain accuracy from a few screening items better than a single reliability score does, then multidimensional routing is warranted and should beat scalar routing on cost per correct label under bursty demand. If the structure does not replicate, the method falls back to scalar reliability, and it reports that it did so.

This differs from the JASA paper in three ways. Discovery and verification use disjoint workers. Profiles are used to admit and route new workers rather than to relabel a fixed matrix. The output includes a pass/fail test of the structural assumption.

Research Program

The worker model is a multidimensional item-response model. Item-response theory is the testing framework that models the probability of a correct answer from the respondent's ability and the item's difficulty. Here we write

$$\text{logit}\,P(y_{ij}=\text{correct}) = a_{d(i)}^\top \theta_j - b_i,$$

where $\theta_j$ is worker $j$'s vector of $K$ latent skills, $a_{d(i)}$ is the loading vector shared by all items in task domain $d(i)$, and $b_i$ is item $i$'s difficulty. With $K=1$ this reduces to scalar reliability of the kind EffiARA estimates. Efficient Annotator Reliability Assessment with EffiARA collapses each worker to one score and uses symbolic equations to plan annotator overlap. We reuse its overlap planning so that the gold-item design yields a dense enough worker-by-domain accuracy matrix.

Density matters because the file notes that sparse worker-by-task matrices often give correlation matrices that are not positive definite, which breaks factor extraction. Our design therefore gives every worker the same small block of gold items in each domain. Factor analysis runs on those dense blocks, never on the sparse live queue.

The pipeline follows split-sample psychometric validation, as used by Validity and Reliability of the Secondary Traumatic Stress Scale—Chinese Version. That study ran factor analysis on one half of its sample and confirmatory factor analysis on the other, and found a two-factor structure where the Western literature assumed three. Confirmatory factor analysis tests a fixed loading pattern against new data. The precedent constrains us usefully: structure can fail to transfer across populations, so we test transfer explicitly rather than assume it.

On the discovery workers, we choose $K$ by parallel analysis and rotate obliquely, because skills are expected to correlate. We lock the loadings. On the verification workers, we run two checks: confirmatory fit (CFI, RMSEA, SRMR) and a predictive check. In the predictive check, each verification worker's $\theta$ is estimated from only $m$ screening items, and the model must predict that worker's accuracy on the remaining domains.

The routing layer borrows the base/burst split from hybrid serving systems. MArk: Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving serves steady load on virtual machines and absorbs spikes with serverless functions. Towards Hybrid Cloud-assisted Crowdsourced Live Streaming: Measurement and Analysis keeps regular broadcasters on private datacenters and sends irregular ones to the public cloud using a "stable index."

Our analogue works as follows:

  • Reserved tier. Vetted workers with well-estimated profiles handle base demand.
  • Burst tier. When queue depth threatens a labeling deadline, open-crowd workers enter through an $m$-item screen. Each is assigned to the domain where its predicted accuracy, net of per-task wage, is highest. A worker is rejected if no domain clears the accuracy floor.

Neither cloud paper models label quality, and neither crowdsourcing paper models deadlines. The cloud literature supplies the allocation pattern, not the mechanism.

Baselines

Majority vote with random assignment. Every task goes to any available worker, and labels are aggregated by majority. This shows what the whole apparatus buys.

Scalar reliability routing. This is the same pipeline with $K=1$. Workers are screened on the same $m$ items and weighted by one reliability score, in the style of EffiARA. EffiARA reports raising F1-macro "from 0.691 to 0.740" over majority vote on RUC-MCD. This is the ablation that isolates our key ingredient, multidimensional profiles. It receives identical screening items, wages, and deadlines. Beating it means a lower cost per correct label at matched deadline compliance, with confidence intervals over repeated simulated surges that exclude zero.

Full cross-domain covariance selection. This is Cross-Domain-Aware Worker Selection with Training for Crowdsourced Annotation. It fits a multivariate normal over per-domain accuracy and adds learning-gain estimation and median elimination. It reports target-domain accuracy of 0.798 on RW-1, against 0.771 for median elimination and 0.764 for uniform sampling. It is the strongest competitor for cold-start routing because it already exploits cross-domain correlation. We run it on our data with the same screening budget and the same training batches. Beating it means matching or exceeding its RW-1 accuracy while using fewer screening items per admitted worker, because a $K$-factor model has far fewer parameters than a full covariance.

JASA subgroup latent-factor aggregation. This is the strongest offline label aggregator. It sees the full collected matrix, so it has an advantage our online router lacks. We compare final label accuracy. Our target is to come within its accuracy while meeting deadlines it was never designed for, not to beat it.

Why not just…?

Why not just use one reliability score per worker? This is the most serious threat. If skill correlations across domains are dominated by a single general factor, $K=1$ is enough and the extra machinery is waste. The file contains no evidence settling this for annotators. EffiARA explicitly treats one score as sufficient, and Sun et al. found cross-domain correlation worth modeling but did not test dimensionality. Our verification step is designed to answer this question directly. If parallel analysis picks $K=1$, or if $K>1$ predicts held-out accuracy no better than $K=1$, we report scalar routing as the recommendation.

Why not just evaluate the JASA method's label accuracy on held-out data instead of running factor analysis? Held-out label accuracy measures aggregation quality for workers already seen. It says nothing about whether a new worker's few screening answers place them correctly, which is exactly what burst routing needs. For that reason our "confirmatory" step is mostly the predictive check on new workers, with fit indices as a secondary diagnostic. If the fit indices and the predictive check disagree, we trust prediction.

Why not just use the full covariance of Sun et al.? With $D$ domains, a full covariance needs on the order of $D^2$ parameters, while a $K$-factor model needs about $DK$. The gap should matter most when screening is short and domains are many. Whether it matters in practice is an empirical question that the screening-length sweep in the experiments answers. If the full covariance matches the factor model at every $m$, the factor model's only remaining advantage is interpretability, and we say so.

Why not just hire more vetted workers? A reserved pool sized for peak demand sits idle in troughs. HCloud makes the same argument for compute: purely reserved provisioning is suboptimal under fluctuating load, as Delimitrou shows in HCloud: Resource-Efficient Provisioning in Shared Cloud Systems. We include a "reserved-only, sized to peak" arm so that the reader can see this cost directly.

Proposed Experiments

Existing datasets are too small for stable factor analysis. RW-1 and RW-2 have 27 and 35 workers, and the Prolific study in Incentivizing High-Quality Human Annotations with Golden Questions has 90. The primary experiment is therefore a new collection. We will recruit about 800 workers, each completing a fixed gold block across six to eight image and text domains. Image domains are modeled on Bluebird, Dog, and the RW plant/animal tasks. Text domains are modeled on RUC-MCD misinformation and news framing. Workers are split by recruitment wave into discovery and verification halves. A wave split rather than a random split makes replication across time a harder test than a random split.

Existing datasets serve as secondary checks:

  • RW-1/RW-2 allow a direct comparison with Sun et al.'s 0.798.
  • Bluebird allows comparison with the JASA results.
  • The NYCC humor ratings (yguooo/newyorker_caption_ranking) let us test whether a replicable rater structure exists in a subjective task without gold answers, using rater agreement as the indicator. There is no reported result on Lacuna for factor structure of NYCC raters.

The routing experiment replays the collected responses under simulated demand. Bursty arrivals are generated by a Markov-modulated Poisson process, the bursty-load generator used to stress MArk. Each simulated batch has a labeling deadline. We measure three outcomes: label accuracy, cost per correct label, and the fraction of batches completed by deadline.

Three checks decide the outcome.

  1. Replication. The structure must replicate across waves: Tucker congruence of loadings above 0.9, together with acceptable confirmatory fit.
  2. Prediction. With $m \in \{3, 5, 10, 20\}$ screening items, multidimensional profiles must predict held-out domain accuracy better than $K=1$ and no worse than the full covariance.
  3. Routing. Under surge, factor-based routing must lower cost per correct label relative to scalar routing at equal deadline compliance.

If the first check fails, the project's contribution becomes a negative result with practical value: evidence that annotator skill structure is not stable enough to route on.