When a sighted crowd worker writes a description of a photo for a blind user, nobody checks whether that description lets the user get anything done. Crowdsourcing platforms judge workers with hidden test items that have one correct answer. Captioning research scores text by its word overlap with other people's descriptions. Neither signal tracks usefulness. We propose to grade each description at collection time by a downstream test. A text-only reader model receives only the description and must answer held-out questions about the image. These include the question the blind photographer actually asked. Each question is first checked against a ceiling: a model with full image access tries it, and only questions that are answerable from the image count. The resulting score works as a hidden quality check and also sets the worker's bonus. Workers are paid for the information they transfer, not for agreeing with peers or passing trap items.
This matters most where existing quality control fails, which is assistive data. Photos taken by blind users are often blurred or badly framed. A large share of their questions cannot be answered at all (VizWiz Grand Challenge reports 28.6%). A good description can also be written in many legitimate ways. Under these conditions a worker who honestly writes "the label is cut off" looks the same as a careless one under trap-question schemes. Checking each question against the image-access ceiling separates "this image cannot support an answer" from "this worker did not transfer what was there." We treat accessibility as a constraint built into the data pipeline from the start, not a filter applied after a captioner is trained. Training data is where design decisions lock in.
Core Claim
Utility-graded collection consists of three steps: score descriptions by question answering from the description alone, measure each question against an image-access ceiling, and pay bonuses on that score. We claim it does two things better than current quality control. It separates diligent from low-effort describers more sharply. It also yields descriptions with which blind and low-vision (BLV) participants complete more of their own visual tasks.
The claim is specific to the incentive. A large part of any gain should come from workers changing what they write when they know usefulness is rewarded, not only from filtering descriptions afterward. If post-hoc filtering with the same score works just as well, the proactive part of the claim fails, and we say so in the analysis.
Research Program
The scoring step reuses the QA-on-caption protocol of CaptionQA. In that protocol, a text-only reader (Qwen2.5-72B) answers dense multiple-choice questions from a caption alone. CaptionQA applies this to captions written by models, and found that model captions lose 9% to over 32% of the image's usable information. We apply the same reader to crowd workers' descriptions, as a live grading and payment signal rather than a benchmark.
For an image $x$, let $Q_x$ be its probe questions. The pool contains the original VizWiz question plus questions written by a model with image access, in the style of CaptionQA's taxonomy, and checked by two independent annotators. Let $o(q)$ be 1 if a vision-language model given the image answers $q$ correctly in a majority of samples, and 0 otherwise. Let $a(d,q)$ be 1 if the reader given only description $d$ answers correctly. The utility of a description is
$$U(d) = \frac{\sum_{q \in Q_x} o(q)\, a(d,q)}{\sum_{q \in Q_x} o(q)}.$$
Questions that cannot be answered from the image drop out. A worker is never penalized for an image that does not support an answer. If no question survives, the item is used as a check on whether the worker correctly reports that the image is unanswerable.
Stangl et al. find that BLV readers value longer descriptions, while Balepur et al. find that raters favor brevity regardless of usefulness. $U$ on its own rewards dumping every detail, so we also test a cost-aware variant. This variant divides $U$ by estimated screen-reader listening time above a budget. Whether to use it is decided by the BLV study, not assumed.
The incentive borrows its contract structure from Liu et al.. They model annotation as a Stackelberg game, a leader-follower setup in which the requester sets a base pay and a bonus tied to a test on hidden golden questions. They pick golden questions with high certainty using a reward model, and they prove that workers who aim to "just barely pass" slow down quality estimation. Our setting departs from theirs in one key way. Their mechanism needs items with a single high-certainty answer, and that requirement breaks down on subjective, ambiguous assistive images. We replace answer certainty with ceiling-checked answerability, which is measured per question rather than assumed.
Because probe questions are never shown to workers, the test is hidden by construction. Every item is graded, so no special trap items exist for workers to spot.
Per-worker reliability is then aggregated with EffiARA. EffiARA computes recursive reliability scores from self-agreement and peer agreement and produces reliability-weighted soft labels. We feed it utility scores instead of label agreement. This also handles change over time: a rolling per-worker estimate tracks fatigue across a batch, which Liu et al. name as an open limitation.
The proactive-design literature supplies the motivation, not the method. Bi et al. document costly retrofitting when accessibility is deferred. Accessibility Beyond the Desktop argues that isolated add-on tools fail where native infrastructure succeeds. Dognin et al. report that the winning VizWiz 2020 captioner, at a CIDEr score of 81.04, was optimized against crowd references that never encoded functional details such as dosages or expiration dates. Our reading is that the defect sits in how the reference data was collected, so that is where we intervene.
Baselines
The naive reference point is standard platform collection. Workers are admitted by a 95% approval rating, descriptions are accepted unless obviously empty, and pay is flat. Graham et al. show this lets through workers who spend 0.22 seconds per item, against 10.22 seconds for diligent workers. In their study only 49% of workers passed a proper quality test.
The strongest existing quality-control competitor is the golden-question contract of Liu et al., adapted as closely as possible. Hidden check items are VizWiz questions whose 10 crowd answers agree, the closest analogue to their high-certainty selection. Workers on those items must answer the question in addition to describing the image. They report an 11.1% accuracy spread between diligent and non-diligent groups in a 90-worker Prolific trial, against 7.0% for instructed attention checks. That spread is the number to beat on worker separation.
The second strong competitor is agreement-based reliability with no utility signal. EffiARA is run on pairwise semantic similarity between descriptions of the same image. Cook et al. report that EffiARA soft labels raised F1-macro from 0.691 to 0.740 over majority vote on RUC-MCD.
The third competitor is metric-based filtering with SPURTS and contextual CLIPScore. Stangl et al. report correlations with BLV overall ratings of R = 0.41 and R = 0.201 respectively, against 0.075 for plain CLIPScore.
All arms see the same images, the same worker pool drawn at the same time, and the same total payment budget. Bonuses are redistributed rather than added, so pay amount is not a confound.
The key ablation uses $U$ purely as a post-hoc filter on flat-pay collection. This isolates the incentive effect from the selection effect. A second ablation removes the image-access ceiling, which tests whether ceiling-checking is what rescues the method on unanswerable images.
Beating the strongest baseline means a larger diligent versus non-diligent spread than the adapted golden-question arm, at equal budget. It also means a statistically significant gain in BLV task success, as defined below, over both the golden-question and EffiARA arms.
Why not just…?
Why not just use an LLM or VLM judge to rate each description? This is the most serious threat, because it needs no question bank. CaptionQA notes that model judges are non-deterministic and biased toward verbose text. That is exactly the bias that would be rewarded given Stangl et al.'s length findings. Still, nothing in the research file shows a judge failing on BLV task outcomes, so we test it as an extra arm. A VLM given the image rates each description 1–10, and we compare its correlation with BLV task success against $U$'s. If the judge matches $U$, the simpler method wins and we would recommend it. The ceiling-checking contribution would then reduce to handling unanswerable images, which we test separately.
Why not just have a frontier model write the descriptions and skip the crowd? VizWiz Grand Challenge reports that VizWiz-trained models reach about 47% accuracy, against a 75% human baseline, on blind users' real photos. Even GPT-5 captions lose about 9% of the usable information on cleaner images in CaptionQA. Our scoring is independent of who writes the description, so we include GPT-5-class descriptions as an arm. If they match or beat utility-graded crowd descriptions on VizWiz images, the remaining value of the method is as an audit of model captions. We would report that result plainly.
Why not just filter with agreement, timing, and attention checks, which are cheap and already work? Agreement measures whether workers say similar things, not whether they say the useful thing. Timing catches the 0.22-second optimizers that Graham et al. identified, but not a worker who writes fluent, generic text slowly. The EffiARA-on-similarity arm plus a timing filter tests this directly. Our prediction is that it removes speed-runners about as well as $U$ does, but falls short on the subgroup of fluent, low-information descriptions. We will report separation on that subgroup separately, using descriptions labeled by BLV participants.
Proposed Experiments
Collection uses about 1,500 VizWiz images, stratified so roughly a quarter are flagged unanswerable by their 10 crowd answers, plus Stangl et al.'s 54 image-context pairs to test contextual descriptions. Following Liu et al.'s design, workers on Prolific are randomly assigned to diligent and non-diligent conditions through instructions and time pressure, and each condition is crossed with the collection arms. The first outcome is worker separation: the gap in mean score between conditions and the area under the curve for detecting non-diligent workers, measured per arm. The sharpest Gap-4 test is the false-flag rate of honest "cannot tell" descriptions on unanswerable images. Trap-based schemes should flag these honest descriptions. Ceiling-checked $U$ should not.
The decisive check is a within-subject study with BLV participants, aiming for more than the 16 in Stangl et al. Each participant hears, through their own screen reader, descriptions produced by each arm for images paired with their original VizWiz question or a task such as finding the expiration date. We record answer correctness, time to answer, and self-reported confidence. Analysis uses item response theory, a statistical model that separates each participant's skill from each item's difficulty, following Balepur et al.'s approach. The same study fixes the length budget: if the cost-aware $U$ predicts BLV success better than raw $U$, it becomes the default.
Two robustness checks address CaptionQA's caveat that scores depend on the reader. The first rescores with a second reader of a different family and requires a rank correlation between readers above 0.8. The second compares raw scores and length-matched scores so that gains cannot be explained by longer text.
A final downstream test trains one captioner on each arm's accepted data. Each captioner is evaluated with VizWiz CIDEr, with held-out QA-on-caption utility, and on a subset of the BLV tasks. This shows whether the collection-time intervention carries through to the trained model.
References
- VizWiz Grand Challenge: Answering Visual Questions from Blind People
- CaptionQA: Is Your Caption as Useful as the Image Itself?
- Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metrics
- A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Users
- Incentivizing High-Quality Human Annotations with Golden Questions
- Efficient Annotator Reliability Assessment with EffiARA
- Accessibility in Software Practice: A Practitioner's Perspective
- Accessibility Beyond the Desktop
- Image Captioning as an Assistive Technology: Lessons Learned from VizWiz 2020 Challenge
- Crowd-Sourcing of Human Judgments of Machine Translation Fluency