Teaching the Crowd with the Better Scan: Cross-Modal Feedback for Months-Long Medical Labeling

Many medical images are hard to read because sick and healthy tissue look nearly identical in the standard scan. A flat polyp under ordinary white light and a hypodense metastasis on routine CT are two examples. Often a second, more expensive modality separates them cleanly. Narrow-band imaging, an endoscopy mode that filters light to highlight surface vessels, does this for polyps, and spectral CT, which splits X-ray attenuation by energy, does it for liver lesions. Gamified crowdsourcing can already produce expert-level labels for subtle findings. However, the evidence comes from short contests paid in cash, and engagement research shows that short evaluations capture people mainly during the early novelty spike. What happens over months is largely unmeasured: whether people keep labeling, and above all whether they keep labeling the hard cases correctly.

This note proposes a labeling game built around a different kind of feedback. After a participant judges a white-light image, the game does not just announce an expert's answer. It shows the paired narrow-band image, the physical evidence that settles the case. The bet is that feedback which teaches people why they were wrong builds lasting skill and confidence in that skill. Longitudinal studies identify that sense of capability, rather than points or prizes, as what keeps performance up over time. If so, this feedback should stop the slow, hidden drop in accuracy on ambiguous cases that we expect contest and leaderboard designs to suffer once their novelty fades. We test this in a six-month randomized deployment. Success is measured by retention and by the accuracy of models trained on the labels each design produces.

Core Claim

In ambiguous imaging, engagement decay hurts data quality more than it reduces volume. As novelty wears off, participants should keep labeling easy cases correctly and quietly lose accuracy on the cases that matter. Cross-modal reveal feedback can prevent this, because it shows annotators the evidence that resolves the case rather than only the verdict. The claim is falsifiable on three axes:

  1. Retention curves over 26 weeks.
  2. Accuracy on a stratum of "ambiguous" items, defined as cases where white light alone misleads but narrow-band imaging resolves.
  3. The test accuracy of a classifier trained on each arm's labels from each month.

Where This Sits

Three cross-cutting studies motivate the design.

  • Clinical-grade labels from a crowd. Gamified Crowdsourcing as a Novel Approach to Lung Ultrasound Dataset Labeling ran an eight-day cash-prize contest. Crowd consensus reached 87.9% concordance with an expert panel, against 85.0% for individual experts. The crowd plateaued after about 75 cases with feedback, and roughly seven filtered opinions per clip sufficed. Its feedback was the expert answer, its horizon was eight days, and its motivation was money. We change all three.
  • Engagement mechanics can damage data quality. Experiments on Motivational Feedback for Crowdsourced Workers followed 437 workers for six months on a text task. Leaderboards were the most durable motivator and raised accuracy from 56% to 79%. But stacking badges, leaderboards and rewards led workers to game the system, and a no-feedback group beat every non-gamified feedback group.
  • Targeted retention works on easy tasks. Improving Paid Microtasks through Gamification and Adaptive Furtherance Incentives kept 77% of workers who were about to quit, against 27% under random offers. The authors explicitly leave open whether this transfers to cognitively demanding tasks.

The engagement literature supplies the hypothesis. A Longitudinal Study of Self-Regulated Learning in an Online Game-Based EFL Formative Assessment Blended Learning Environment followed 358 students for a year. Extrinsic motivation was significantly lower among two-term participants (p = 0.047), and self-efficacy was the only variable that differed across performance tiers. Our reveal mechanic is an attempt to manufacture self-efficacy rather than tokens. Work showing that open-ended systems without milestones lose users (Yadong Xi et al.) and that leaderboards demotivate low-ranked players (Ping Zhang) constrains the design. It argues for structured progression and against public ranking as the main driver.

On the imaging side, Colorectal Polyp Classification from White-light Colonoscopy Images via Domain Alignment supplies both the data and the conceptual template. It trains a white-light student network under a narrow-band teacher using adversarial and contrastive alignment, reaching 85.3% accuracy on the CPC-Paired dataset. It is the same teacher-student idea, except that our student is a person and the alignment happens through feedback. Liver lesion localisation and classification with convolutional neural networks: a comparison between conventional and spectral computed tomography shows the same gap in CT. There, 40 keV monoenergetic images reach 89.9% accuracy against 85.4% for conventional CT, so the same mechanic could later be replicated there. Advances in Artificial Intelligence to Reduce Polyp Miss Rates during Colonoscopy documents that models trained only on polyp-positive frames raise false alarms on healthy tissue. The crowd's 99.3% accuracy on pathology-free ultrasound clips suggests humans can supply the missing negatives.

Research Program

Participants. We build a mobile labeling game for medical students and early residents. For this group, getting better at reading endoscopy is a plausible intrinsic reward, which distinguishes them from anonymous microtask workers.

Two item streams. - The feedback-and-gold stream comes from CPC-Paired: 258 white-light and narrow-band pairs from 123 patients, with hyperplastic-versus-adenomatous labels. It is split by patient into a feedback pool, a hidden gold pool, and a held-out model test set that no participant ever sees. - The production stream is unpaired white-light frames from public polyp collections (Kvasir, CVC-ClinicDB), mixed with polyp-free frames. Participants judge polyp presence and, where visible, likely type.

Ambiguous stratum. Before deployment, we flag CPC-Paired items as "ambiguous" if the cited domain-alignment model's white-light-only ablation misclassifies them, or if the expert panel disagrees on the white-light image alone.

Experimental arms. Participants are randomized into five arms with identical base compensation and item pools:

  • Silent arm: labeling with no feedback.
  • Contest arm: monthly cash-prize sprints with expert-answer feedback, reproducing the lung ultrasound design.
  • Leaderboard arm: points, badges and a leaderboard.
  • Reveal arm: after each feedback-pool judgment, the participant sees the narrow-band counterpart beside their white-light image, with the decisive region highlighted. The arm also has milestone-based progression and private skill curves instead of public rank. Scoring uses a proper scoring rule on a stated confidence, a rule under which reporting your true belief is the score-maximizing strategy. This rewards calibrated honesty rather than speed, which is our guard against gaming.
  • Answer-only arm: identical to the reveal arm except that feedback shows only the expert label. It isolates the narrow-band image itself as the active ingredient.

Measurement. Every arm sees hidden gold items at a fixed rate, so accuracy can be tracked weekly per participant. We separately log:

  • time per item, where a sharp drop at stable volume is our behavioral signature of gaming;
  • confidence calibration;
  • churn.

Aggregation. Labels are pooled with the same quality-weighted consensus in all arms. We then train one fixed white-light classifier (ResNet50, same recipe) on each arm's labels from months 1, 3 and 6 separately. This converts engagement differences into model differences.

Baselines

Silent arm (naive reference). This is the honest floor. It matters because Experiments on Motivational Feedback for Crowdsourced Workers found no-feedback beat several feedback designs, so feedback could plausibly hurt. Any arm must exceed its month-6 gold accuracy on ambiguous items to justify its added complexity.

Contest arm (strongest human-labeling competitor). This reproduces the design with the best published evidence for crowd labels of subtle medical findings: 87.9% concordance with experts on 198 test clips, reported in Gamified Crowdsourcing as a Novel Approach to Lung Ultrasound Dataset Labeling. It is run as monthly sprints on the same items with the same prize budget spread across the six months, so it is not handicapped into a single burst. Beating it means two things. First, higher ambiguous-stratum gold accuracy in months 4 to 6, with the gap emerging over time rather than present from week one. Second, models trained on reveal-arm month-6 labels outperform those trained on contest-arm month-6 labels. A week-one gap would point to a teaching effect rather than a sustainability effect, and we would report it as such.

Leaderboard arm. This is the strongest long-horizon engagement design in the file, the durable motivator in the six-month crowdsourcing study, so it is the main threat on retention. It runs with the same pools and pay. The reveal arm must match its retention while showing less ambiguous-stratum decay and no time-per-item collapse.

Model-only domain alignment (strongest automated competitor). The reported figure is 85.3% accuracy on CPC-Paired, from Colorectal Polyp Classification from White-light Colonoscopy Images via Domain Alignment. We rerun it on our patient-level split so the comparison uses the same test images. We also give it the production frames as unlabeled data with pseudo-labels, so it has the same image access as the crowd arms, with tuning budget matched to the crowd-trained classifier. The crowd-label route counts as competitive if its classifier matches this number on our split without needing paired images at training time. It counts as superior if it beats this number, especially on the ambiguous stratum.

Answer-only arm (ablation). Comparing it with the reveal arm isolates the cross-modal image as the cause of any effect. The proposal's specific claim survives only if the reveal arm beats it on late-deployment ambiguous accuracy.

Why not just…?

Why not just train the domain-alignment network and skip humans? Because it needs paired scans, and paired sets are tiny (258 pairs here). Its own ablation drops to 82.9% without alignment, showing how much it leans on that pairing. Human labels on large unpaired white-light corpora are the route to scale. Automated systems also stumble exactly where crowds are strong: the all-polyp bias of Advances in Artificial Intelligence to Reduce Polyp Miss Rates during Colonoscopy, and open-set collapse in Open Set Recognition for Endoscopic Image Classification: A Deep Learning Approach on the Kvasir Dataset, where a Swin model fell to 63.7%. This is still a real threat. The model-only baseline above settles it directly, and if crowd-trained models do not beat it, the labeling program is not worth its cost.

Why not just run short cash sprints repeatedly, resetting novelty each time? This is the most serious alternative, and it is why the contest arm runs monthly sprints rather than one burst. If repeated sprints hold ambiguous-stratum accuracy as well as the reveal arm at lower engineering cost, the honest conclusion is that money plus resets suffices. We would report that.

Why not just fix gaming with quality-weighted aggregation, whatever the game design? Aggregation removes noisy annotators but cannot recover signal that nobody supplies. If every participant drifts on hard cases, the consensus drifts with them. Because all arms share the same aggregation, any remaining between-arm difference in model accuracy is attributable to design.

Why not just pay experts? The lung ultrasound study found crowd consensus more accurate than the average individual expert in leave-one-out analysis (87.4% vs 80.8%). Clinicians miss 2% to 26% of polyps in practice. Expert time is the scarce resource the program is meant to economize, and here it is spent only on building gold sets.

Decisive Checks

Main analysis. We fit a mixed-effects model of weekly gold accuracy, with terms for arm, week, and their interaction, separately for ambiguous and easy items.

Prediction that would confirm the hypothesis. A significant negative slope on ambiguous items in the contest and leaderboard arms after roughly month two, a flat slope on easy items, and a flat slope in the reveal arm. Classifiers trained on month-6 labels should show the same ordering on the held-out CPC-Paired split.

Results that would refute the specific contribution: - No decay in any arm, meaning the novelty concern is unfounded for this population. - Decay that is identical across arms, meaning feedback content does not matter. - Reveal equal to answer-only, meaning the narrow-band image is not the active ingredient.

Secondary check on gaming. Do time-per-item collapses predict later gold errors per participant, and are they rarer under confidence-based scoring?

Replication. If the effect holds, we repeat the reveal-versus-answer comparison on conventional versus spectral liver CT, with 40 keV images as the reveal.