Abstract
Crowdsourcing image labels via independent majority voting often fails on "hard" images near decision boundaries. This paper introduces Consensus Agreement Games (CAG), a collaborative method where small cliques of workers must reach a consensus, which is then verified against an independent second clique. CAG reduces labeling errors by 37.8% on difficult datasets like MINC and MIT Places by leveraging social correction and tiered incentives.
Introduction
The performance of computer vision models depends heavily on the quality of labeled training data. The standard approach—independent majority voting on platforms like Amazon Mechanical Turk—is effective for clear-cut cases but breaks down on "hard" images with subtle features. These low-agreement samples are often discarded, leading to datasets biased toward easy examples. This work addresses the challenge of obtaining accurate labels for these difficult images through a collaborative framework called Consensus Agreement Games (CAG).
Methods: Consensus Agreement Games
CAG shifts the labeling paradigm from independent redundancy to structured collaboration. Workers are organized into small cliques (typically 2-3 people) and participate in a multi-stage verification process.
Intra-Clique Collaboration
Within a clique, workers see their peers' choices in real-time. They are encouraged to discuss or adjust their votes until a unanimous consensus is reached. This social dynamic allows workers to highlight subtle visual cues, effectively correcting individual errors through collective reasoning.
Inter-Clique Agreement and Incentives
To ensure the validity of the consensus, the system requires two independent cliques to reach the same conclusion for the same image. A tiered payoff structure ($P_1 > P_2 > P_3$) incentivizes this behavior:
- $P_1$ (Consensus Agreement): Highest reward; both cliques agreed.
- $P_2$ (Clique Consensus): Medium reward; the clique agreed internally, but the other clique differed.
- $P_3$ (Discord): Lowest reward; the clique failed to reach internal agreement.
This structure encourages workers to seek the "objective truth" that another independent group is likely to find.
Datasets and Experimental Setup
The authors evaluated CAG on two challenging domains:
- MINC (Materials in Context): Identifying materials such as glass, metal, or paper in real-world scenes. The study focused on samples where previous independent voting achieved only marginal agreement.
- MIT Places: Scene recognition involving mutually confusing categories like bedrooms, hotel rooms, and nurseries.
Results and Discussion
CAG demonstrated a significant improvement over traditional methods:
- Error Reduction: Compared to a baseline of 7 independent workers (Vote7), CAG reduced the error rate by 37.8%.
- Accuracy: On the MINC dataset, error rates dropped from 19.4% to 11.8%. On MIT Places, errors fell from 23.5% to 14.9%.
- Social Correction: The collaborative process was found to be self-correcting; cliques corrected initially wrong labels 100 times, while only being misled 44 times.
While CAG is more expensive and slower than independent voting, it serves as a powerful "refinement" tool for the most difficult portions of a dataset. Future research could explore scaling this method to larger label sets or more complex annotation tasks.
Conclusion
Consensus Agreement Games provide a robust mechanism for labeling difficult images by transforming a solitary task into a collaborative game. By aligning incentives with inter-group agreement, CAG successfully extracts high-quality ground truth from the crowd where traditional methods fail.
Summary updated