TL;DR
This paper tackles the challenge of enabling crowd workers to tackle complex writing tasks by developing and evaluating "structured communication" mechanisms. They found that different communication methods are more effective at different stages of writing, and propose a workflow that dynamically selects the best mechanism to improve efficiency and quality.
What Problem Does This Paper Address?
Traditional crowdsourcing platforms excel at simple, independent microtasks. However, there's a growing interest in using crowds for more complex tasks like writing, which inherently require context and mutual understanding between requesters and workers. The core problem is that existing platforms lack effective ways to transfer this crucial context. Open-ended communication, while effective on platforms like Upwork, is often too costly and time-consuming for typical microtask settings. This paper aims to bridge this gap by exploring structured, low-overhead communication methods that can facilitate complex collaborative writing.
Methods
The authors developed five "structured communication mechanisms" to facilitate context transfer for writing tasks. These mechanisms were designed iteratively, drawing inspiration from expert writers on platforms like Upwork. The core idea is to break down communication into smaller, more manageable "microtasks" for the requester. The mechanisms include:
- Ask Questions (Q&A): Both general (applicable to any paragraph) and specific (related to the content).
- Free-form comment and edit: Similar to track changes in word processors.
- Highlight strengths and weaknesses: Requester marks parts of the text they like (green) or dislike (red).
- Identify the main problem and fix it: Requester answers two questions: "What is the main problem with this paragraph?" and "What is your recommendation for the next person who edits it?"
To evaluate these mechanisms, they conducted two studies:
- Study 1 (Value): Measured how much each mechanism improved the quality of crowd-written paragraphs. They used an expert reviewer to provide feedback via each mechanism and then had other crowd workers edit the paragraphs based on this feedback.
- Study 2 (Cost): Measured the cognitive cost (mental demand and hard work) imposed on requesters using each mechanism. They used researchers as proxies for requesters.
Based on these studies, they designed a workflow for crowdsourcing written content that dynamically selects the most appropriate communication mechanism based on the current quality of the paragraph and the mechanism's cost-value trade-off.
Datasets
The paper does not explicitly list or introduce a formal dataset in the traditional sense. Instead, they created a corpus of 66 initial "mediocre" paragraphs for their studies. These paragraphs were generated through a two-step process: first, crowd workers generated lists of ideas for given prompts, and then other crowd workers wrote paragraphs based on these lists. This approach aimed to simulate real-world crowd-written content with varying quality.
Key Insights and Ideas
- Context is King, but Cost Matters: While communication is vital for complex tasks, the cost to the requester is a significant factor. Structured communication aims to balance value with cost.
- Writing Stage Dictates Mechanism: Different communication mechanisms are effective at different stages of the writing process. For early, low-quality drafts, direct problem identification is more valuable. For more refined drafts, subtle improvements might be needed.
- Low-Quality Content Benefits Most: Structured communication is most effective when improving low-to-average quality paragraphs. High-quality paragraphs can sometimes be made worse by miscommunication.
- "Main Problem" is a Sweet Spot: The "Identify the main problem and fix it" mechanism proved particularly effective for improving low-quality paragraphs, offering a good balance of value and cost.
- Highlighting is Efficient: Highlighting is a low-cost mechanism that can be as effective as more involved methods like free-form commenting for average-quality paragraphs.
- Structured Communication Reduces Requester Burden: By framing communication as microtasks, requesters can provide valuable feedback with less effort.
Key Results
- Study 1 (Value):
- All mechanisms improved low and average quality paragraphs.
- For low-quality paragraphs, the "Main problem" mechanism showed significantly greater improvement than "Highlight" and "Q&A specific."
- Editing reduced the quality of initially high-quality paragraphs, suggesting limited communication can be counter-productive.
- Study 2 (Cost):
- "Comment & edit" and "Ask Questions" were the most costly mechanisms for requesters.
- "Main problem" and "Highlight" were the least costly.
- The cost of a mechanism is not always correlated with its value.
Limitations and Future Work
- Focus on Requester Cost: The studies primarily measured the cognitive cost to the requester. Future work should also consider the cost to workers and other factors like accessibility for users with limited input devices.
- Domain Specificity: The research is focused on writing. Future work should explore adapting these mechanisms to other domains (e.g., visual design) and investigate domain-specific communication needs.
- Generating Good Questions: The "Ask Questions" mechanism highlighted a challenge: crowd workers often struggle to formulate effective clarification questions. Future research should explore better methods for question generation.
- Subjectivity of Evaluation: Evaluating writing quality is inherently subjective. The paper notes that novices may struggle with this, and future systems might benefit from leveraging the judgment of crowds who represent the intended audience or expert writers.
- Dynamic Role Assignment: The paper suggests exploring how to identify the best actors (requester, expert, crowd) for specific tasks and how these roles can be dynamic.
Summary updated