TL;DR
This paper tackles the complex problem of fairly and robustly allocating human content reviewers to various harmful content categories on online platforms. They extend traditional market equilibrium models to account for forecasted workloads, time-dependent scheduling, and the desire for smooth, predictable staffing, demonstrating their approach with real-world data from Meta.
What Problem Does This Paper Address?
The sheer volume of online content necessitates robust content moderation systems. This paper focuses on the challenge of allocating human reviewers to different categories of harmful content. Key issues include:
- Forecasted Workloads: Content moderation systems need to handle predicted volumes of content in various categories, but actual workloads can deviate.
- Time-Dependent Allocation: Efficient scheduling requires planning reviewer assignments across multiple time periods, considering both per-period and overall capacity.
- Smooth Allocation: Partners providing reviewer services prefer stable, predictable schedules to minimize staffing changes and associated costs.
- Fairness: Allocations must be fair, ensuring that different content categories (buyers) receive equitable treatment, especially when there's excess capacity.
Methods
The authors leverage tools from fair division and market equilibrium theory, specifically adapting the Fisher market model. Their approach introduces several novel aspects:
- Robust Allocation with Hard Demands: They extend the Eisenberg-Gale program to incorporate "hard demands" (forecasted workloads) for content categories. This ensures these demands are met, and any excess capacity is allocated fairly. This is framed as a "robust" allocation, providing a buffer against actual workload fluctuations.
- Time-Dependent Allocation: They develop extensions of fair allocation methods to handle multiple time periods. This involves considering both per-time-period and overall supply constraints for reviewers. Two main approaches for combining utilities across time are explored: summing utilities (direct extension of single-period fairness) and taking the geometric mean (fairness at a more granular, per-period level).
- Smooth Allocation: To address the desire for stable schedules, they introduce a penalty term into the objective function. This term penalizes large variations in allocation between consecutive time periods, encouraging smoother staffing. They explore both constraint-based and penalty-based approaches for achieving smoothness.
- Fairness Concepts: They analyze allocations using leximin and competitive equilibrium from equal incomes (CEEI), and max Nash welfare (MNW), comparing their properties and equivalences under different scenarios.
Datasets
The paper uses real-world data obtained from Meta (formerly Facebook) for demonstration and evaluation. This data includes information on reviewer partners, work types (content categories), demands, supplies, and valuations.
Key Insights and Ideas
- Market Equilibrium as a Framework for Content Moderation: The paper demonstrates that established economic models like Fisher markets can be effectively adapted to solve practical operational problems in content moderation.
- Balancing Fairness and Robustness: The core contribution lies in finding allocations that are not only fair but also robust to uncertainty in workload, a critical aspect for real-world systems.
- The Trade-off Between Smoothness and Efficiency: Introducing smoothness constraints can lead to less efficient utilization of reviewer capacity, highlighting a key business trade-off that needs careful parameter tuning.
- Leximin vs. EG/MNW: The paper shows that while leximin and EG/MNW solutions often coincide in simpler settings (e.g., binary valuations, unit budgets), they can diverge in more complex scenarios (e.g., bi-valued valuations, budget equal to demand), with leximin sometimes prioritizing egalitarianism over business needs.
Key Results
- The proposed formulations successfully incorporate forecasted workloads, time-dependent scheduling, and smoothness preferences into fair allocation mechanisms.
- Through experiments with Meta data, they show that their methods can produce allocations that are both fair and robust, and that smoothness parameters can be tuned to balance operational efficiency with staffing stability.
- They provide theoretical results on the equivalence and non-equivalence of different fairness concepts (leximin, MNW, CEEI) under various valuation and budget settings, offering guidance on which approach to use.
- For the specific application to Meta's content review, they observe that when budgets equal demands, EG solutions allocate oversupply multiplicatively, while leximin can lead to allocations that don't align with business goals by favoring buyers with the smallest demand.
Limitations and Future Work
- Infeasibility with High Demands and Variation: The paper notes that constraint-based approaches to enforce smoothness can lead to infeasible problems when demands are high and demand variation is significant.
- Parameter Tuning for Smoothness: The choice of the smoothness parameter ($\gamma$) is crucial and requires careful consideration to balance utility maximization with staffing stability, as a very smooth allocation might waste capacity.
- Generalizing to Non-Binary Valuations: While the paper touches upon bi-valued valuations, a deeper exploration of general non-binary valuations for robustness and fairness could be a future direction.
- Dynamic Workload Prediction: The current model relies on forecasted workloads. Integrating more dynamic and adaptive workload prediction mechanisms could further enhance robustness.
Summary updated