TL;DR
This paper argues that computational social choice (COMSOC) research needs to move beyond theoretical models and embrace empirical, data-driven approaches that incorporate real human behavior. The author highlights their work in building systems and collecting data to bridge the gap between idealized agent models and how humans actually make decisions, leading to more impactful real-world applications.
What Problem Does This Paper Address?
Computational Social Choice (COMSOC) uses computer science tools like algorithms and complexity analysis to study group decision-making, covering areas like voting and resource allocation. While this has led to significant theoretical progress, the author argues that a crucial element is missing: a deep understanding of how humans actually behave in these scenarios. Traditional COMSOC models often make simplifying assumptions about agent rationality and preferences that don't hold up in the real world. This disconnect limits the practical impact of COMSOC research. The paper aims to address this by advocating for and demonstrating the importance of empirical, data-driven research that integrates human behavior and leads to the development of real-world systems.
Methods
The author's approach centers on a blend of theoretical innovation, experimental analysis, and practical tool building. Key methods include:
- Empirical Studies: Conducting studies using large datasets of real-world preference data to analyze how agents behave in voting and allocation scenarios. This involves mining existing data and creating artificial elections to test theoretical assumptions.
- System Building: Developing practical, user-friendly systems and websites that allow users to deploy and experiment with various COMSOC mechanisms. Examples include PREFLIB.ORG for preference data and OPERA for voting systems.
- Data Collection and Curation: Establishing platforms like PREFLIB.ORG to gather, unify, and make accessible large collections of real-world preference data from diverse domains.
- Bridging Theory and Practice: Using empirical findings to inform and refine theoretical models of agent behavior, and conversely, using theoretical insights to design better experiments and systems.
- Human-Centered Design: Focusing on understanding how systems are used in practice and incorporating ethical considerations and societal constraints into system design.
Datasets
The paper discusses the use and creation of several datasets:
- Real-world preference data: Thousands of instances from domains including voting, resource allocation, ranking, and rating, unified into a common format on PREFLIB.ORG.
- Election data: Collected from professional societies and historical elections to analyze paradoxes and voter utility models.
- Doodle data: Used to study behavioral effects in group decision-making systems.
- OPERA system data: Used to devise querying methodologies for group decision-making.
- Experimental data from Meir et al. [2020b]: Used to propose novel voter decision models for strategic voting.
The paper also highlights the establishment of PREFLIB.ORG and PREFLIB TOOLS as a significant contribution to the community, providing a unified repository of preference data for research.
Key Insights and Ideas
- The "messy" reality of human decision-making is crucial: Idealized models in COMSOC often fail to capture how humans actually behave, leading to a disconnect from real-world impact.
- Empirical data is the bridge: Integrating data-driven studies and real-world system building is essential for advancing COMSOC research and making it more impactful.
- Systems foster a feedback loop: Building practical systems not only helps users but also generates valuable data and insights that can inform theoretical advancements.
- PREFLIB.ORG is a community enabler: Providing accessible, unified preference data democratizes research and encourages empirical verification.
- Ethical considerations are paramount: As AI systems become more pervasive, embedding societal values, fairness, and justice into their design is critical.
Key Results
The paper doesn't present specific quantitative results in terms of accuracy or performance metrics for a single system. Instead, it highlights the impact of the described research program:
- Challenging common assumptions: Empirical studies have shown that many assumptions about worst-case agent behavior, preference domain restrictions, and election paradoxes do not occur in practice.
- Advancing understanding of voting paradoxes: Data and automated proof techniques have led to a deeper understanding of the causes of these paradoxes.
- Developing practical tools: Websites like Spliddit and OPERA, and systems like Pynx, provide users with access to advanced COMSOC algorithms for fair division, voting, and more.
- Informing theoretical models: Empirical findings have led to the proposal of novel voter decision models and a better understanding of strategic behavior.
- Facilitating broader research: The PREFLIB repository has enabled numerous studies across COMSOC and related fields.
Limitations and Future Work
- Limitations: The paper implicitly acknowledges that the "messy and sometimes downright irrational ways that choices and systems can be used" are complex to fully model and integrate. The focus on empirical data also means that the findings are tied to the specific datasets and contexts studied.
- Future Work: The author suggests several avenues for future exploration, including:
- Work in social networks.
- Developing more sophisticated voter models.
- Online allocation systems.
- Further integration of ethical considerations and societal constraints into system design.
- Continuing to build systems that have broad, significant impact in the real world.
Summary updated