TL;DR
This paper offers a comprehensive, mixed-methods analysis of the first four years of the ACM FAccT conference, examining its research themes, underlying values, impact, and community dynamics. It reveals a strong focus on machine learning and fairness, while also highlighting community-identified gaps in governance, broader societal impact, and inclusivity, offering concrete suggestions for future directions.
What Problem Does This Paper Address?
The field of Fairness, Accountability, and Transparency (FAccT) in socio-technical systems has rapidly grown, with the ACM FAccT conference serving as its primary venue. However, concerns have been raised within the community about potential shortcomings, such as an overemphasis on narrow technical topics at the expense of broader societal issues like AI governance. This paper addresses the need for a systematic, reflexive analysis of FAccT's contributions, identifying its strengths, weaknesses, and areas for improvement to foster more impactful research and a healthier academic community.
Methods
The authors employed a mixed-methods approach, combining qualitative and quantitative techniques.
- Qualitative Methods:
- Manual Coding: A protocol was developed to review and code prior FAccT papers, analyzing their topics, methods, datasets, and disciplinary roots. This was also used to code open-ended survey responses regarding community perspectives on shortcomings and recommendations.
- Survey Design: A web-based survey was administered to FAccT affiliates, featuring both open-ended and Likert-scale questions to gather insights on scholarship, impact, and community.
- Quantitative Methods:
- Topic Modeling (LDA): Latent Dirichlet Allocation was used on the full text of FAccT papers (2018-2021) to identify prevalent research themes and their evolution over time.
- Citation Network Analysis (Louvain Algorithm): A community detection algorithm was applied to the citation network of FAccT articles to identify clusters of related research and their interconnections.
Datasets
- FAccT Publications Corpus: Full-text articles from the ACM FAccT conference proceedings from 2018-2021 were used for topic modeling and manual coding.
- Citation Network Data: Data from Semantic Scholar Open Research Corpus (S2ORC) and the AMiner citation network dataset were used for citation network analysis.
- Survey Responses: Data collected from a survey administered to FAccT community members and affiliates.
Key Insights and Ideas
- Dominance of ML and Fairness: The analysis consistently shows a strong focus on machine learning (ML) and fairness-related topics within FAccT scholarship, often intertwined. This suggests that while FAccT aims for broader socio-technical systems, its current research landscape is heavily shaped by ML challenges.
- Underrepresentation of Broader Societal Values: Beyond fairness, accountability, and transparency, participants identified a range of other crucial moral and social values (e.g., dignity, respect, diversity, inclusion, harm prevention, truth) that FAccT scholarship should address more extensively.
- Perceived Gap in Real-World Impact: While FAccT has been successful in fostering interdisciplinary interactions and raising awareness, there's a perceived disconnect between scholarly contributions and tangible real-world impact, particularly in certain application domains like education and internet advertising.
- Community Concerns about Practices: The FAccT community expressed concerns regarding peer review processes, inclusivity, the influence of industry, and the potential for "fair-washing" through narrow technical solutions.
- Emergence of Legal and Social Science Focus: Despite the ML dominance, there's a growing trend in papers related to law, social sciences, and ethics over the years, indicating a potential broadening of the FAccT scope.
Key Results
- Topic Prevalence: Fairness was the most prevalent topic (69% of publications), followed by transparency (26%) and accountability (26%), with long-term impact at 17%. Other topics like privacy and human factors were present in 31% of papers.
- Subtopic Popularity: Within fairness, discrimination/group-level fairness was most prevalent. For transparency, explainability was key. For accountability, professional codes and standards were prominent. For long-term impact, trust/disinformation was a leading subtopic.
- Methodological Landscape: 18.9% of papers used quantitative empirical methods, and 32.9% used qualitative empirical methods. 61.3% were broadly STEM, and 29.7% were HSA.
- Dataset Usage: The most frequently used public datasets were Adult Income, COMPAS, and German Credit.
- Interdisciplinary Exposure: Participants rated FAccT's exposure to insights from other disciplines highly (mean 3.95/5).
- Peer Review Quality: Perceptions of peer review quality were more moderate (mean 3.11/5), with suggestions for improvement.
- Domain Impact: Criminal justice systems and hiring/employment received the highest scores for broader impact, while education and internet advertising received the lowest.
- Moral Foundations: Analysis using Moral Foundations Theory suggests FAccT scholarship leans towards "care" and "fairness" foundations, common in liberal cultures, with less emphasis on "loyalty," "authority," and "sanctity."
Limitations and Future Work
- Limitations:
- The study acknowledges its own potential biases due to the authors' affiliation with FAccT.
- The survey response rate was 6.5%, which might limit the generalizability of community perspectives.
- The analysis of FAccT publications was limited to 2018-2021, missing earlier years.
- The study did not attempt to describe end-to-end deployments in practice, relying on community perceptions of impact.
- Future Work/Suggestions:
- Increased Transparency: The conference organization should be more transparent, especially regarding industry relationships and the peer review process.
- Enhanced Inclusivity: Foster a more inclusive environment by appreciating diverse modes of inquiry and contributions.
- Stronger Stakeholder Engagement: Work more closely with stakeholders and practitioners to achieve positive, real-world impact.
- Critical Evaluation of Assumptions: Develop a more standardized set of terms and norms to improve clarity and effectiveness of communication.
- Focus on Governance and Accountability: Invest more intellectual effort in pressing issues of governance and accountability.
- Broaden Value Scope: Expand the range of moral and social values addressed in scholarship.
- Strengthen Real-World Connections: Enhance connections to real-world issues, practices, and stakeholders.
- Improve Community Practices: Address concerns related to peer review, inclusivity, and industry influence.
- Conceptual Work: Encourage more conceptual work, particularly drawing on Science and Technology Studies (STS).
Summary updated