Ethics and Governance of Artificial Intelligence: Evidence from a Survey of Machine Learning Researchers

Baobao Zhang, Markus Anderljung, Lauren Kahn, Noemi Dreksler, Michael C. Horowitz, Allan Dafoe

Abstract: Machine learning (ML) and artificial intelligence (AI) researchers play an important role in the ethics and governance of AI, including taking action against what they perceive to be unethical uses of AI. Nevertheless, this influential group's attitudes are not well understood, which undermines our ability to discern consensuses or disagreements between AI/ML researchers. To examine these researchers' views, we conducted a survey of those who published in the top AI/ML conferences (N = 524). We find that AI/ML researchers place high levels of trust in international organizations and scientific organizations to shape the development and use of AI in the public interest; moderate trust in most Western tech companies; and low trust in national militaries, Chinese tech companies, and Facebook. While the respondents were overwhelmingly opposed to AI/ML researchers working on lethal autonomous weapons, they are less opposed to researchers working on other military applications of AI, particularly logistics algorithms. A strong majority of respondents think that AI safety research should be prioritized and that ML institutions should conduct pre-publication review to assess potential harms.

Introduction

While AI ethics and governance are hot topics for policymakers and the public, the voices of the people actually developing the algorithms—machine learning (ML) researchers—are often missing from the data. This is a problem because these researchers are the "first responders" to technical risks; they are best positioned to identify emerging hazards and develop technical solutions. Without understanding where this influential group stands, it is difficult to build a consensus on regulations or identify where technical experts might clash with public opinion or corporate interests. This study aims to fill this gap by quantifying researcher attitudes on trust, military use, and publication norms.

Methodology

The authors conducted a pre-registered survey between September and October 2019, targeting researchers who published at the two most prestigious AI conferences: NeurIPS (Neural Information Processing Systems) and ICML (International Conference on Machine Learning). To prevent survey fatigue and order bias, respondents (N=524) were shown randomized subsets of issues and actors. The team used multiple linear regression to analyze associations and compared results to a 2016 survey of AI experts and a 2018 survey of the US general public.

Trust and Institutional Governance

A defining characteristic of the AI/ML research community is its "internationalist bias." Researchers place significantly more trust in international organizations (UN, EU) and non-governmental scientific bodies (like AAAI) than the general public does. Conversely, they are far more skeptical of national militaries and specific corporate entities.

Trust ratings for various actors across national, international, corporate, and other categories, comparing AI/ML researchers to the US public.

On a 0–3 scale, scientific organizations (2.12) and the EU (1.98) were the most trusted. In contrast, Facebook (0.58) and the Chinese military (0.30) received the lowest trust ratings. This data suggests that technical experts favor governance frameworks led by scientific and international bodies over nationalistic or purely corporate-led initiatives.

Prioritizing AI Governance Challenges

When asked to rank the importance of various AI governance challenges, researchers prioritized technical and social risks over geopolitical ones. Preventing criminal justice bias and ensuring autonomous vehicle safety were rated as the most critical challenges.

Comparison of mean issue importance for AI governance challenges between AI/ML researchers and the US public.

Notably, "U.S.–China competition" was rated as the least important challenge by researchers—a sharp contrast to the high priority placed on this issue by the US public and policymakers. This disconnect highlights a potential friction point between the technical community's globalist outlook and the national security focus of many governments.

Ethical Stances and Publication Norms

The survey revealed a nuanced stance on military applications. While there is overwhelming opposition to working on Lethal Autonomous Weapons Systems (LAWS)—with 31% of researchers stating they would resign if asked to work on them—there is much higher acceptance for AI in military logistics. This suggests the "red line" for experts is specifically the automation of lethal force.

Furthermore, a "safety culture" shift is evident: 68% of researchers believe AI safety should be prioritized more than it currently is. This growing concern for safety is accompanied by a shift in publication norms. While 84% believe high-level methods must always be shared, only 22% feel the same about trained models, reflecting a growing caution regarding the potential for malicious use of ready-to-deploy AI. Additionally, 59% of respondents supported the idea of institutions conducting internal ethical reviews before research is made public.

Limitations

The study acknowledges several limitations, including a sample bias toward academic conference participants rather than product-focused industry researchers. The sample was also 91% male, reflecting the existing gender gap in the field. Finally, as the data was collected in late 2019, attitudes may have evolved following the recent explosion in Large Language Models (LLMs) and the global pandemic.

Summary updated

Abstract

Machine learning (ML) and artificial intelligence (AI) researchers play an important role in the ethics and governance of AI, including taking action against what they perceive to be unethical uses of AI (Belfield, 2020; Van Noorden, 2020). Nevertheless, this influential group's attitudes are not well understood, which undermines our ability to discern consensuses or disagreements between AI/ML researchers. To examine these researchers' views, we conducted a survey of those who published in the top AI/ML conferences (N = 524). We compare these results with those from a 2016 survey of AI/ML researchers (Grace, Salvatier, Dafoe, Zhang, & Evans, 2018) and a 2018 survey of the US public (Zhang & Dafoe, 2020). We find that AI/ML researchers place high levels of trust in international organizations and scientific organizations to shape the development and use of AI in the public interest; moderate trust in most Western tech companies; and low trust in national militaries, Chinese tech companies, and Facebook. While the respondents were overwhelmingly opposed to AI/ML researchers working on lethal autonomous weapons, they are less opposed to researchers working on other military applications of AI, particularly logistics algorithms. A strong majority of respondents think that AI safety research should be prioritized and that ML institutions should conduct pre-publication review to assess potential harms. Being closer to the technology itself, AI/ML re-searchers are well placed to highlight new risks and develop technical solutions, so this novel attempt to measure their attitudes has broad relevance. The findings should help to improve how researchers, private sector executives, and policymakers think about regulations, governance frameworks, guiding principles, and national and international governance strategies for AI.
BibTeX
@article{zhang2021ethics,
  author = {Baobao Zhang and Markus Anderljung and Lauren Kahn and Noemi Dreksler and Michael C. Horowitz and Allan Dafoe},
  title = {{Ethics and Governance of Artificial Intelligence: Evidence from a Survey of Machine Learning Researchers}},
  journal = {Journal of Artificial Intelligence Research},
  year = {2021},
  doi = {10.1613/jair.1.12895},
}
Figure