Errare humanum est, perseverare autem diabolicum: A Follow-Up Study on the Human-Likeness of an AI Othello Player

Béatrice Biancardi, Enrico Lauletta, Antonio Norelli, Alessandro Panconesi, Maurizio ManciniIVA 2023 · DOI

TL;DR

This paper investigates whether an AI Othello player, Olivaw, is perceived as more human-like than another AI player, Saio, and how game outcomes influence this perception. They found that while Olivaw wasn't perceived as more human-like than Saio, losing players were consistently rated as more human-like than winning players, suggesting a bias in how we attribute human traits based on performance.

What Problem Does This Paper Address?

The increasing sophistication of AI agents in complex games like Othello raises questions about their perceived "human-likeness." While AI has surpassed human performance in many games, understanding how humans perceive the behavior of these agents is crucial for human-AI interaction. Previous work suggested that Olivaw, an AI that learns through self-play without relying on human knowledge or handcrafted rules, might exhibit human-like qualities. This study aims to systematically investigate this perception, specifically comparing Olivaw to another AI (Saio) and examining the influence of game outcomes on how human-like an opponent is perceived. The core challenge is to quantify and understand the subjective experience of human-likeness in AI agents, moving beyond mere performance metrics.

Methods

The researchers designed an experimental study using a custom-built "Othello Game Evaluation App." Participants watched pre-recorded Othello matches played by a human expert against three types of opponents: another human expert (H), Olivaw (O), or Saio (S). The study manipulated three independent variables:

  • Identity of the opponent: Human, Olivaw, or Saio.
  • Outcome of the match: Opponent won (+) or lost (-).
  • Color of the opponent: Black (B) or White (W).

Participants were asked to rate the human-likeness of the opponent player on a sliding scale from "100% human" to "100% computer." The study controlled for player competence, number of errors, and used pre-determined openings to ensure a balanced game state and avoid AI-specific opening strategies that might cue identity.

Datasets

Not discussed in the paper. The study used pre-recorded Othello matches generated by a human expert playing against different AI agents and human opponents.

Key Insights and Ideas

  • Losing is Human-Like: The most striking finding is that players who lost a match were perceived as significantly more human-like than those who won. This suggests a cognitive bias where we attribute human fallibility and imperfection to losing, while attributing success to superior, potentially non-human, capabilities.
  • Identity Didn't Matter (for Olivaw vs. Saio): Contrary to their initial hypothesis, the researchers did not find a significant difference in perceived human-likeness between Olivaw and Saio. This implies that, in this experimental setup, the specific learning approach of Olivaw (self-play reinforcement learning) did not inherently make it appear more human-like than a more traditional AI like Saio.
  • Opening Strategy Matters (Qualitatively): Participant comments hinted that the use of pre-defined, standard openings (controlled in the experiment) might have influenced their perception of human-likeness, as participants noted these openings are not typical of AI agents. This suggests that adherence to human-like strategic conventions can impact perception.

Key Results

A repeated-measures ANOVA revealed a significant effect of the outcome of the match on human-likeness ratings.

  • Opponent players were perceived as more human-like when losing (Mean = 73.17%) compared to when winning (Mean = 57.67%).
  • No significant effect of the opponent's identity (Human vs. Olivaw vs. Saio) was found on human-likeness ratings.
  • No significant interaction effects were found between the variables.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Ambiguity in "Human-Likeness" Question: Participants interpreted the "human-likeness" question differently. Some equated it with making mistakes (leading to high ratings for beginners or even AI), while others might have focused on strategic sophistication.
  • Influence of Pre-defined Openings: While controlled, the use of pre-defined openings might have inadvertently influenced perceptions, as participants commented on their non-AI nature.
  • Future Work:
    • Isolate Middle Game: Future studies could focus on the middle game of matches to avoid the influence of openings on perception.
  • Deeper Questioning: More specific questions about playing style (e.g., "defensive," "attacking," "random," "planned") could provide a more nuanced understanding of human-likeness.
  • Investigate "Why": Further research is needed to understand the cognitive mechanisms behind the "losing is human" bias.

Summary updated

BibTeX
@inproceedings{biancardi2023errare,
  author = {Béatrice Biancardi and Enrico Lauletta and Antonio Norelli and Alessandro Panconesi and Maurizio Mancini},
  title = {{Errare humanum est, perseverare autem diabolicum: A Follow-Up Study on the Human-Likeness of an AI Othello Player}},
  booktitle = {IVA},
  year = {2023},
  doi = {10.1145/3570945.3607327},
}

Loading related content...

Figure