Are Akpans Trick or Treat: Unveiling Helpful Biases in Assistant Systems

Jiao Sun, Yu Hou, Jiin Kim, Nanyun Peng

TL;DR

This paper introduces a systematic way to measure the "helpfulness" of AI assistants and uncovers a significant geographical bias: state-of-the-art models like ChatGPT and GPT-3 are consistently more helpful when answering questions about highly-developed countries than less-developed ones. By decomposing helpfulness into relevance, usefulness, and informativeness, the authors highlight a hidden fairness gap that could discourage marginalized groups from using these systems.

What Problem Does This Paper Address?

While we often evaluate AI assistants on technical metrics like "goal success rate" or "fluency," we rarely measure how helpful they actually feel to a human user. More importantly, we don't know if that helpfulness is distributed fairly across the globe.

If you ask an AI how to make German home fries, it might give you a detailed recipe. But if you ask about Akpans (a Beninese dessert), the model might respond with confusion or even nonsensical fear (e.g., "Are you playing a trick on me?"). This discrepancy isn't just a technical quirk; it’s a fairness issue. If AI systems perform poorly for users from specific regions, those users are disproportionately discouraged from using the technology, creating a feedback loop that further marginalizes their data and needs.

Methods

The authors developed a multi-stage pipeline to quantify and analyze this bias:

  1. Defining Helpfulness: They broke "helpfulness" down into three core dimensions:
  2. Relevant & Coherent: Is the response on-topic and logical?
  3. Useful: Does it actually address the user's question or move the task forward?
  4. Informative: Does it provide specific, non-generic details?
  5. Data Collection & Annotation: They collected human annotations on both human-human (TaskMaster) and human-machine (GPT-3 generated) dialogues to create a ground-truth dataset for helpfulness.
  6. The Helpfulness Classifier: They trained a RoBERTa-based classifier to automatically evaluate these three dimensions. To ensure the model learned nuanced helpfulness rather than just spotting "gibberish," they used a secondary "filtering classifier" to remove low-quality, repetitive, or obviously broken machine responses from the training set.
  7. Fairness Analysis: Using the UN’s Human Development Index (HDI), they categorized countries into four tiers (Very High to Low). They then queried models (GPT-3, ChatGPT, BLENDER) about cuisines, holidays, and landmarks from these countries and used their classifier to measure the performance gap.
Figure 1: We collect a high-quality human-annotated helpfulness dataset, build an automatic helpfulness classifier, and then investigate the associated fairness issue. Data are annotated from human-human and human-machine dialogues. Using partial negative examples, we train a filtering classifier to filter out instances that could have been too easy for the model to learn and introduce bias. We then use this filter to clean annotated data and use the cleaned data to build the helpfulness classifier.

Datasets

  • TaskMaster-1 & 2: Used as a proxy for general goal-oriented human-human dialogues.
  • Wikipedia: Used to source thousands of factual concepts (5,443 cuisines, 754 holidays, 4,735 landmarks) linked to specific countries.
  • New Annotated Dataset: The authors introduced a new dataset of 1,192 GPT-3 generated dialogues with fine-grained human labels for the three dimensions of helpfulness.

Key Insights and Ideas

  • The "Helpfulness" Decomposition: By treating relevance as a prerequisite for usefulness and informativeness, the authors created a more robust metric for assistant quality than simple accuracy.
  • Geographical Bias is Pervasive: The bias isn't just about the model "not knowing" a fact; it affects every level of the interaction. For less-developed countries, models struggle even to remain coherent or relevant.
  • Hallucination as a Bias Signal: In some cases, GPT-3 produced bizarre, defensive responses to queries about less-developed regions, suggesting that the model's "uncertainty" in these low-data areas manifests as unhelpful behavior.
  • Zero-Shot Limitations: The study found that their specialized RoBERTa classifier actually outperformed zero-shot GPT-4 and ChatGPT at identifying helpful responses, highlighting the value of task-specific training.

Key Results

The quantitative results confirmed a clear "helpfulness gradient" tied to a country's development level:

  • GPT-3 Cuisine Scenario: The ratio of helpful responses was 20.15% for "Very High" HDI countries but dropped to 12.35% for "Low" HDI countries.
  • ChatGPT Performance: While ChatGPT is generally more helpful than older models, the gap remains. In the holiday scenario, its helpfulness score for "Very High" HDI countries was significantly higher than for "Low" HDI countries.
  • Dimension Breakdown: As shown in the figure below, the trend of higher performance for developed countries stays consistent across all three dimensions (relevance, usefulness, and informativeness).
Breakdown of helpfulness dimensions for GPT-3 and ChatGPT across different HDI levels. Note the downward trend as development levels decrease.

Limitations and Future Work

  • Debiasing: The paper identifies the problem but does not yet provide a solution. The authors suggest that "constrained decoding"—forcing the model to use specific knowledge during generation—could be a promising fix.
  • Factuality: The helpfulness classifier measures if a response looks helpful, but it doesn't explicitly verify if the facts are true. A model could be "helpfully" wrong (hallucinating a recipe).
  • Scope: The study focused on factual information seeking. Future work could explore if these biases persist in more subjective areas like emotional support or creative writing.

Summary updated

Abstract

Information-seeking AI assistant systems aim to answer users' queries about knowledge in a timely manner. However, both the human-perceived helpfulness of information-seeking assistant systems and its fairness implication are under-explored. In this paper, we study computational measurements of helpfulness. We collect human annotations on the helpfulness of dialogue responses, develop models for automatic helpfulness evaluation, and then propose to use the helpfulness level of a dialogue system towards different user queries to gauge the fairness of a dialogue system. Experiments with state-of-the-art dialogue systems, including ChatGPT, under three information-seeking scenarios reveal that existing systems tend to be more helpful for questions regarding concepts from highly-developed countries than less-developed countries, uncovering potential fairness concerns underlying the current information-seeking assistant systems.
BibTeX
@misc{sun2022are,
  author = {Jiao Sun and Yu Hou and Jiin Kim and Nanyun Peng},
  title = {{Are Akpans Trick or Treat: Unveiling Helpful Biases in Assistant Systems}},
  year = {2022},
  eprint = {2205.12554},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.12554},
}

Loading related content...

Figure