Are Akpans Trick or Treat: Unveiling Helpful Biases in Assistant Systems

Jiao Sun, Hou Yu, Ji‐In Kim, Nanyun Peng

Abstract

Information-seeking AI assistant systems aim to answer users' queries about knowledge in a timely manner. However, both the human-perceived helpfulness of information-seeking assistant systems and its fairness implication are under-explored. In this paper, we study computational measurements of helpfulness. We collect human annotations on the helpfulness of dialogue responses, develop models for automatic helpfulness evaluation, and then propose to use the helpfulness level of a dialogue system towards different user queries to gauge the fairness of a dialogue system. Experiments with state-of-the-art dialogue systems, including ChatGPT, under three information-seeking scenarios reveal that existing systems tend to be more helpful for questions regarding concepts from highly-developed countries than less-developed countries, uncovering potential fairness concerns underlying the current information-seeking assistant systems.

Introduction

While AI assistants are typically evaluated on technical metrics like fluency or goal completion, human-perceived helpfulness remains under-explored. This paper addresses a critical fairness concern: the "helpfulness gap." If an AI provides a detailed recipe for a Western dish but responds to a query about a Beninese Akpan by asking if the user is playing a trick, the system is fundamentally biased. This research seeks to quantify how AI utility fluctuates based on a country's development level, using helpfulness as a core metric for fairness.

Methods

The researchers developed a three-stage framework to evaluate helpfulness and fairness. First, helpfulness was defined across three dimensions: relevance/coherence, usefulness, and informativeness. Second, they trained a RoBERTa-large based Helpfulness Classifier using human-annotated dialogues. This process involved a preliminary filtering stage to remove low-quality "AI-isms" from the training data. Finally, a fairness audit was conducted by categorizing 189 countries by their Human Development Index (HDI) and prompting models like GPT-3 and ChatGPT with queries related to local cuisines, holidays, and landmarks.

Overview of the Dialogue Helpfulness Evaluation and Geographical Fairness Analysis pipeline.

Key Results

The audit revealed a consistent downward trend: the lower a country's HDI, the less helpful the AI response. For GPT-3, the helpfulness ratio for cuisines dropped from 21.10% for high-HDI countries to 12.35% for low-HDI countries. While ChatGPT showed higher overall helpfulness, the bias remained prominent; in holiday-related queries, its helpfulness ratio was 57.90% for "Very High" HDI countries versus 34.21% for "Low" HDI countries. The models particularly struggled with being "Informative" for underrepresented regions, often providing vague or generic responses.

Helpfulness ratios across different HDI levels for GPT-3 and ChatGPT in three scenarios: Cuisine, Holiday, and Landmark.

Conclusion

This study highlights a significant geographical bias in state-of-the-art AI assistants. While these models are powerful, their utility is not equitably distributed, posing fairness challenges for global deployment. Future work should focus on improving factuality and developing debiasing techniques, such as knowledge-constrained decoding, to ensure that AI systems serve all users regardless of their geographical or cultural background. By highlighting these biases, this research serves as a wake-up call for developers to ensure AI intelligence is distributed more equitably across the globe.

Summary updated

Abstract

Information-seeking AI assistant systems aim to answer users' queries about knowledge in a timely manner. However, both the human-perceived helpfulness of information-seeking assistant systems and its fairness implication are under-explored. In this paper, we study computational measurements of helpfulness. We collect human annotations on the helpfulness of dialogue responses, develop models for automatic helpfulness evaluation, and then propose to use the helpfulness level of a dialogue system towards different user queries to gauge the fairness of a dialogue system. Experiments with state-of-the-art dialogue systems, including ChatGPT, under three information-seeking scenarios reveal that existing systems tend to be more helpful for questions regarding concepts from highly-developed countries than less-developed countries, uncovering potential fairness concerns underlying the current information-seeking assistant systems.
BibTeX
@misc{sun2022are,
  author = {Jiao Sun and Hou Yu and Ji‐In Kim and Nanyun Peng},
  title = {{Are Akpans Trick or Treat: Unveiling Helpful Biases in Assistant Systems}},
  year = {2022},
  eprint = {2205.12554},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.12554},
}
Figure