Pregnant Questions: The Importance of Pragmatic Awareness in Maternal Health Question Answering

Neha Srikanth, Rupak Sarkar, Heran Mane, Elizabeth Aparicio, Quynh C. Nguyen, Rachel Rudinger, Jordan L. Boyd-Graber

Introduction

In high-stakes domains like maternal and infant health, a user’s question often carries "pregnant" meaning—implicit assumptions that may be medically inaccurate or even life-threatening. For example, if a mother asks about using non-dairy milk for a newborn, a standard AI might simply list brands, failing to correct the dangerous assumption that newborns can safely consume non-dairy milk instead of breast milk or formula.

Standard Large Language Models (LLMs) often suffer from sycophancy, tending to align with the user’s view rather than challenging it. This paper argues that for a QA system to be truly helpful, it must possess pragmatic awareness: the ability to infer the asker's intent and address unexpressed information needs, especially when those needs involve correcting harmful misconceptions.

Figure 1: An example of how a standard QA system might miss a false underlying assumption compared to a pragmatically aware system.

Methodology

The researchers developed a framework to move beyond factoid-based QA into the realm of linguistic pragmatics. Their approach involves categorizing hidden beliefs into two types: Presuppositions (assumptions taken for granted) and Implicatures (beliefs suggested by context).

To address these, the authors modified the standard "Retrieve-Rerank-Read" architecture into an Inference-Augmented Pipeline. Instead of searching only for the original question, the system extracts potential inferences, retrieves specific evidence for each to determine its veracity, and feeds both the question and verified inferences into the "Reader" model (e.g., GPT-3.5, Mistral-7B). This ensures the final response explicitly addresses underlying assumptions.

Figure 3: Comparison between a standard baseline pipeline and the proposed Pragmatic Inference Augmentation pipeline.

Dataset and Insights

The study utilized a novel dataset of 2,727 pragmatic inferences derived from 500 questions across three sources: ROSIE (a domain-specific QA system), Reddit (real-world maternal subreddits), and Natural Questions (Google search queries). Expert-annotated veracity labels and supporting evidence from trusted medical sources (e.g., CDC, Mayo Clinic) were included.

Key findings from the dataset analysis include:

  • Detection Difficulty: False beliefs are often presented as implicatures (softer suggestions) rather than hard presuppositions, making them harder for standard NLP tools to detect.
  • The Expert Standard: Human health experts naturally provide context to ensure askers do not act on dangerous misunderstandings, a standard current LLMs struggle to meet. GPT-3.5, for instance, failed to generate 63% of the inferences identified by human experts.

Results and Safety

The inference-augmented approach showed significant improvements over baseline models:

  • Human Preference: Expert annotators rated augmented answers as better or more complete 75% of the time when questions contained plausible false assumptions.
  • Completeness: Augmented models achieved higher completeness scores (Mean Likert 4.45/5) than default models.
  • Safety: By explicitly prompting the model to address assumptions, the system was significantly more likely to debunk harmful myths, such as the incorrect belief that certain medications should be taken before a vaccine to prevent fever.

Conclusion

While promising, the study notes that pragmatic norms vary across cultures and languages. Future work should focus on "inherent" pragmatic awareness, where retrievers automatically search for evidence challenging a query's intent. Furthermore, systems must learn to challenge user beliefs tactfully to maintain trust while ensuring medical safety.

Summary updated

BibTeX
@inproceedings{srikanth2024pregnant,
  author = {Neha Srikanth and Rupak Sarkar and Heran Mane and Elizabeth Aparicio and Quynh C. Nguyen and Rachel Rudinger and Jordan L. Boyd-Graber},
  title = {{Pregnant Questions: The Importance of Pragmatic Awareness in Maternal Health Question Answering}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.403},
}
Figure