TL;DR
This paper introduces DP, an automated system designed to detect presuppositions in survey questions, which can lead to inaccurate responses. DP uses linguistic rules and syntactic analysis to flag potentially problematic questions, aiding survey designers in creating clearer and more reliable questionnaires.
What Problem Does This Paper Address?
Survey questions often contain implicit assumptions, known as presuppositions. For example, the question "When did you graduate from college?" presupposes that the respondent did graduate from college. If a respondent never attended college, they are forced to either not answer or provide a false response, leading to biased survey data. While skilled survey designers aim to avoid such issues, subtle presuppositions can be missed. This paper addresses the challenge of automatically identifying these problematic presuppositions in survey questions to improve the quality and accuracy of survey data.
Methods
The core of the paper is the development of DP, a "Detector for Presuppositions." DP operates in several stages:
- Content Analysis & Indicator Identification: The researchers analyzed survey questions with known presupposition problems to create a list of "indicators" – specific words, phrases, or grammatical structures that signal a presupposition. They also identified indicators for the absence of presuppositions.
- Linguistic Analysis Pipeline:
- Part-of-Speech Tagging: Eric Brill's rule-based tagger is used to assign a grammatical category (e.g., noun, verb, adjective) to each word in a question.
- Syntactic Parsing: Abney's SCOL/CASS system, an "extreme bottom-up parser," analyzes the grammatical structure of the sentence, identifying chunks of information.
- Decision Tree Structure: The identified indicators are integrated into a decision tree. This tree first checks for indicators of no presupposition. If none are found, it then checks for indicators of a presupposition. Finally, it assesses the reliability of the indicator to determine if a problem is likely.
- Classification: DP not only detects a presupposition but also attempts to classify its type (e.g., presupposing an action, a habit, a possession) by using expressions from a predefined table. This classification helps the survey designer understand the nature of the potential problem.
Datasets
- Corpus of Survey Questions: The initial development and analysis of presupposition indicators were based on a corpus of 550 questions from U.S. Census Bureau questionnaires.
- Problematic and Unproblematic Samples: From this corpus, 22 questions identified as containing problematic presuppositions (rated by human experts) and 22 questions rated as unproblematic were selected for analysis and testing.
- Human Ratings: Approximately 90 questions were rated by three human experts on a four-point scale to serve as a benchmark for evaluating DP's performance. These ratings were later converted to a binary "problem" vs. "no problem" scale.
Key Insights and Ideas
- Subtle Presuppositions are Tricky: Even well-designed questionnaires can contain subtle presuppositions that are easily missed by human designers, highlighting the need for automated tools.
- Indicator-Based Approach is Effective: By focusing on specific linguistic indicators and syntactic structures, DP can effectively identify presuppositions without needing to understand the full semantic context of a question.
- Classification Adds Value: Identifying the type of presupposition (e.g., habit, possession) provides actionable feedback to survey designers, allowing them to target their revisions more effectively.
- Trade-off Between Recall and Precision: The updated version of DP prioritizes recall (finding all potential presuppositions) over precision (minimizing false alarms). This is a pragmatic choice, as the ultimate decision of whether a presupposition is problematic rests with the human survey methodologist.
Key Results
- Initial DP Performance: The initial version of DP showed comparable agreement with human raters to the agreement between human raters themselves. When evaluated against a strict "complete agreement" criterion (Pcomp), DP achieved a hit rate of 0.72 and a false alarm rate of 0.35.
- Updated DP Performance: After modifications, the updated DP showed improved performance, particularly with the Pcomp criterion. It achieved a hit rate of 0.90 and a false alarm rate of 0.52, with a perfect recall rate of 1.00. The d' score, measuring discrimination power, significantly improved to 1.3.
- Decision Tree Update Impact: The update to the decision tree structure, prioritizing presupposition indicators over indicators against them, was identified as the primary driver of performance improvement.
Limitations and Future Work
- Context Independence: DP currently evaluates questions in isolation, without considering the context of preceding questions. This can lead to frequent detections of presuppositions that have already been established (e.g., multiple questions about "this person" after the first question establishes the referent).
- Future Work: DIP (Detector of Incorrect Presuppositions): The authors propose developing a "Detector of Incorrect Presuppositions" (DIP) that would incorporate context. This would involve mechanisms to determine if a presupposed entity has been established in the discourse.
- Integrating Context Models: Future work could involve integrating models like Kintsch's Construction-Integration (CI) model or Latent Semantic Analysis (LSA) to perform context analyses and resolve reference ambiguity.
- Handling Questionnaire Structure: A significant challenge for DIP is handling the flexible context in questionnaires, where the distance between a question establishing a presupposition and a question building on it can vary widely. This requires computationally efficient methods for managing and querying past context.
Summary updated