Abstract
Subjectivity is the expression of internal opinions or beliefs which cannot be objectively observed or verified, and has been shown to be important for sentiment analysis and word-sense disambiguation. Furthermore, subjectivity is an important aspect of user-generated data. In spite of this, subjectivity has not been investigated in contexts where such data is widespread, such as in question answering (QA). We therefore investigate the relationship between subjectivity and QA, while developing a new dataset. We compare and contrast with analyses from previous work, and verify that findings regarding subjectivity still hold when using recently developed NLP architectures. We find that subjectivity is also an important feature in the case of QA, albeit with more intricate interactions between subjectivity and QA performance. For instance, a subjective question may or may not be associated with a subjective answer. We release an English QA dataset (SubjQA) based on customer reviews, containing subjectivity annotations for questions and answer spans across 6 distinct domains.
Introduction: The Subjectivity Gap
Most progress in Question Answering (QA) has been built on a foundation of objective, encyclopedic knowledge. Benchmarks like SQuAD rely on Wikipedia, where answers are typically verifiable facts. However, real-world user queries often involve "private states"—internal opinions or beliefs that cannot be objectively verified. For example, a user might ask "Is this hotel actually romantic?" or "Is the plot of this movie too slow?"
Current QA systems, trained primarily on factual data, often struggle when encountering user-generated content like reviews. These models haven't learned to navigate the linguistic markers of opinion. To address this, researchers from the University of Copenhagen and Megagon Labs introduced SubjQA, a benchmark designed to move NLP beyond factual retrieval and into the nuanced world of human subjectivity.
Dataset Construction and Methodology
The authors developed a sophisticated pipeline to ensure the dataset was inherently subjective and required semantic reasoning rather than simple keyword matching.
The construction process involved several key stages:
- Opinion Extraction: Identifying
<opinion, aspect>pairs (e.g.,<"hilarious", "book">) within reviews. - Neighborhood Model Construction: Using matrix factorization to find "neighbors"—concepts that are semantically related but linguistically distinct. This forces models to understand the broader context of a review.
- Topic and Review Selection: Selecting relevant reviews based on the extracted opinions and their semantic neighbors.
- Question Generation and Labeling: Human annotators generated natural questions based on selected topics and labeled both questions and answer spans on a 1–5 subjectivity scale.
Dataset Characteristics
SubjQA contains over 10,000 English examples across six domains: TripAdvisor, Restaurants, Movies, Books, Electronics, and Grocery. Analysis of the dataset revealed several unique properties:
- Subjectivity Correlation: There is a strong link between question and answer subjectivity. Subjective questions (79.8% of the dataset) typically yield subjective answers.
- The "How" Factor: Unlike factual datasets dominated by "Who" or "When" questions, 58% of SubjQA questions start with "How" (e.g., "How is the battery life?"). This necessitates a deeper understanding of descriptive language.
- Linguistic Mismatch: Subjective QA is inherently more difficult because questions and answers often use entirely different vocabularies, requiring complex reasoning rather than simple entity matching.
Experimental Results
The researchers evaluated several state-of-the-art models, including BERT, on the SubjQA dataset. The results highlight a significant "factual bias" in current architectures.
Key findings include:
- Performance Drop: A BERT model achieving 92.9% F1 on the factual SQuAD dataset saw its performance drop to a mean of 74.1% F1 across SubjQA domains.
- Multi-Task Learning (MTL) Gains: By using an MTL framework to simultaneously predict answer spans and subjectivity labels, performance improved to 76.3% F1. This demonstrates that explicitly modeling subjectivity helps systems navigate review data more effectively.
- Domain Variation: Models performed better in domains like "Movies" and "Books," where opinions are often explicit, but struggled in "Grocery" and "Electronics," where the distinction between technical specifications and subjective experience is more blurred.
Conclusion
SubjQA represents a significant step toward teaching machines to understand human opinion. While current transformer models excel at factual retrieval, they face substantial challenges in subjective contexts. Future work will focus on scaling these methods and integrating subjectivity-aware models into real-world applications like customer support, where understanding user sentiment is as critical as providing factual information.
Summary updated