A Hybrid Approach to Finding Relevant Social Media Content for Complex Domain Specific Information Needs

Delroy Cameron, Amit Sheth, Nishita Jaykumar, Krishnaprasad Thirunarayan, Gaurish Anand, Gary A. SmitharXiv 2014 · arXiv

This paper presents a hybrid approach to finding relevant social media content for complex, domain-specific information needs, specifically targeting the epidemiology of prescription drug abuse. The authors introduce PREDOSE, a Semantic Web platform that moves beyond simple keyword matching by integrating ontologies, lexicons, and rule-based query interpretation.

TL;DR

The researchers developed a hybrid search framework that uses a two-level context-free grammar (CFG) to bridge the gap between formal ontologies and the messy, "intelligible constructs" (like dosage and sentiment) found in social media. By combining structured background knowledge with domain-specific rules, their system significantly outperforms general-purpose search engines in retrieving relevant documents for complex epidemiological queries.

What Problem Does This Paper Address?

Standard search engines (like Google) and even contemporary semantic search systems often fail when faced with complex, domain-specific queries. In the world of prescription drug abuse epidemiology, a researcher doesn't just want to find mentions of a drug; they want to find mentions of "excessive dosage," "negative sentiments," or specific "routes of administration" (e.g., snorting vs. swallowing).

The challenge is a "misalignment" between user needs and formal knowledge models. Ontologies are great for structured concepts, but they rarely capture the nuances of how people talk about intensity, frequency, or fuzzy intervals (e.g., "a bit more than 30mg" or "5 years ago"). These "intelligible constructs" are vital for domain experts but are often missing from formal ontologies.

Methods

The core of the approach is a hybrid information retrieval framework consisting of three main components: a Query Processor, a Semantic Metadata Extractor, and a Query Matcher.

System Architecture

The standout feature is the use of a two-level Context-Free Grammar (CFG):

  1. Top-Level CFG: Defines broad "template patterns" (e.g., <ENTITY> <PRONOUN> <DOSAGE> <INTENSITY>). This allows experts to build complex queries by selecting categories rather than just typing keywords.
  2. Low-Level CFG: Interprets specific expressions within those categories. It handles four types of data:
  3. Ontological Concepts: Standard drug names and their slang synonyms (e.g., mapping "bupe" to Buprenorphine).
  4. Lexicon Concepts: Emotions and sentiments (e.g., "felt pretty good").
  5. Lexico-Ontology Concepts: Concepts with partial representation in both, like side effects.
  6. Rule-Based Expressions: Complex numeric and temporal data like dosage ("greater than 4mg") and frequency ("twice daily").

The system uses SystemT and Annotation Query Language (AQL) to extract these patterns from the text corpus offline, creating a database of document annotations that the Query Matcher can quickly filter.

Query Interpretation Workflow

Datasets

The system was evaluated using a massive corpus of over 1 million web forum posts related to drug abuse.

  • Slang Mapping: The researchers used a gold-standard dataset of 600 posts to observe slang usage, finding a staggering 33:1 ratio of slang terms to standard labels for drugs like Buprenorphine.
  • Evaluation Baseline: Because no standardized "gold standard" exists for this specific domain, the authors conducted a user-driven evaluation comparing PREDOSE against Google, Hakia, and DuckDuckGo.

Key Insights and Ideas

  • Slang is King: In social media drug subcultures, slang isn't just common; it's the primary way people communicate. A search system that doesn't map "subbies" or "tex" to Buprenorphine will miss the vast majority of relevant data.
  • The Power of "Intelligible Constructs": The ability to interpret the semantics of operators like "greater than" (>) is a major differentiator. While Google might find the number "4" and the word "mg" near each other, PREDOSE understands the intent of finding dosages exceeding a specific threshold.
  • Template-Based Querying: By forcing users to use templates (as shown in the derivation table below), the system ensures that the query is structured in a way that the underlying grammar can interpret, reducing the ambiguity of natural language.
Derivation of system query strings using the CFG

Key Results

In a comparative evaluation with six human evaluators:

  • Scenario 1 (Excessive Dosage): PREDOSE retrieved 16/20 relevant results. Google came in second with 14/20, but its results were often serendipitous—it didn't actually "understand" the dosage constraint, whereas PREDOSE's results all strictly met the ">4mg" criteria.
  • Scenario 2 (Negative Sentiments): PREDOSE retrieved 11/20 relevant results, outperforming Google (9/20). PREDOSE was able to identify diverse sentiment expressions (e.g., "rough," "nauseous," "hated") that weren't explicitly in the user's initial query but were captured by the low-level grammar.
  • Precision: The system's ability to filter by specific template classes (like <ROA> for Route of Administration) allowed for much higher precision in finding specific behaviors compared to keyword-based systems.

Limitations and Future Work

  • Scalability of Grammar: Currently, the 60 template patterns and 150+ low-level productions were created manually. The authors acknowledge that manual specification is cumbersome and suggest that future work should focus on automated grammar composition.
  • Entity Disambiguation: The system struggles with highly ambiguous terms (e.g., "alcohol" or "heroin" appearing in non-abuse contexts). Implementing context-aware disambiguation is a priority.
  • Ranking: The current system retrieves relevant documents but doesn't rank them by "relevance score." Adding a sophisticated ranking algorithm would improve the user experience for high-frequency topics.
  • Domain Portability: While the framework is designed to be portable, it still requires significant domain expertise to set up for new fields (e.g., cardiology or materials science).

Summary updated

Abstract

While contemporary semantic search systems offer to improve classical keyword-based search, they are not always adequate for complex domain specific information needs. The domain of prescription drug abuse, for example, requires knowledge of both ontological concepts and 'intelligible constructs' not typically modeled in ontologies. These intelligible constructs convey essential information that include notions of intensity, frequency, interval, dosage and sentiments, which could be important to the holistic needs of the information seeker. We present a hybrid approach to domain specific information retrieval (or knowledge-aware search) that integrates ontology-driven query interpretation with synonym-based query expansion and domain specific rules, to facilitate search in social media. Our framework is based on a context-free grammar (CFG) that defines the query language of constructs interpretable by the search system. The grammar provides two levels of semantic interpretation: 1) a top-level CFG that facilitates retrieval of diverse textual patterns, which belong to broad templates and 2) a low-level CFG that enables interpretation of certain specific expressions that belong to such patterns. These low-level expressions occur as concepts from four different categories of data: 1) ontological concepts, 2) concepts in lexicons (such as emotions and sentiments), 3) concepts in lexicons with only partial ontology representation, called lexico-ontology concepts (such as side effects and routes of administration (ROA)), and 4) domain specific expressions (such as date, time, interval, frequency and dosage) derived solely through rules. Our approach is embodied in a novel Semantic Web platform called PREDOSE developed for prescription drug abuse epidemiology. Keywords: Knowledge-Aware Search, Ontology, Semantic Search, Background Knowledge, Context-Free Grammar
BibTeX
@misc{cameron2014hybrid,
  author = {Delroy Cameron and Amit Sheth and Nishita Jaykumar and Krishnaprasad Thirunarayan and Gaurish Anand and Gary A. Smith},
  title = {{A Hybrid Approach to Finding Relevant Social Media Content for Complex Domain Specific Information Needs}},
  year = {2014},
  eprint = {1411.3761},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1411.3761},
}
Figure