FrameIt: Ontology Discovery for Noisy User-Generated Text

Dan Iter, Alon Halevy, Wang-Chiew TanEMNLP 2018

TL;DR

This paper introduces FRAMEIT, a system that helps users discover application-specific ontologies and train semantic role labeling (SRL) models directly from noisy user-generated text. It addresses the challenge of extracting structured information when the desired schema isn't known beforehand, a common issue with real-world text data.

What Problem Does This Paper Address?

Extracting structured data from text is crucial for many NLP applications, but it often relies on pre-defined ontologies (schemas) that are application-dependent and frequently unknown. Existing approaches either use general linguistic ontologies (like FrameNet) that don't capture application-specific nuances or require significant manual effort to define new schemas and create training data. This paper tackles the problem of automatically discovering these application-specific frames and training extractors for them, especially from noisy user-generated text where standard linguistic structures might be less prevalent.

Methods

FRAMEIT offers a two-phase workflow:

  1. Exploration and Discovery: Users interact with the corpus to identify patterns and define frames (schemas) and their slots (attributes). This phase uses several exploration features:
  2. Discovery: Finding common patterns and defining frames/slots.
  3. Granularity: Determining the appropriate level of detail for frames.
  4. Para-phrasing Detection: Identifying different ways concepts are expressed.
  5. Slot Dictionaries: Building lists of possible values for slots.
  6. Exploration Operators: Random sentence sampling, keyword/lemma search, and identifying common structures (using a novel rank score to weigh structure specificity).
  7. Nearby Sentences: Finding semantically similar sentences to expose variations.
  8. Mapping to Existing Frame Systems: Leveraging FrameNet, PropBank, and WordNet to aid discovery.
  9. SRL Training and Refinement:
    • Bootstrapping with Weak Supervision: FRAMEIT uses the exploration phase to automatically generate noisy training data. This data is then used to bootstrap an initial SRL model.
  10. Active Learning: The system provides an interface for users to refine the SRL model by labeling challenging examples, iteratively improving precision and recall.

Datasets

The paper evaluates FRAMEIT on three datasets of user-generated text:

  • HappyDB: A corpus of 100K crowdsourced "happy moments."
  • TripAdvisor hotel reviews: A subsampled dataset of 78K reviews.
  • ANES 2008 presidential election survey: 2K free-form responses.

No new benchmarks were introduced; instead, existing datasets were used to demonstrate FRAMEIT's effectiveness in capturing application-specific concepts.

Key Insights and Ideas

  • Application Frames vs. Linguistic Frames: The paper strongly argues that application-specific frames (like "MEALS" or "PROMOTION" in HappyDB) are qualitatively different from general linguistic frames (like FrameNet's "Food" or "Commerce_buy"). Relying solely on linguistic frames can lead to either low precision or low recall for application-specific tasks.
  • The Value of Weak Supervision and Active Learning: FRAMEIT demonstrates that combining automatically generated weak supervision with targeted active learning can efficiently build effective SRL models, even with noisy initial data.
  • User-Centric Ontology Discovery: The system empowers users who may be domain experts but not NLP specialists to define relevant ontologies by interacting with their data.
  • Noisy Text is a Distinct Challenge: The focus on user-generated text highlights the need for systems that can handle the inherent messiness and variability of such data.

Key Results

  • Corpus Coverage: FRAMEIT achieved significant corpus coverage, capturing over 70% of sentences in HappyDB and ANES 2008, and 62% in TripAdvisor reviews, with a modest amount of work.
  • SRL Performance: The trained SRL models achieved F1 scores around 0.70-0.80 on average across frames and datasets, demonstrating good performance in extracting structured information.
  • Active Learning Improvement: Active learning iterations showed significant improvements in F1 scores, reducing F1 error by 24%-46% for some models.
  • Efficiency: Discovering and training frames was efficient, with exploration taking minutes per frame and active learning iterations requiring minimal labeling time.

Limitations and Future Work

  • User Effort: While FRAMEIT reduces manual effort, it still requires user interaction for ontology discovery and active learning. The paper acknowledges that quantifying human effort is challenging.
  • Ontology Quality: The quality of the final ontology and SRL model is limited by the challenges of automatically suggesting frames and determining their granularity.
  • Future Work: The authors propose making FRAMEIT more active in suggesting frames and granularities. They also aim to develop models that can automatically discover all aspects of an activity and group related terms, moving towards more model-based structure generation. Specifically, they want to address:
  • Automatically finding diverse sentences for a frame without changing its meaning (e.g., expanding "dinner" to include "lunch" and "breakfast").
  • Automatically discovering and grouping related aspects of an activity (e.g., for "meals," discovering "bought," "cooked," "delicious," "gross").

Summary updated

BibTeX
@inproceedings{iter2018frameit,
  author = {Dan Iter and Alon Halevy and Wang-Chiew Tan},
  title = {{FrameIt: Ontology Discovery for Noisy User-Generated Text}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}

Loading related content...

Figure