Using Frame Semantics for Knowledge Extraction from Twitter

Anders Søgaard, Barbara Plank, Héctor Martínez Alonso

Abstract

Knowledge bases (KBs) such as Wikipedia and Freebase are essential for modern AI applications, yet they frequently suffer from low recall and reporting bias, missing niche relations and common-sense facts. Twitter represents a potential real-time goldmine for filling these gaps, but its informal, "noisy" nature often causes traditional Natural Language Processing (NLP) tools to fail. This paper demonstrates that frame semantic parsing is a robust tool for extracting structured knowledge from Twitter. By focusing on high-level conceptual "frames" rather than strict grammatical rules, the researchers achieved a significant reduction in extraction errors, successfully harvesting real-world facts that are often absent from formal databases.

Introduction

Extracting facts from social media is notoriously difficult due to the "domain shift" from formal news text to informal tweets characterized by missing punctuation, slang, and non-standard grammar. Most NLP tools are trained on clean datasets and struggle with the chaotic structure of Twitter. The authors address this challenge by investigating whether frame semantics—a top-down approach to meaning—can provide the necessary resilience for robust knowledge extraction.

Methodology

The researchers compared four distinct approaches for extracting "triples" (Subject-Relation-Object facts) from tweets:

  1. Syntactic Dependency Parsing: A bottom-up method mapping the full grammatical structure of a sentence.
  2. Semantic Role Labeling (SRL): An approach identifying "who did what to whom" based on verb predicates.
  3. Frame Semantic Parsing: Utilizing the SEMAFOR system, this top-down approach assigns conceptual templates (frames) like Research or Giving to text.
  4. Rule-based Extraction (REVERB): An off-the-shelf system using patterns to identify relations.

The core hypothesis is that frame semantics is more robust because it relies less on "function words" (e.g., the, of, a) and strict syntax, which are often mangled in tweets. Instead, it focuses on core "targets" or keywords that evoke specific meanings. To further refine the results, the authors employed sentiment analysis as a proxy for factuality, assuming that "neutral" tweets are more likely to contain objective facts than highly opinionated ones.

Examples of extracted triples and their calculated neutrality scores.

Datasets and Evaluation

To evaluate these methods, the authors utilized several datasets:

  • Twitter Fact Dataset: 70,000 tweets mentioning 60 specific entities (people, locations, and organizations) sourced from Freebase.
  • New Benchmark: A newly introduced dataset of tweets multiply annotated by three experts using FrameNet 1.5 standards.
  • Training Data: The SEMAFOR parser was trained on standard FrameNet exemplar sentences and SemEval 2007 data.

Results and Key Insights

The experimental results highlighted the superiority of frame semantic parsing in handling informal text. Professional annotators rated the extractions based on several criteria, including intelligibility and correctness.

Performance comparison of different extraction systems across various metrics.

Key findings include:

  • Superior Intelligibility: Frame semantic parsing was the most effective, with 77% of its extractions being intelligible to humans, significantly outperforming syntactic parsing (63%) and REVERB (64%).
  • High Accuracy: Among intelligible extractions, frame semantics produced "true" facts 76% of the time.
  • Resilience to Noise: While syntactic parsers experienced a dramatic performance drop when moving from news to Twitter, frame semantic parsers remained stable.
  • Knowledge Discovery: The potential for discovering new knowledge is high; less than 1% of the entity pairs found via dependency parsing were already present in Freebase.

The "top-down" nature of frame semantics allows it to ignore local noise (like hashtags or typos) by focusing on a "trigger" word and its associated arguments. Furthermore, the study confirmed that Twitter contains many facts absent from encyclopedic sources, though sentiment filtering is necessary to separate these from sarcasm and noise.

Limitations and Future Work

Despite its success, the system faces several hurdles. Entity disambiguation remains a challenge, as the system may confuse different entities with the same name (e.g., a movie vs. a city). There is also a trade-off between recall and precision; while frame semantics is more accurate, it currently extracts fewer total facts than some other methods.

Future research could involve more sophisticated factuality models to detect sarcasm and metaphors more effectively than simple sentiment analysis. Additionally, improving argument disambiguation—moving beyond "head word" extraction—could prevent information loss and provide more nuanced facts.

Summary updated

Abstract

Knowledge bases have the potential to advance artificial intelligence, but often suffer from recall problems, i.e., lack of knowledge of new entities and relations. On the contrary, social media such as Twitter provide abundance of data, in a timely manner: information spreads at an incredible pace and is posted long before it makes it into more commonly used resources for knowledge extraction. In this paper we address the question whether we can exploit social media to extract new facts, which may at first seem like finding needles in haystacks. We collect tweets about 60 entities in Freebase and compare four methods to extract binary relation candidates, based on syntactic and semantic parsing and simple mechanism for factuality scoring. The extracted facts are manually evaluated in terms of their correctness and relevance for search. We show that moving from bottom-up syntactic or semantic dependency parsing formalisms to top-down frame-semantic processing improves the robustness of knowledge extraction, producing more intelligible fact candidates of better quality. In order to evaluate the quality of frame semantic parsing on Twitter intrinsically, we make a multiply frame-annotated dataset of tweets publicly available.
BibTeX
@inproceedings{sgaard2015using,
  author = {Anders Søgaard and Barbara Plank and Héctor Martínez Alonso},
  title = {{Using Frame Semantics for Knowledge Extraction from Twitter}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2015},
  doi = {10.1609/aaai.v29i1.9524},
}
Figure