How Important is Syntactic Parsing Accuracy? An Empirical Evaluation on Rule-Based Sentiment Analysis

Carlos Gómez-Rodríguez, Iago Alonso-Alonso, David VilaresarXiv 2017 · DOI · arXiv

TL;DR

State-of-the-art syntactic parsers have become highly accurate, but this accuracy comes at a steep computational cost. This paper demonstrates that for downstream rule-based sentiment analysis, expensive state-of-the-art parsers perform no better than much faster, less accurate alternatives. For practical applications, developers should prioritize parsing speed over accuracy.

What Problem Does This Paper Address?

Many Natural Language Processing (NLP) applications rely on syntactic parsing—the process of uncovering the internal structural relationships of a sentence—to extract meaning. For instance, understanding who likes what in a sentence requires knowing which noun connects to which verb.

While parsing algorithms have advanced significantly, the most accurate models require substantial computational resources. This poses a challenge for large-scale, real-time applications like social media sentiment analysis (SA), where millions of texts are processed daily. The authors tackle a fundamental engineering question: Does higher syntactic parsing accuracy actually translate to better performance in downstream sentiment analysis, or are we wasting computational power?

Methods

The authors conduct a task-oriented evaluation by plugging different dependency parsers into a state-of-the-art sentiment analysis system called UUUSA (Universal, Unsupervised, Uncovered Sentiment Analysis). UUUSA calculates text polarity (positive, negative, or neutral) by traversing a sentence's dependency tree and applying grammatical rules to handle linguistic phenomena like negation, intensification, and conditional clauses.

To see how parsing quality affects sentiment accuracy, they test four distinct dependency parsers that span the speed-accuracy spectrum:

  1. MaltParser: A highly optimized, transition-based parser using greedy deterministic search. It is incredibly fast but less accurate.
  2. Stanford RNN Parser: A popular transition-based neural network parser that balances robust accuracy with greedy search.
  3. TurboParser: A graph-based parser using approximate variational inference, offering high accuracy at a higher computational cost.
  4. YaraParser: A transition-based parser utilizing beam search and dynamic oracles to achieve state-of-the-art accuracy.

The syntactic structures generated by these parsers are illustrated using two dominant linguistic representations: constituency grammar (Figure 1) and dependency grammar (Figure 2).

An example of a constituency parse tree, which breaks a sentence down into phrasal constituents.
An example of a dependency parse tree, which maps binary grammatical relations directly between words.

Datasets

The parsers were trained and evaluated using the English Universal Treebank v2.0, a large corpus containing nearly 40,000 dependency trees mapped from the Penn Treebank.

For the downstream sentiment analysis evaluation, the authors used three standard benchmarks:

  • Taboada and Grieve (2004): 400 long, general-domain reviews.
  • Pang and Lee (2004): 2,000 long movie reviews.
  • Pang and Lee (2005): 1,821 short, single-sentence movie reviews.

Key Insights and Ideas

  • The "Good Enough" Threshold: Downstream sentiment analysis accuracy plateaus early. Once a parser achieves a Labeled Attachment Score (LAS) of roughly 80% (which can be achieved using just 10% of the training treebank data), further improvements to parsing accuracy yield zero benefit to sentiment classification.
  • Why Accuracy Doesn't Matter (Much):
  • Crucial dependencies are easy: The grammatical relations most critical to sentiment—such as adjectival modifiers (amod) and negations (neg)—tend to be short-distance and highly frequent. Even the least accurate parsers achieve over 92% precision on these specific relations.
  • Sentence Redundancy: Sentences often contain multiple expressions of sentiment. If a parser makes an error in one part of a long sentence, the system can usually still extract the correct overall sentiment from the remaining intact structure.
  • Coarse-grained Sufficiency: The sentiment rules often treat different sub-types of modifiers identically. A parsing error that confuses an adverbial modifier for an adjectival modifier counts against the parser's accuracy score, but has no impact on the final sentiment output.

Key Results

  • Parsing Performance vs. Speed: YaraParser achieved the highest parsing accuracy (91.84% LAS) but required 39.85 seconds to parse the short movie review corpus. MaltParser had the lowest accuracy (88.35% LAS) but completed the task in just 4.02 seconds—nearly 10x faster than YaraParser and 24x faster than TurboParser (97.34 seconds).
  • Sentiment Accuracy: Despite the notable gap in parsing metrics, the final sentiment classification accuracies across all four parsers were statistically indistinguishable. On the Pang and Lee (2004) dataset, accuracies tightly clustered between 72.00% and 72.95%. Chi-squared significance tests confirmed that the choice of parser had no relevant influence on the results ($p \ge 0.49$).

Limitations and Future Work

A key limitation is that this study evaluates a purely rule-based sentiment analysis framework. The impact of parsing accuracy on machine-learning or deep-learning-based downstream pipelines remains an open question. Additionally, while the authors provide compelling hypotheses for why parsing errors don't degrade sentiment scores, a rigorous linguistic quantification of these factors is left for future work.

Moving forward, the authors plan to design even faster parsing algorithms that intentionally sacrifice marginal accuracy for raw speed, and intend to explore whether this speed-over-accuracy trade-off holds true for other high-level NLP tasks like aspect extraction and question answering.

Summary updated

Abstract

Syntactic parsing, the process of obtaining the internal structure of sentences in natural languages, is a crucial task for artificial intelligence applications that need to extract meaning from natural language text or speech. Sentiment analysis is one example of application for which parsing has recently proven useful. In recent years, there have been significant advances in the accuracy of parsing algorithms. In this article, we perform an empirical, task-oriented evaluation to determine how parsing accuracy influences the performance of a state-of-the-art rule-based sentiment analysis system that determines the polarity of sentences from their parse trees. In particular, we evaluate the system using four well-known dependency parsers, including both current models with state-of-the-art accuracy and more innacurate models which, however, require less computational resources. The experiments show that all of the parsers produce similarly good results in the sentiment analysis task, without their accuracy having any relevant influence on the results. Since parsing is currently a task with a relatively high computational cost that varies strongly between algorithms, this suggests that sentiment analysis researchers and users should prioritize speed over accuracy when choosing a parser; and parsing researchers should investigate models that improve speed further, even at some cost to accuracy.
BibTeX
@misc{gomezrodriguez2017how,
  author = {Carlos Gómez-Rodríguez and Iago Alonso-Alonso and David Vilares},
  title = {{How Important is Syntactic Parsing Accuracy? An Empirical Evaluation on Rule-Based Sentiment Analysis}},
  year = {2017},
  eprint = {1706.02141},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1706.02141},
}
Figure