Polarity Classification with Multitask Learning Syntax Augmentation

Muhammad Imran, Carlos Gómez-RodríguezarXiv 2026 · arXiv

TL;DR

This paper demonstrates how to "bake" syntactic knowledge into sentiment analysis models using multi-task learning (MTL), eliminating the need for slow, external parsers during inference. Their proposed Contrastive-MTL approach uses token-level contrastive losses to help the model internalize linguistic structures like negation and intensification, outperforming traditional rule-based pipelines while running up to 9x faster.

What Problem Does This Paper Address?

Sentiment Analysis (SA) isn't just about spotting "good" or "bad" words. Linguistic phenomena like negation ("not happy") and intensification ("extremely happy") can flip or amplify sentiment, and understanding these requires knowing the syntactic structure of a sentence—which words modify which.

While researchers have successfully integrated syntactic parsers into SA pipelines to handle these complexities, there is a major catch: computational overhead. Conventional dependency parsers are slow, creating a bottleneck for real-time or large-scale applications. The authors aim to bridge this gap: how can we give a neural model the "linguistic intelligence" of a parser without the heavy performance tax at runtime?

Methods

The authors propose two Multi-Task Learning (MTL) strategies that use syntactic annotations during training but discard the extra "machinery" during inference. Both use XLM-RoBERTa-large as a shared backbone.

  1. Standard-MTL: This is an explicit approach. The model has two heads: one for sentiment classification and one for dependency parsing (treated as a sequence labeling task). It learns to predict both simultaneously.
  2. Contrastive-MTL: This is the more novel, implicit approach. It uses two key mechanisms:
  3. Syntax-Guided Attention Pooling: During training, the model uses gold dependency labels to bias its attention mechanism, learning which syntactic roles (like adjectives or intensifiers) are most important for sentiment.
  4. Contrastive Auxiliary Objectives: The model uses two token-level contrastive losses. One pulls together representations of tokens that share the same dependency relation (e.g., all subjects), and the other pulls together tokens from sentences with the same sentiment label.

Crucially, at inference time, the parsing heads and syntactic biases are removed. The model relies entirely on the "internalized" syntactic structure learned by the encoder during training.

Figure 1: Overall architecture of the proposed Multi-Task Learning Framework: Joint Dependency Parsing (Standard-MTL) and Contrastive Syntax-Guided Attention (Contrastive-MTL)

Datasets

The researchers evaluated their models across 11 datasets spanning 7 languages:

  • MARC (Multilingual Amazon Reviews Corpus): English, Spanish, French, and German.
  • SemEval-2022 Task 10: Seven corpora covering hotel, news, and multi-domain reviews in English, Spanish, Norwegian, Basque, and Catalan.

This diversity allowed them to test the robustness of their approach across different linguistic structures and resource levels.

Key Insights and Ideas

  • Implicit is Better than Explicit: Contrastive-MTL proved more robust than Standard-MTL. While Standard-MTL sometimes struggled depending on the dataset or the way syntax was encoded, the contrastive approach consistently improved performance.
  • Internalizing the "Rules": Through a mechanism analysis, the authors found that the Contrastive-MTL encoder actually learns to linearly separate dependency relations. Even though it isn't asked to predict them at runtime, its internal representations are organized by syntactic role.
  • Emergent Attention: At inference, the model "naturally" attends to tokens with specific syntactic roles—like adverbial modifiers (advmod) and adjectives (amod)—that carry the most sentiment. It essentially learns to mimic the hand-crafted rules used in older, slower systems.
  • The "Shortcut" Warning: The authors found that using syntax-biased attention without the contrastive loss actually degraded the encoder's syntactic knowledge. The model would take a "shortcut," relying on the provided labels during training rather than learning the structure itself. The contrastive loss is essential to force the encoder to learn.

Key Results

  • Superior Accuracy: Using the "relative" encoding scheme, Contrastive-MTL outperformed a syntax-naive baseline on 10 out of 11 datasets.
  • Beating the Pipeline: It surpassed a state-of-the-art rule-based pipeline (SELSP) on all six datasets where a comparison was possible.
  • Massive Speedup: Because the model discards the parser at inference, it is 1.5x to 9.2x faster than pipeline approaches that must parse every sentence before classifying it.
  • Language Agnostic: The benefits were consistent across all seven languages tested, suggesting the method effectively captures universal syntactic patterns relevant to sentiment.

Limitations and Future Work

  • Reliance on "Silver" Labels: The training process requires syntactic labels generated by an off-the-shelf parser. If that parser has systematic biases or errors, they might be baked into the MTL model.
  • Coarse-Grained Focus: This study focused on sentence- and document-level polarity. The authors suggest that extending this to fine-grained or aspect-based sentiment analysis (where you need to link specific sentiments to specific targets) is a logical and promising next step.
  • Alternative Syntactic Features: Future work could explore using constituency parses or even simpler features like negation scope markers within the same contrastive framework to find the optimal balance between linguistic richness and training efficiency.

Summary updated

Abstract

Sentiment Analysis (SA) is a fundamental NLP task concerned with the identification and categorization of subjective information in text. While lexicon-based methods offer transparency, they often struggle with linguistic phenomena such as negation and intensification. Syntactic parsing has been integrated into SA pipelines to address this, but conventional parsers introduce significant computational overhead. Recent work shows that reformulating dependency parsing as a Sequence Labeling (SEL) task can improve efficiency. We propose and compare two strategies for integrating syntactic knowledge into a neural sentiment analysis model under unified Multi-Task Learning (MTL). The first, Standard-MTL, jointly learns dependency parsing (via sequence labeling) and sentiment polarity classification using a shared encoder with two task-specific heads. The second, Contrastive-MTL, provides implicit syntactic guidance through a contrastive auxiliary objective and a syntax-biased attention pooling mechanism, without requiring explicit label prediction. Both rely on syntactic annotations during training and eliminate need for hand-crafted compositional rules or external parsing at inference. We evaluate eleven datasets spanning seven languages---the MARC corpus (English, Spanish, French, German) and seven datasets from SemEval-2022 Task 10 (English, Spanish, Norwegian, Basque, Catalan)---using four established SEL encodings. With the relative encoding, Contrastive-MTL outperforms a syntax-naive baseline on 10 of 11 datasets and surpasses the external rule-based pipeline (SELSP) that parses at inference on all six datasets where the latter applies, while being 1.5--9$\times$ faster at evaluation. Standard-MTL is more dataset- and encoding-dependent. These results show implicit syntactic supervision during training can match or exceed explicit parsing at inference without computational overhead.
BibTeX
@misc{imran2026polarity,
  author = {Muhammad Imran and Carlos Gómez-Rodríguez},
  title = {{Polarity Classification with Multitask Learning Syntax Augmentation}},
  year = {2026},
  eprint = {2611.08323},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08323},
}
Figure