Negative Focus Detection via Contextual Attention Mechanism

Longxiang Shen, Bowei Zou, Yu Hong, Guodong Zhou, Qiaoming Zhu, AiTi AwEMNLP 2019 · OpenReview

Abstract

Negation is a universal but complicated linguistic phenomenon, as a negated statement often carries both an explicit negative focus and implicit positive meanings. Precisely detecting the negative focus in context is critical for understanding these statements, yet capturing relevant contextual information remains an open challenge. We propose an attention-based neural network framework consisting of a Bidirectional Long Short-Term Memory (BiLSTM) network and a Conditional Random Fields (CRF) layer to encode order information and long-range dependencies. We introduce two novel attention mechanisms—word-level and topic-level contextual attention—to leverage information across sentences. Experimental results on the SEM’12 shared task corpus demonstrate that our approach achieves state-of-the-art performance, yielding an absolute improvement of 2.11% in accuracy.

Introduction

In linguistics, identifying the "negative focus"—the specific part of a sentence being negated—is essential for interpreting the intended meaning. For example, in the sentence, “Mutual fund trades don’t take effect until the market close,” the bolded phrase is the focus. Removing it reveals an implicit positive assertion: that mutual fund trades do take effect (just at a different time).

While human speakers use vocal stress and intonation to signal focus, text lacks these cues. Consequently, models must rely on surrounding context. Previous NLP approaches often struggled to capture "inter-sentence" information effectively. This research addresses the gap by modeling how adjacent sentences influence the interpretation of negation through specialized attention mechanisms.

Methodology

The core of the proposed framework is a BiLSTM-CRF architecture, a robust standard for sequence labeling tasks. The BiLSTM captures bidirectional dependencies within the sentence, while the CRF layer ensures the global optimality of the predicted label sequence. To incorporate context, the model utilizes two distinct attention mechanisms:

  1. Word-level Contextual Attention: This measures the correlation between words in the target sentence and those in the preceding and following sentences, highlighting contrasts or thematic continuations.
  2. Topic-level Contextual Attention: Using Latent Semantic Indexing (LSI), the model generates topical distributions for words and sentences. It then calculates the similarity between the "topic" of a word in the negated sentence and the broader thematic context of the surrounding text.
The BiLSTM-CRF framework integrated with contextual attention mechanisms.

Furthermore, the model integrates Semantic Role (SR) labels as input features. Because negative focus frequently aligns with specific semantic roles (such as time, manner, or location), these structural cues provide a significant performance boost.

Experimental Results

The models were evaluated using the SEM’12 shared task corpus, which contains 3,544 instances of negative focus derived from PropBank financial news. The results indicate that the proposed attention-based models significantly outperform baseline neural architectures and previous state-of-the-art methods.

Comparison of accuracy between existing methods and the proposed attention-based models on the SEM’12 corpus.

The T-Att BiLSTM-CRF (utilizing topic-level attention) achieved the highest accuracy of 70.51%, marking a 2.11% absolute improvement over the previous best-performing graph-based model (WTGM). Notably, the addition of attention mechanisms provided a substantial jump over the standard BiLSTM-CRF (67.28%).

Discussion and Insights

Several key findings emerged from the experimental analysis:

  • The Importance of the "Next Sentence": The following sentence was found to be more informative for identifying negative focus than the preceding one, likely because writers often use subsequent text to clarify the scope of a negation.
  • Topic vs. Word Attention: Topic-level attention slightly outperformed word-level attention, suggesting that thematic consistency is a more reliable indicator of focus than literal word repetition. Interestingly, combining both mechanisms led to a slight performance drop, likely due to feature redundancy and noise.
  • Error Analysis: Errors primarily stemmed from two sources: "Negative Trigger" errors (24%), where the model incorrectly labeled the negation cue (e.g., "not") as the focus, and annotation inconsistencies (40%), where human labels did not align neatly with semantic roles.

Conclusion

This work demonstrates that contextual attention mechanisms, particularly at the topic level, are highly effective for negative focus detection. By leveraging inter-sentence information and semantic roles within a BiLSTM-CRF framework, the model achieves a new state-of-the-art on the SEM’12 benchmark. Future research may explore more sophisticated ways to combine these attention types or integrate rigid logical constraints to resolve contextual ambiguities.

Summary updated

Abstract

Negation is a universal but complicated linguistic phenomenon, which has received considerable attention from the NLP community over the last decade, since a negated statement often carries both an explicit negative focus and implicit positive meanings. For the sake of understanding a negated statement, it is critical to precisely detect the negative focus in context. However, how to capture contextual information for negative focus detection is still an open challenge. To well address this, we come up with an attention-based neural network to model contextual information. In particular, we introduce a framework which consists of a Bidirectional Long Short-Term Memory (BiLSTM) neural network and a Conditional Random Fields (CRF) layer to effectively encode the order information and the long-range context dependency in a sentence. Moreover, we design two types of attention mechanisms, word-level contextual attention and topic-level contextual attention, to take advantage of contextual information across sentences from both the word perspective and the topic perspective, respectively. Experimental results on the SEM’12 shared task corpus show that our approach achieves the best performance on negative focus detection, yielding an absolute improvement of 2.11% over the state-of-the-art. This demonstrates the great effectiveness of the two types of contextual attention mechanisms.
BibTeX
@inproceedings{shen2019negative,
  author = {Longxiang Shen and Bowei Zou and Yu Hong and Guodong Zhou and Qiaoming Zhu and AiTi Aw},
  title = {{Negative Focus Detection via Contextual Attention Mechanism}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=PXoDcr8moV},
}
Figure