Abstract
Negation is a universal but complicated linguistic phenomenon, as a negated statement often carries both an explicit negative focus and implicit positive meanings. Precisely detecting the negative focus in context is critical for understanding these statements, yet capturing relevant contextual information remains an open challenge. We propose an attention-based neural network framework consisting of a Bidirectional Long Short-Term Memory (BiLSTM) network and a Conditional Random Fields (CRF) layer to encode order information and long-range dependencies. We introduce two novel attention mechanisms—word-level and topic-level contextual attention—to leverage information across sentences. Experimental results on the SEM’12 shared task corpus demonstrate that our approach achieves state-of-the-art performance, yielding an absolute improvement of 2.11% in accuracy.
Introduction
In linguistics, identifying the "negative focus"—the specific part of a sentence being negated—is essential for interpreting the intended meaning. For example, in the sentence, “Mutual fund trades don’t take effect until the market close,” the bolded phrase is the focus. Removing it reveals an implicit positive assertion: that mutual fund trades do take effect (just at a different time).
While human speakers use vocal stress and intonation to signal focus, text lacks these cues. Consequently, models must rely on surrounding context. Previous NLP approaches often struggled to capture "inter-sentence" information effectively. This research addresses the gap by modeling how adjacent sentences influence the interpretation of negation through specialized attention mechanisms.
Methodology
The core of the proposed framework is a BiLSTM-CRF architecture, a robust standard for sequence labeling tasks. The BiLSTM captures bidirectional dependencies within the sentence, while the CRF layer ensures the global optimality of the predicted label sequence. To incorporate context, the model utilizes two distinct attention mechanisms:
- Word-level Contextual Attention: This measures the correlation between words in the target sentence and those in the preceding and following sentences, highlighting contrasts or thematic continuations.
- Topic-level Contextual Attention: Using Latent Semantic Indexing (LSI), the model generates topical distributions for words and sentences. It then calculates the similarity between the "topic" of a word in the negated sentence and the broader thematic context of the surrounding text.
Furthermore, the model integrates Semantic Role (SR) labels as input features. Because negative focus frequently aligns with specific semantic roles (such as time, manner, or location), these structural cues provide a significant performance boost.
Experimental Results
The models were evaluated using the SEM’12 shared task corpus, which contains 3,544 instances of negative focus derived from PropBank financial news. The results indicate that the proposed attention-based models significantly outperform baseline neural architectures and previous state-of-the-art methods.
The T-Att BiLSTM-CRF (utilizing topic-level attention) achieved the highest accuracy of 70.51%, marking a 2.11% absolute improvement over the previous best-performing graph-based model (WTGM). Notably, the addition of attention mechanisms provided a substantial jump over the standard BiLSTM-CRF (67.28%).
Discussion and Insights
Several key findings emerged from the experimental analysis:
- The Importance of the "Next Sentence": The following sentence was found to be more informative for identifying negative focus than the preceding one, likely because writers often use subsequent text to clarify the scope of a negation.
- Topic vs. Word Attention: Topic-level attention slightly outperformed word-level attention, suggesting that thematic consistency is a more reliable indicator of focus than literal word repetition. Interestingly, combining both mechanisms led to a slight performance drop, likely due to feature redundancy and noise.
- Error Analysis: Errors primarily stemmed from two sources: "Negative Trigger" errors (24%), where the model incorrectly labeled the negation cue (e.g., "not") as the focus, and annotation inconsistencies (40%), where human labels did not align neatly with semantic roles.
Conclusion
This work demonstrates that contextual attention mechanisms, particularly at the topic level, are highly effective for negative focus detection. By leveraging inter-sentence information and semantic roles within a BiLSTM-CRF framework, the model achieves a new state-of-the-art on the SEM’12 benchmark. Future research may explore more sophisticated ways to combine these attention types or integrate rigid logical constraints to resolve contextual ambiguities.
Summary updated