Introduction
The Belief and Sentiment (BeSt) task, part of the NIST Tactical Information Extraction (TAC) 2016 workshop, requires systems to identify the source of an attitude, the target of that attitude, and the nature of the attitude itself (belief or sentiment). This task is significantly more complex than standard sentiment analysis, as it necessitates a deep understanding of entity relationships and linguistic attribution across diverse genres like discussion forums and newswire. The Cornell-Michigan-Pittsburgh team (CORNPITTMICH) developed a hybrid system that combines rule-based linguistic pruning with machine learning architectures, including Long Short-Term Memory (LSTM) networks and Factorization Machines, to address these challenges in both English and Chinese.
System Architecture
The CORNPITTMICH system employs distinct pipelines for English and Chinese, tailored to the specific linguistic properties of each language.
English Pipeline
The English system follows a two-stage architecture:
- Link Prediction: This stage determines whether a relationship exists between a potential source and a target. The team utilized rule-based pruning to eliminate unlikely pairs based on the distance between entities. For the remaining candidates, Factorization Machines were employed to capture interactions between sparse features, providing a robust mechanism for link identification.
- Attitude Classification: Once a link is established, the system classifies the belief or sentiment. The team experimented with two approaches: Multinomial Logistic Regression for a baseline and an LSTM-based encoder to capture sequential dependencies and nuanced sentiment markers.
Chinese Pipeline
The Chinese system adopts a more modular approach, decoupling sentiment detection from source identification:
- Sentiment Analysis: A hybrid model utilizing a single-layer LSTM with average pooling. This model integrates word vectors, part-of-speech (POS) tags, and specialized sentiment dictionaries to enhance its predictive power.
- Source Identification: This component relies on linguistic heuristics. In forum contexts, the system defaults to the post author. In newswire contexts, it identifies "reporting speech" verbs (e.g., "say," "claim") to locate the entity responsible for the statement.
- Heuristic Filtering: A final layer applies scenario-based thresholds. For instance, in sentences containing only a single entity, the system relaxes its confidence requirements, assuming that entity is the likely source.
Data and Training
The primary training data was provided by the TAC 2016 BeSt task, covering English and Chinese across forum and newswire formats. To improve the Chinese sentiment model, the team supplemented the training set with approximately 4,000 annotated sentences from Weibo. Additionally, the TextGrocery tool was utilized to train linear models for belief filtering in the Chinese pipeline.
Key Insights
The CORNPITTMICH approach highlights several critical strategies for sentiment and belief extraction:
- The Power of Pruning: By using simple linguistic rules to prune the search space, the system reduces noise, allowing the more complex machine learning models to focus on high-probability candidates.
- Hybridity: The integration of LSTMs for sentiment and rule-based logic for structural tasks (like source identification) proves more effective than a purely end-to-end neural approach, especially in data-constrained environments.
- Contextual Thresholding: Adjusting model confidence based on the complexity of the sentence structure (e.g., the number of entities present) allows for more flexible and accurate predictions.
Results and Discussion
The team submitted three variations for each language, optimizing for different performance metrics:
- Submission 1: Balanced F-score.
- Submission 2: High Recall, aimed at capturing the maximum number of links.
- Submission 3: High Precision, ensuring the accuracy of identified links.
The Factorization Machine approach in the English system was particularly noteworthy for its ability to combine real-valued scores with binary rule-based decisions, offering granular control over the link prediction process.
Conclusion
The CORNPITTMICH system demonstrates that a hybrid approach—leveraging the strengths of both deep learning and classical rule-based linguistics—is highly effective for the BeSt task. While the system performed well, future work remains in refining belief classification beyond "certain belief" and improving source identification in cases of indirect or metaphorical attribution.
Summary updated