TL;DR
This paper introduces JESSI, a novel system for suggestion mining that combines BERT with a non-BERT encoder to overcome BERT's instability on out-of-domain data. JESSI achieved second place in the SemEval-2019 Task 9, demonstrating the effectiveness of this hybrid approach for robust suggestion detection.
What Problem Does This Paper Address?
The paper tackles the challenge of "suggestion mining," a subfield of opinion mining focused on identifying direct requests for product improvements (e.g., "you should add a dark mode"). While opinion mining is well-studied, suggestion mining has received less attention. The SemEval-2019 Task 9 specifically highlighted four key challenges: sparse occurrences of suggestions, figurative language, different domains (electronics vs. hotels), and complex sentence structures. A major hurdle identified by the authors is the observed instability of powerful pre-trained models like BERT when applied to data from domains different from those they were primarily trained on, even with domain adaptation techniques.
Methods
JESSI is built upon a "joint encoder" architecture, combining two main components:
- BERT-based Encoder: Instead of using BERT's [CLS] token for classification, JESSI utilizes BERT as a word encoder, fine-tuning it and then using the word embeddings from its layers. An RNN-based sentence encoder (BiSRU) is then applied on top of these BERT word encodings to produce a sentence representation.
- CNN-based Encoder: A separate CNN encoder is trained from scratch. It takes concatenated GloVe and CoVe word embeddings as input and uses a convolution operation with multiple filter sizes. An attention mechanism is then employed to create a sentence representation from the CNN's output.
These two sentence representations are then concatenated and fed into a simple MLP classifier. For the out-of-domain task (Subtask B), a domain adversarial training module is incorporated. This module trains a domain classifier to distinguish between the source and target domains, and its gradients are reversed during backpropagation to prevent the main classifier from learning domain-specific features, thus encouraging domain invariance.
Datasets
The primary dataset used is from the SemEval-2019 Task 9: Suggestion Mining from Online Reviews and Forums. This dataset includes:
- Training Data: From the electronics domain.
- Trial Data: From both electronics and hotels domains.
- Test Data: From both electronics and hotels domains.
The authors emphasize that they did not use any additional external data, relying solely on the provided datasets.
Key Insights and Ideas
- BERT's Out-of-Domain Instability: A crucial finding is that while BERT excels on in-domain data, it becomes significantly unstable when applied to out-of-domain samples, even with domain adversarial training. This highlights a critical limitation of relying solely on large pre-trained models for cross-domain tasks.
- Hybrid Encoders for Robustness: Combining BERT with a non-BERT encoder (like a CNN) significantly mitigates BERT's instability. This suggests that leveraging diverse feature extraction methods provides a more robust representation, especially when domain shifts occur.
- Domain Adversarial Training is Crucial: For out-of-domain tasks, the domain adversarial training module is vital for achieving better performance and stability, preventing the model from overfitting to domain-specific cues.
- RNN vs. CNN as BERT's Sentence Encoder: The choice of sentence encoder appended to BERT matters. For the in-domain task (Subtask A), a CNN encoder worked better, while for the out-of-domain task (Subtask B), an RNN-based encoder (BiSRU) proved more effective.
Key Results
- Competitive Performance: JESSI achieved second place in the SemEval-2019 Task 9, with an F1-score of 77.78% on Subtask A (in-domain) and 79.59% on Subtask B (out-of-domain).
- Stability Improvement: Ablation studies showed that combining BERT with a CNN encoder improved performance and stability. Appending an RNN encoder to BERT further enhanced both performance and stability for out-of-domain samples.
- Out-of-Domain Robustness: The paper provides evidence (Table 3) that vanilla BERT exhibits highly variable F-scores (as low as 0.00) across runs for out-of-domain data, whereas JESSI-B shows much more stable and higher performance.
Limitations and Future Work
- BERT Size Limitation: For Subtask B, the authors were limited to using BERT-base due to memory constraints and expect performance to improve with BERT-large.
- Domain Adversarial Training Nuances: While effective, the paper notes that domain adversarial training is crucial for out-of-domain performance, implying that further tuning or alternative domain adaptation techniques could be explored.
- Future Work: The authors implicitly suggest that exploring different combinations of encoders and further refining domain adaptation strategies would be beneficial for improving suggestion mining systems, especially in cross-domain scenarios.
Summary updated