ThisIsCompetition at SemEval-2019 Task 9: BERT is unstable for out-of-domain samples

Cheon-Eum Park, Juae Kim, Hyeon-gu Lee, Reinald Kim Amplayo, Harksoo Kim, Jungyun Seo, Changki Lee

TL;DR

This paper introduces JESSI, a novel system for suggestion mining that combines BERT with a non-BERT encoder to overcome BERT's instability on out-of-domain data. JESSI achieved second place in the SemEval-2019 Task 9, demonstrating the effectiveness of this hybrid approach for robust suggestion detection.

What Problem Does This Paper Address?

The paper tackles the challenge of "suggestion mining," a subfield of opinion mining focused on identifying direct requests for product improvements (e.g., "you should add a dark mode"). While opinion mining is well-studied, suggestion mining has received less attention. The SemEval-2019 Task 9 specifically highlighted four key challenges: sparse occurrences of suggestions, figurative language, different domains (electronics vs. hotels), and complex sentence structures. A major hurdle identified by the authors is the observed instability of powerful pre-trained models like BERT when applied to data from domains different from those they were primarily trained on, even with domain adaptation techniques.

Methods

JESSI is built upon a "joint encoder" architecture, combining two main components:

  • BERT-based Encoder: Instead of using BERT's [CLS] token for classification, JESSI utilizes BERT as a word encoder, fine-tuning it and then using the word embeddings from its layers. An RNN-based sentence encoder (BiSRU) is then applied on top of these BERT word encodings to produce a sentence representation.
  • CNN-based Encoder: A separate CNN encoder is trained from scratch. It takes concatenated GloVe and CoVe word embeddings as input and uses a convolution operation with multiple filter sizes. An attention mechanism is then employed to create a sentence representation from the CNN's output.

These two sentence representations are then concatenated and fed into a simple MLP classifier. For the out-of-domain task (Subtask B), a domain adversarial training module is incorporated. This module trains a domain classifier to distinguish between the source and target domains, and its gradients are reversed during backpropagation to prevent the main classifier from learning domain-specific features, thus encouraging domain invariance.

Datasets

The primary dataset used is from the SemEval-2019 Task 9: Suggestion Mining from Online Reviews and Forums. This dataset includes:

  • Training Data: From the electronics domain.
  • Trial Data: From both electronics and hotels domains.
  • Test Data: From both electronics and hotels domains.

The authors emphasize that they did not use any additional external data, relying solely on the provided datasets.

Key Insights and Ideas

  • BERT's Out-of-Domain Instability: A crucial finding is that while BERT excels on in-domain data, it becomes significantly unstable when applied to out-of-domain samples, even with domain adversarial training. This highlights a critical limitation of relying solely on large pre-trained models for cross-domain tasks.
  • Hybrid Encoders for Robustness: Combining BERT with a non-BERT encoder (like a CNN) significantly mitigates BERT's instability. This suggests that leveraging diverse feature extraction methods provides a more robust representation, especially when domain shifts occur.
  • Domain Adversarial Training is Crucial: For out-of-domain tasks, the domain adversarial training module is vital for achieving better performance and stability, preventing the model from overfitting to domain-specific cues.
  • RNN vs. CNN as BERT's Sentence Encoder: The choice of sentence encoder appended to BERT matters. For the in-domain task (Subtask A), a CNN encoder worked better, while for the out-of-domain task (Subtask B), an RNN-based encoder (BiSRU) proved more effective.

Key Results

  • Competitive Performance: JESSI achieved second place in the SemEval-2019 Task 9, with an F1-score of 77.78% on Subtask A (in-domain) and 79.59% on Subtask B (out-of-domain).
  • Stability Improvement: Ablation studies showed that combining BERT with a CNN encoder improved performance and stability. Appending an RNN encoder to BERT further enhanced both performance and stability for out-of-domain samples.
  • Out-of-Domain Robustness: The paper provides evidence (Table 3) that vanilla BERT exhibits highly variable F-scores (as low as 0.00) across runs for out-of-domain data, whereas JESSI-B shows much more stable and higher performance.

Limitations and Future Work

  • BERT Size Limitation: For Subtask B, the authors were limited to using BERT-base due to memory constraints and expect performance to improve with BERT-large.
  • Domain Adversarial Training Nuances: While effective, the paper notes that domain adversarial training is crucial for out-of-domain performance, implying that further tuning or alternative domain adaptation techniques could be explored.
  • Future Work: The authors implicitly suggest that exploring different combinations of encoders and further refining domain adaptation strategies would be beneficial for improving suggestion mining systems, especially in cross-domain scenarios.

Summary updated

Abstract

This paper describes our system, Joint Encoders for Stable Suggestion Inference (JESSI), for the SemEval 2019 Task 9: Suggestion Mining from Online Reviews and Forums. JESSI is a combination of two sentence encoders: (a) one using multiple pre-trained word embeddings learned from log-bilinear regression (GloVe) and translation (CoVe) models, and (b) one on top of word encodings from a pre-trained deep bidirectional transformer (BERT). We include a domain adversarial training module when training for out-of-domain samples. Our experiments show that while BERT performs exceptionally well for in-domain samples, several runs of the model show that it is unstable for out-of-domain samples. The problem is mitigated tremendously by (1) combining BERT with a non-BERT encoder, and (2) using an RNN-based classifier on top of BERT. Our final models obtained second place with 77.78% F-Score on Subtask A (i.e. in-domain) and achieved an F-Score of 79.59% on Subtask B (i.e. out-of-domain), even without using any additional external data.
BibTeX
@inproceedings{park2019thisiscompetition,
  author = {Cheon-Eum Park and Juae Kim and Hyeon-gu Lee and Reinald Kim Amplayo and Harksoo Kim and Jungyun Seo and Changki Lee},
  title = {{ThisIsCompetition at SemEval-2019 Task 9: BERT is unstable for out-of-domain samples}},
  booktitle = {SemEval @ NAACL},
  year = {2019},
  url = {https://openreview.net/forum?id=7aplCDqY_ND},
}
Figure