TL;DR
This paper investigates the challenge of incorporating extra-linguistic context, specifically speaker information, into pre-trained language models (like BERT) for character identification in dialogues. While their BERT-based model shows improvements in understanding linguistic context, it struggles to effectively integrate speaker information, leading to worse performance on pronouns that heavily rely on it, highlighting the difficulty of adding such context to existing models.
What Problem Does This Paper Address?
Understanding natural language often requires more than just the words themselves. Humans naturally use "extra-linguistic context" – information outside the literal text – to interpret meaning. For instance, knowing who is speaking in a dialogue is crucial for understanding pronouns like "I" or "you." Current state-of-the-art Natural Language Processing (NLP) models, even powerful pre-trained language models like BERT, primarily focus on the linguistic context (the sequence of words). This paper tackles the problem of how to effectively integrate extra-linguistic context, specifically speaker identity, into these pre-trained models for a task called "character identification" in dialogues. This task involves identifying which character in a TV show dialogue is being referred to by a mention.
Methods
The authors propose a novel architecture that combines a pre-trained BERT model with a specialized "entity library." The BERT model processes the input dialogue text to capture linguistic context. Crucially, the speaker's identity (represented by an embedding) is then concatenated with the BERT output for each token. This combined representation is fed into a multi-layer perceptron (MLP). The MLP's output is then compared to the "entity library," which is a learnable matrix containing embeddings for each character. This comparison, using dot products, helps predict the most likely character.
This approach differs from previous work (like Aina et al., 2018, 2019) which used LSTMs and speaker embeddings but did not leverage pre-trained language models like BERT. The key novelty here is processing the input text with BERT first, then incorporating speaker information, contrasting with earlier methods that processed speaker and token information jointly from the start. They also experimented with a self-attention layer to better integrate speaker information, but this did not yield improvements.
Datasets
The study uses the dataset from the SemEval 2018 task 4, which consists of dialogue scripts from the TV show Friends. This dataset was used for both training and evaluation. The task is to predict the character referred to by each referring expression within the dialogue.
Key Insights and Ideas
- BERT's strength is linguistic context: The BERT-based model significantly improved performance on expressions that rely heavily on linguistic context, such as third-person pronouns and proper nouns. This is attributed to BERT's inherent ability to create rich contextualized word representations.
- Extra-linguistic context is hard for BERT: Despite improvements in linguistic understanding, the model struggled with extra-linguistic context, particularly first- and second-person pronouns. These pronouns directly refer to participants in the dialogue, and the model's performance on them degraded compared to previous LSTM-based models.
- The "space" for extra-linguistic information is missing: The authors hypothesize that pre-trained language models like BERT lack an inherent "space" or mechanism to easily incorporate and contextualize extra-linguistic information. Adding it at the output level, as they did, doesn't allow it to be fully integrated with the linguistic understanding.
- Entity library remains valuable: The "entity library" component, which explicitly encodes character information, continued to be beneficial across all model variations, reinforcing its importance for referential tasks.
- Trade-offs exist: The BERT-based model and the previous LSTM-based model have complementary strengths. BERTEnt excels at linguistic context, while LSTMEnt is better at extra-linguistic context. The overall accuracy of LSTMEnt was higher due to the dataset's distribution, which heavily featured first- and second-person pronouns.
Key Results
The proposed BERT-based model (BERTEnt) did not outperform the previous state-of-the-art LSTM-based model (LSTMEnt) on the overall character identification task. While BERTEnt showed better F1 scores for "all entities" in some comparisons, it was worse on other metrics. Specifically:
- BERTEnt was better than LSTMEnt for third-person pronouns and proper nouns.
- BERTEnt was significantly worse than LSTMEnt for second-person pronouns.
- First-person pronoun performance was good for both, with random embeddings + entity library surprisingly performing well, suggesting the linguistic context was less important than speaker identity.
Limitations and Future Work
The authors acknowledge that integrating extra-linguistic information into pre-trained language models is a significant challenge. They suggest that future work could focus on developing models that can dynamically switch between linguistic and extra-linguistic sensitivity. A promising direction is to extend BERT into a "multi-modal two-stream model, specialized on dialogue," which might provide a better framework for integrating different types of information. They also note that the Friends dataset is relatively small for deep learning and suggest using more task-specific data. Experiments on other dialogue-oriented tasks are also recommended to validate their findings.
Summary updated