Finding One Missing Puzzle of Contextual Word Embedding: Representing Contexts as Manifold
Lists
Only you see this
Abstract
The current understanding of contextual word embedding interprets the representation by associating each token to a vector that is dynamically modulated by the context. However, this “token-centric” understanding does not explain how a model represents context itself, leading to a lack of characterization from such a perspective. In this work, to establish a rigorous definition of “context representation”, we formalize this intuition using a category theory framework, which indicates the necessity of including the information from both tokens and how transitions happen among different tokens in a given context. As a practical instantiation of our theoretical understanding, we also show how to leverage a manifold learning method to characterize how a representation model (i.e., BERT) encodes different contexts and how a representation of context changes when going through different components such as attention and FFN. We hope this novel theoretic perspective sheds light on the further improvements in Transformer-based language representation models.
BibTeX
@inproceedings{hu2022finding,
author = {Hailin Hu and Rong Yao and Cheng LI},
title = {{Finding One Missing Puzzle of Contextual Word Embedding: Representing Contexts as Manifold}},
booktitle = {International Conference on Learning Representations},
year = {2022},
url = {https://openreview.net/forum?id=m7zsaLt1Sab},
}Loading related content...