Inducing Grammar from Long Short-Term Memory Networks by Shapley Decomposition

Yuhui Zhang, Allen NieACL 2020 · OpenReview

Abstract

The principle of compositionality has deep roots in linguistics: the meaning of an expression is determined by its structure and the meanings of its constituents. However, modern neural network models such as long short-term memory network process expressions in a linear fashion and do not seem to incorporate more complex compositional patterns. In this work, we show that we can explicitly induce grammar by tracing the computational process of a long short-term memory network. We show: (i) the multiplicative nature of long short-term memory network allows complex interaction beyond sequential linear combination; (ii) we can generate compositional trees from the network without external linguistic knowledge; (iii) we evaluate the syntactic difference between the generated trees, randomly generated trees and gold reference trees produced by constituency parsers; (iv) we evaluate whether the generated trees contain the rich semantic information.
BibTeX
@inproceedings{zhang2020inducing,
  author = {Yuhui Zhang and Allen Nie},
  title = {{Inducing Grammar from Long Short-Term Memory Networks by Shapley Decomposition}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=jln7w5YWtP1},
}

Loading related content...

Figure