88 - A Structural Probe for Finding Syntax in Word Representations, with John Hewitt

In this episode, we invite John Hewitt to discuss his take on how
to probe word embeddings for syntactic information. The basic idea
is to project word embeddings to a vector space where the L2
distance between a pair of words in a sentence approximates the
number of hops between them in the dependency tree. The proposed
method shows that ELMo and BERT representations, trained with no
syntactic supervision, embed many of the unlabeled, undirected
dependency attachments between words in the same sentence. Paper:
https://nlp.stanford.edu/pubs/hewitt2019structural.pdf GitHub
repository: https://github.com/john-hewitt/structural-probes Blog
post: https://nlp.stanford.edu/~johnhew/structural-probe.html
Twitter thread:
https://twitter.com/johnhewtt/status/1114252302141886464 John's
homepage: https://nlp.stanford.edu/~johnhew/