Transformers are Graph Neural Networks

Interested in contributing?

Learn more about how you can write and get involved with The Gradient by filling out this short form: Writing + Contributing
Interview about the origins and purpose of The Gradient

Transformers are Graph Neural Networks

While Graph Neural Networks are used in recommendation systems at Pinterest, Alibaba and Twitter, a more subtle success story is the Transformer architecture, which has taken the NLP world by storm.

Through this post, I want to establish a link between Graph Neural Networks (GNNs) and Transformers. I'll talk about the intuitions behind model architectures in the NLP and GNN communities, make connections using equations and figures, and discuss how we can work together to drive future progress.