The Gradient
The Gradient: Perspectives on AI
Sasha Rush: Building Better NLP Systems

Sasha Rush: Building Better NLP Systems

On architectures for NLP, efficiency, prompting, open-source, and cultivating research environments.

No transcript...

In episode 113 of The Gradient Podcast, Daniel Bashir speaks to Professor Sasha Rush.

Professor Rush is an Associate Professor at Cornell University and a Researcher at HuggingFace. His research aims to develop natural language processing systems that are safe, fast, and controllable. His group is interested primarily in tasks that involve text generation, and they study data-driven probabilistic methods that combine deep-learning based models with probabilistic controls. He is also interested in open-source NLP and deep learning, and develops projects to make deep learning systems safer, clearer, and easier to use.

Have suggestions for future podcast guests (or other feedback)? Let us know here or reach us at

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSS
Follow The Gradient on Twitter


  • (00:00) Intro

  • (01:47) Professor Rush’s background

  • (03:23) Professor Rush’s reflections on prior work—importance of learning and inference

  • (04:58) How much engineering matters in deep learning, the Rush vs. Frankle Bet

  • (07:12) On encouraging and incubating good research

  • (10:50) Features of good research environments

  • (12:36) 5% bets in Professor Rush’s research: State-Space Models (SSMs) as an alternative to Transformers

  • (15:58) SSMs vs. Transformers

  • (18:53) Probabilistic Context-Free Grammars—are (P)CFGs worth paying attention to?

  • (20:53) Sequence-level knowledge distillation: approximating sequence-level distributions

    • (25:08) Pruning and knowledge distillation — orthogonality of efficiency techniques

    • (26:33) Broader thoughts on efficiency

  • (28:31) Works on prompting

    • (28:58) Prompting and In-Context Learning

    • (30:05) Thoughts on mechanistic interpretability

    • (31:25) Multitask prompted training enables zero-shot task generalization

    • (33:48) How many data points is a prompt worth?

    • (35:13) Directions for controllability in LLMs

    • (39:11) Controllability and safety

  • (41:23) Open-source work, deep learning libraries

    • (42:08) A story about Professor Rush’s post-doc at FAIR

    • (43:51) The impact of PyTorch

    • (46:08) More thoughts on deep learning libraries

    • (48:48) Levels of abstraction, PyTorch as an interface to motivate research

  • (50:23) Empiricism and research commitments

  • (53:32) Outro


1 Comment
The Gradient
The Gradient: Perspectives on AI
Deeply researched, technical interviews with experts thinking about AI and technology.