Davidad Dalrymple: Towards Provably Safe AI

Episode 137

I spoke with Davidad Dalrymple about:

• His perspectives on AI risk

• ARIA (the UK’s Advanced Research and Invention Agency) and its Safeguarded AI Programme

Enjoy—and let me know what you think!

---

Davidad is a Programme Director at ARIA. He was most recently a Research Fellow in technical AI safety at Oxford. He co-invented the top-40 cryptocurrency Filecoin, led an international neuroscience collaboration, and was a senior software engineer at Twitter and multiple startups.

---

Find me on Twitter for updates on new episodes, and reach me at editor@thegradient.pub for feedback, ideas, guest suggestions.

Subscribe to The Gradient Podcast:  Apple Podcasts  | Spotify | Pocket Casts | RSS
Follow The Gradient on Twitter

Outline:

• (00:00) Intro

• (00:36) Calibration and optimism about breakthroughs

• (03:35) Calibration and AGI timelines, effects of AGI on humanity

• (07:10) Davidad’s thoughts on the Orthogonality Thesis

• (10:30) Understanding how our current direction relates to AGI and breakthroughs

• (13:33) What Davidad thinks is needed for AGI

  • (17:00) Extracting knowledge

  • (19:01) Cyber-physical systems and modeling frameworks

• (20:00) Continuities between Davidad’s earlier work and ARIA

• (22:56) Path dependence in technology, race dynamics

• (26:40) More on Davidad’s perspective on what might go wrong with AGI

• (28:57) Vulnerable world, interconnectedness of computers and control

• (34:52) Formal verification and world modeling, Open Agency Architecture

  • (35:25) The Semantic Sufficiency Hypothesis

  • (39:31) Challenges for modeling

  • (43:44) The Deontic Sufficiency Hypothesis and mathematical formalization

  • (49:25) Oversimplification and quantitative knowledge

  • (53:42) Collective deliberation in expressing values for AI

• (55:56) ARIA’s Safeguarded AI Programme

• (59:40) Anthropic’s ASL levels

• (1:03:12) Guaranteed Safe AI —

  • (1:03:38) AI risk and (in)accurate world models

  • (1:09:59) Levels of safety specifications for world models and verifiers — steps to achieve high safety

• (1:12:00) Davidad’s portfolio research approach and funding at ARIA

• (1:15:46) Earlier concerns about ARIA — Davidad’s perspective

• (1:19:26) Where to find more information on ARIA and the Safeguarded AI Programme

• (1:20:44) Outro

Links:

• Davidad’s Twitter

• ARIA homepage

  • Safeguarded AI Programme

• Papers

  • Guaranteed Safe AI

  • Davidad’s Open Agency Architecture for Safe Transformative AI

  • Dioptics: a Common Generalization of Open Games and Gradient-Based Learners (2019)

  • Asynchronous Logic Automata (2008)