We consider responses to recent letters calling for pauses on AI experiments and a memory-efficient optimization method that fine-tunes LLMs without using gradients.
Thanks for The Gradient. Such a valuable resource! I tend to focus on the point of agreement between the doomers and hype beasts. Which is faith that there will be quick, continuous growth to AGI from current models, pretending that big data LLMs are that something that close to close. It seems clear to me there needs to be at least one more breakthrough like the optimization one in -2012. Given so much of the cutting edge is led by big companies (and the CCP!) with big budgets for hardware, IMO that adds a while other set of risks and potential outcomes. Thoughts?
Update #52: The Ironies in Pausing AI and Finetuning LLMs without Backpropagation
Thanks for The Gradient. Such a valuable resource! I tend to focus on the point of agreement between the doomers and hype beasts. Which is faith that there will be quick, continuous growth to AGI from current models, pretending that big data LLMs are that something that close to close. It seems clear to me there needs to be at least one more breakthrough like the optimization one in -2012. Given so much of the cutting edge is led by big companies (and the CCP!) with big budgets for hardware, IMO that adds a while other set of risks and potential outcomes. Thoughts?
This is the Prisoner's Dilemma writ large. You have inspired me to write more about this! Thank you.