Has AI found a new Foundation?

Over a hundred researchers at Stanford think so. We are not so sure.

Has AI found a new Foundation?

In August, 32 faculty and 117 research scientists, postdocs, and students at Stanford University, long one of the biggest players in AI, declared that there has been a “sweeping paradigm shift in AI”. They coined a new term, “Foundation Models” to characterize the new paradigm, joined forces in a “Center for Research on Foundation Models”, and published the massive 212-page report “On the Opportunities and Risks of Foundation Models.”

Although the term is new, the general approach is not. You train a big neural network (like the well-known GPT-3) on an enormous amount of data), and then you adapt (“fine-tune”) the model to a bunch of more specific tasks (In the words of the report, "a foundation model ...[thus] serves as [part of] the common basis from which many task-specific models are built via adaptation".). The basic model thus serves as the “foundation” (hence the term) of AIs that carry out more specific tasks. The approach started to gather momentum in 2018, when Google developed the natural language processing model called BERT, and it became even more popular with the introduction last year of OpenAI’s GPT-3.

The broader AI community has had decidedly mixed reactions to the announcement from Stanford and some noted scientists have voiced skepticism or opposition. At the Workshop on Foundation Models, Jitendra Malik, a renowned expert in computer vision at Berkeley, said, “I am going to take a ... strongly critical role, when we talk about them as the foundation of AI ... These models are castles in the air. They have no foundations whatsoever.” Judea Pearl, who won the Turing Award for his seminal work on incorporating probability and causal reasoning in AI, tweeted the caustic question, “What is the scientific principle by which 'Foundation models' can circumvent the theoretical limitations of data-centric methods as we know them…?"