On sparsity in neural networks, what matters in research, and what AI practitioners and society should be thinking about.
Thanks, Daniel, for this stimulating interview. I appreciate Frankle’s deep commitment to empiricism. This interview offers a lot of practical insight for grad students about project and paper design. Build off the work of others. Be prepared to defend your alteration of original setups and variables. Don’t rely too heavily on theory. Follow the evidence. I would have loved to hear Frankle talk more about his empiricism in the context of question and topic selection. This matter seems to go beyond the scope of pure empiricism and opens up into larger philosophical, political, social, and economic concerns. The call for scholars to be more reflective or critical is very much in order, but the criteria that drives such critical processes is still very much in flux. For example, given the rapid state of change in the field, does it truly make sense to shift all research to one single kind of model when 3 to 5 years from now we don’t really have any idea what the files will look like? That said, I appreciate Frankle’s excitement for these new models, and I share his belief that they will continue to play a large role in the AI of the near future!