When BERT Plays The Lottery, All Tickets Are Winning

Does the Lottery Ticket Hypothesis hold for BERT? Experiments with structured and magnitude pruning reveal some good and some bad news.

Interested in contributing?

Learn more about how you can write and get involved with The Gradient by filling out this short form: Writing + Contributing

Does the Lottery Ticket Hypothesis hold for BERT?

Two years after BERT was unveiled to the world, Transformers are still dominating the leaderboards and have spawned numerous follow-up studies. The first version of our attempt to survey BERTology literature provided an overview of about 40 papers in February 2020.

By June, there were over a hundred. The final TACL camera-ready version has about 150 BERT-related citations, and no illusions of completeness: we ran out of journal-allotted pages in August 2020.