Mixture-of-Experts and Trends in Large-Scale Language Modeling with Irwan Bello

EPISODE 569
WATCH
Play Video
Join our list for notifications and early access to events

About this Episode

Today we’re joined by Irwan Bello, formerly a research scientist at Google Brain, and now on the founding team at a stealth AI startup. We begin our conversation with an exploration of Irwan’s recent paper, Designing Effective Sparse Expert Models, which acts as a design guide for building sparse large language model architectures. We discuss mixture of experts as a technique, the scalability of this method, and it's applicability beyond NLP tasks the data sets this experiment was benchmarked against. We also explore Irwan’s interest in the research areas of alignment and retrieval, talking through interesting lines of work for each area including instruction tuning and direct alignment.

Connect with Irwan
Read More

More from TWIML

Leave a Reply

Your email address will not be published.