In this episode of our AI Platforms series, we’re joined by Atul Kale, Engineering Manager on the machine learning infrastructure team at Airbnb.
Subscribe: iTunes / Google Play / Spotify / RSS
Atul and I met at the Strata Data conference a while back to discuss Airbnb’s internal machine learning platform, Bighead. In our conversation, Atul outlines the ML lifecycle at Airbnb and how the various components of Bighead support it. We then dig into the major components of Bighead, which include Redspot, their supercharged Jupyter notebook service, Deep Thought, their real-time inference environment, Zipline, their data management platform, and quite a few others. We also take a look at some of Atul’s best practices for scaling machine learning, and discuss a special announcement that Atul and his team made at Strata.
As many of you know, part of my work involves understanding the way large companies are adopting machine learning, deep learning and AI. While it’s still fairly early in the game, we’re at a really interesting time for many companies. With the first wave of ML projects at early adopter enterprises starting to mature, many of them are asking themselves how can they scale up their ML efforts to support more projects and teams.
Part of the answer to successfully scaling ML is supporting data scientists and machine learning engineers with modern processes, tooling and platforms. Now, if you’ve been following me or the podcast for a while, you know that this is one of the topics I really like to geek out on.
Well, I’m excited to announce that we’ll be exploring this topic in depth here on the podcast over the next several weeks. You’ll hear from folks building and supporting ML platforms at a host of different companies. We’ll be digging deep into the technologies they’re deploying to accelerate data science and ML development in their companies, the challenges they’re facing, what they’re excited about, and more.
In addition, as part of this effort, I’m publishing a series of eBooks on this topic. The first of them takes a bottoms-up look at AI platforms and is focused on the open source Kubernetes platform which is used to deliver scalable ML and infrastructure at OpenAI, Booking.com, Matroid and many more companies. It’ll be available soon on the TWiML web site, and will be followed shortly thereafter by the second book in the series which looks at scaling data science and ML engineering from the top down, exploring the internal platforms companies Facebook, Uber, and Google have built, the process disciplines that they embody, and what enterprises can learn from them.
If this is a topic you’re interested in, I’d encourage you to visit twimlai.com/aiplatforms and sign up to be notified as soon as these books are published.
Mentioned in the Interview
- Machine Learning & Artificial Intelligence at Airbnb
- Follow Airbnb’s tech teams @airbnbeng, @airbnbdata to get the latest news on upcoming Tech Talks and other hosted events!
- Interested in joining the Airbnb Team? Apply here!
- TWiML Presents: AI Platforms 2018
- TWiML Presents: Series page
- TWiML Events Page
- TWiML Meetup
- TWiML Newsletter
“More On That Later” by Lee Rosevere licensed under CC By 4.0