Scaling TensorFlow at LinkedIn with Jonathan Hung

800 800 The TWIML AI Podcast (formerly This Week in Machine Learning & Artificial Intelligence)

Today we’re joined by Jonathan Hung, Sr. Software Engineer at LinkedIn, who we caught up with at TensorFlow World last week.

Jonathan’s presentation at the event focused on LinkedIn’s efforts scaling Tensorflow. In our conversation, we discuss his work as part of the Hadoop infrastructure team, experimenting on Hadoop with various frameworks, and their motivation for using TensorFlow on their pre-existing Hadoop clusters infrastructure. Jonathan also details TonY, or TensorFlow on Yard, LinkedIn’s framework that natively runs deep learning jobs on Hadoop, its relationship with Pro-ML, LinkedIn’s internal AI Platform, which we’ve discussed on earlier episodes of the podcast. Finally, we discuss how far LinkedIn’s Hadoop infrastructure has come since 2017, and their foray into using Kubernetes for research.

Thanks to our Sponsor!

I’d like to send a huge thanks to LinkedIn for sponsoring today’s show! LinkedIn Engineering solves complex problems at scale to create economic opportunity for every member of the global workforce. AI and ML are integral aspects of almost every product the company builds for its members and customers. LinkedIn’s highly structured dataset gives their data scientists and researchers the ability to conduct applied research to improve member experiences. To learn more about the work of LinkedIn Engineering, please visit engineering.linkedin.com/blog.

About Jonathan

Mentioned in the Interview

  • Scaling TensorFlow at LinkedIn
  • TonY (TensorFlow on Yarn)
  • MapReduce
  • Apache Hadoop YARN
  • Productive Machine Learning at LinkedIn with Bee-Chung Chen
  • Productive Machine Learning (Pro-ML)
  • Check it out

    “More On That Later” by Lee Rosevere licensed under CC By 4.0

    Leave a Reply

    Your email address will not be published.