We could not locate the page you were looking for.

Below we have generated a list of search results based on the page you were trying to reach.

404 Error
Akshita is a Senior Research Engineer on the AllenNLP team, involved in R&D for natural language processing (NLP). Most recently, Akshita has been working on the OLMo project, where she has contributed to pretraining dataset construction, model training and inference, and evaluation tools and benchmark. She has also worked on open-source libraries such as allennlp, ai2-tango, etc. Akshita graduated with a Master’s degree in Computer Science from the University of Massachusetts Amherst in 2020, where she worked with Prof. Mohit Iyyer at the intersection of NLP and digital humanities. Previously, Akshita worked at Cerebellum Capital (Summer 2019), and at InFoCusp (2015-2018), where she worked on building a data science platform. In her spare time, Akshita enjoys reading novels, writing (especially poetry), and dancing.
Armineh Nourbakhsh is an Executive Director at J.P. Morgan AI Research, where she leads the Document AI team. Her career spans 15 years of research in Natural Language Processing and Multimodal Machine Learning, and her work has been deployed in award-winning technologies such as Reuters Tracer and Westlaw Quick Check. In 2020, Armineh was the recipient of the WBC Rising Star Award, celebrating the next generation of women leading technology and innovation in Financial Services.
Ben Zhao is the Neubauer Professor of Computer Science at the University of Chicago. He completed his PhD from Berkeley (2004) and his BS from Yale (1997). He is an ACM distinguished scientist, and recipient of the NSF CAREER award, MIT Technology Review’s TR-35 Award (Young Innovators Under 35), ComputerWorld Magazine’s Top 40 Tech Innovators award, Google Faculty award, and IEEE ITC Early Career Award. His work has been covered by media outlets such as Scientific American, New York Times, Boston Globe, LA Times, Wall Street Journal, MIT Tech Review, and Slashdot. He has published more than 160 publications in areas of security and privacy, machine learning, networked systems, Internet measurements and HCI. He served as Program (co)chair for the World Wide Web Conference (WWW 2016) and the ACM Internet Measurement Conference (IMC 2018), and General Co-Chair for ACM HotNets 2020. Over the years, Ben followed his own interests in pursuing research problems that he finds intellectually interesting and meaningful. That’s led him to work on a sequence of areas from P2P networks, online social networks, SDR/open spectrum systems, graph mining and modeling, user behavior analysis, to adversarial machine learning. Since 2016, he mostly worked on security and privacy problems in machine learning and mobile systems.
Mike Miller is a Director of Product Management at AWS, leading multiple initiatives across the AWS organization, focusing on Generative AI. Mike previously led the AWS ML Thought Leadership team, who brought AWS DeepLens, AWS DeepRacer, and AWS DeepComposer to developers worldwide, helping aspiring machine learning developers get hands-on with the latest machine learning technologies. In 2020 Mike’s team launched AWS Panorama, a service that allows organizations to bring computer vision (CV) to on-premises cameras to make predictions locally with high accuracy and low latency. Mike has been with Amazon for over ten years, previously leading product management for Fire TV at Lab126 before joining Amazon Web Services.
I received my PhD from UC Berkeley in 2021, under the mentorship of Bruno Olshausen in the Redwood Center for Theoretical Neuroscience. I’m currently appointed as a Postdoctoral Scholar in the Department of Electrical and Computer Engineering at the University of California, Santa Barbara, where I work with Nina Miolane on applications of group theory and differential geometry to deep learning and neuroscience. I hold affiliate appointments in the Department of Mathematics and Department of Biomedical Engineering at UBC, with collaborators Khanh Dao Duc and Manu Madhav. My research has been supported by the PIMS-Simons Postdoctoral Fellowship, the NSF GRFP, and the Beinecke Scholarship. I’ve worked as a researcher in Intel’s Neuromorphic Computing Lab and AI Division, and have provided teaching support for Berkeley’s graduate-level Machine Learning and Neural Computation courses.  Community organization is an important aspect of my work. I co-organize the NeurIPS Workshop on Symmetry and Geometry in Neural Representations (NeurReps) and the ICML Workshop on Topology, Algebra, and Geometry in Machine Learning (TAG-ML). In 2024, we are taking NeurReps to the Canadian wilderness for a 5-day workshop at the Banff International Research Station.
Dr. Ben Goertzel is a cross-disciplinary scientist, entrepreneur and author.  Born in Brazil to American parents, in 2020 after a long stretch living in Hong Kong he relocated his primary base of operations to the rural Seattle area.   He leads the SingularityNET Foundation, the OpenCog Foundation, and the AGI Society which runs the annual Artificial General Intelligence conference. Dr. Goertzel also chairs the futurist nonprofit Humanity+, and serves as Chief Scientist of AI firms Singularity Studio, Rejuve, SingularityDAO and Xccelerando Media, all parts of the SingularityNET ecosystem. As Chief Scientist of robotics firm Hanson Robotics, he led the software team behind the Sophia robot; as Chief AI Scientist of Awakening Health he leads the team crafting the mind behind Sophia’s little sister Grace. Dr. Goertzel’s research work encompasses multiple areas including artificial general intelligence, natural language processing, cognitive science, machine learning, computational finance, bioinformatics, virtual worlds, gaming, parapsychology, theoretical physics and more.  He has published 25+ scientific books, ~150 technical papers, and numerous journalistic articles, and given talks at a vast number of events of all sorts around the globe.
Today we’re joined by Tom Goldstein, an associate professor at the University of Maryland. Tom’s research sits at the intersection of ML and optimization and has previously been featured in the New Yorker for his work on invisibility cloaks, clothing that can evade object detection. In our conversation, we focus on his more recent research on watermarking LLM output. We explore the motivations behind adding these watermarks, how they work, and different ways a watermark could be deployed, as well as political and economic incentive structures around the adoption of watermarking and future directions for that line of work. We also discuss Tom’s research into data leakage, particularly in stable diffusion models, work that is analogous to recent guest Nicholas Carlini’s research into LLM data extraction. 
Since 2002 I have been a professor in the Computer and Information Science Department at the University of Pennsylvania, where I hold the National Center Chair. I have secondary appointments in the department of Economics, and in the departments of Statistics and Data Science and Operations, Information and Decisions (OID) in the Wharton School. I am the Founding Director of the Warren Center for Network and Data Sciences, where my Co-Director is Rakesh Vohra. I am the faculty founder and former director of Penn Engineering's Networked and Social Systems Engineering (NETS) Program, whose current directors are Andreas Haeberlen and Aaron Roth. I am a faculty affiliate in Penn's Applied Math and Computational Science graduate program. Until July 2006 I was the co-director of Penn's interdisciplinary Institute for Research in Cognitive Science. Since June 2020, I am an Amazon Scholar, focusing on fairness and privacy in machine learning and related topics within Amazon Web Services. I have worked extensively in quantitative and algorithmic trading on Wall Street (including at Lehman Brothers, Bank of America, SAC Capital and Morgan Stanley; see further details below). I often serve as an advisor to technology companies and venture capital firms, and sometimes invest in early-stage technology startups. I occasionally serve as an expert witness or consultant on technology-related legal and regulatory cases. I am an elected Member/Fellow of the National Academy of Sciences, the American Academy of Arts and Sciences, the Association for Computing Machinery, the Association for the Advancement of Artificial Intelligence, and the Society for the Advancement of Economic Theory.
With the widespread news of tech company layoffs, it looks like the industry is entering the inevitable downward phase in the economic cycle. This is the second-and-a-half time I’ve experienced this in my career. The first was the dot-com bust in the early 2000s. The second was the global financial crisis of 2007-2008. The "half" was the blip at the beginning of the pandemic, which while it impacted many, ended up being more of a realignment for those of us fortunate enough to work in technology, with many more opportunities created than destroyed. While I remain optimistic for our field, this will be a painful period for many, both those directly affected by layoffs as well as those for whom uncertainty and the specter of a broader recession loom. And all before we’ve really had an opportunity to mentally recover from 2020 and 2021. If you have been impacted by layoffs, please know that there are a lot of people who want to help, me included. One of the best ways I think I can help is by connecting those who are hiring with those who are looking, and those looking to help in other ways with those who might benefit. Though the current macroeconomic climate is one that will leave many organizations cautious about hiring, savvy organizations with solid business fundamentals will recognize it as a hiring opportunity. If you are in this position and you are hiring data and AI talent, including AI-savvy marketing and business roles, please reach out to me and I'll do my best to connect you with quality candidates. And if you are looking for a job, please reach out as well and I'll do my best to connect you with companies that I learn are hiring. If you’re neither hiring nor looking but would like to help in other ways, consider mentoring or coaching someone, or leading a study group in the TWIML Community for those who want to learn new skills. Let me know if any of those are of interest. If you have other ideas for how I might be able to help out, please let me know. Thank you for considering how you can help those in our community who have been impacted by layoffs. I look forward to hearing from you soon.
Dr. Oren Etzioni was Chief Executive Officer at AI2 from its inception until Sept. 30th, 2022. He now serves as an advisor & board member for AI2, and a technical director of the AI2 Incubator. He is Professor Emeritus, University of Washington as of October 2020 and a Venture Partner at the Madrona Venture Group since 2000. His awards include AAAI Fellow and Seattle’s Geek of the Year. He has founded several companies including Farecast (acquired by Microsoft). He has written over 200 technical papers, as well as commentary on AI for The New York Times, Wired, and Nature. He helped to pioneer meta-search, online comparison shopping, machine reading, and Open Information Extraction.
Benedikt Schifferer is DL engineer at NVIDIA. Before joining NVIDIA, he worked as a data scientist, developed the in-house recommendation engine at Home24, an ecommerce store for furniture. Afterwards, he worked as a data science consultant at McKinsey & Company, presenting the implemented machine learning solutions to technical and non-technical audiences. He got his Msc degree from Columbia University, New York, where Benedikt was a teaching assistant for two courses, Applied Deep Learning and Deep Learning. Benedikt was part of the winning team of the RecSys2020, RecSys2021, RecSys2022 and WSDM2021 Booking.com challenge.
Today we kick off our annual coverage of the CVPR conference joined by Fatih Porikli, Senior Director of Engineering at Qualcomm AI Research. In our conversation with Fatih, we explore a trio of CVPR-accepted papers, as well as a pair of upcoming workshops at the event. The first paper, Panoptic, Instance and Semantic Relations: A Relational Context Encoder to Enhance Panoptic Segmentation, presents a novel framework to integrate semantic and instance contexts for panoptic segmentation. Next up, we discuss Imposing Consistency for Optical Flow Estimation, a paper that introduces novel and effective consistency strategies for optical flow estimation. The final paper we discuss is IRISformer: Dense Vision Transformers for Single-Image Inverse Rendering in Indoor Scenes, which proposes a transformer architecture to simultaneously estimate depths, normals, spatially-varying albedo, roughness, and lighting from a single image of an indoor scene. For each paper, we explore the motivations and challenges and get concrete examples to demonstrate each problem and solution presented.
Been is interested in helping humans to communicate with complex machine learning models: not only by building tools (and tools to criticize them), but also studying their nature, compared to humans. Quanta magazine (written by John Pavlus) is a great description of what I do and why. She believes the language that humans and machines communicate must be human-centered--higher-level, human-friendly concepts--so that it can make sense to everyone , regardless of how much they know about ML. She gave keynote at ICLR 2022 (blog post, video TBD), ECML 2020 and at the G20 meeting in Argentina in 2018. One of my work TCAV received UNESCO Netexplo award, was featured at Google I/O 19' and in Brian Christian's book on The Alignment Problem.
Genevera Allen is an Associate Professor of Electrical and Computer Engineering, Statistics, and Computer Science at Rice University and an investigator at the Jan and Dan Duncan Neurological Research Institute at Texas Children's Hospital and Baylor College of Medicine. She is also the Founder and Faculty Director of the Center for Transforming Data to Knowledge, informally called the Rice D2K Lab. Dr. Allen's research focuses on developing statistical machine learning tools to help scientists make reproducible data-driven discoveries. Her work lies in the areas of interpretable machine learning, optimization, data integration, modern multivariate analysis, and graphical models with applications in neuroscience and bioinformatics. Selected Awards and Honors: - Elected Member, International Statistics Institute, 2021. - Charles W. Duncan Achievement Award for Outstanding Faculty, Rice University, 2021. - Curriculum Innovation Award, George R. Brown School of Engineering, Rice University, 2020. - Research and Teaching Excellence Award, George R. Brown School of Engineering, Rice University, 2017. - NSF Career Award, 2016. - Forbes 30 under 30: Science & Healthcare, 2014. Publication List: https://scholar.google.com/citations?user=gIUd12QAAAAJ&hl=en
I am a Research Scientist at Google. Previously, I completed my Ph.D. at Boston University, advised by Professor and Dean of the College of Arts and Sciences Stan Sclaroff. My primary research focus is computer vision and machine learning. I interned at Amazon working with Javier Romero, Timo Bolkart, Ming C. Lin, and Raja Bala during the Summer of 2021. I interned at Apple AI Research during the 2019 and 2020 Summers where I worked with Dr. Barry-John Theobald and Dr. Nicholas Apostoloff. In 2018 I was a Spring/Summer intern at the NEC-Labs Media Analytics Department, where I worked with Prof. Manmohan Chandraker and Dr. Samuel Schulter. I graduated from Georgia Tech in Fall 2017 with an M.Sc. in Computer Science specializing in Machine Learning, advised by Prof. James Rehg at the Center for Behavioral Imaging. Recently, our work DreamBooth has been selected for a Student Best Paper Honorable Mention Award at CVPR 2023 (0.25% award rate) I have been selected as a Twitch Research Fellowship finalist for the year 2020 and as a second-round interviewee for the Open Phil AI Fellowship. I also appeared on the popular Machine Learning and AI podcast TWIML AI talking about my recent work on defending against deepfakes. While on a 5-year valedictorian scholarship, I obtained my B.Sc. and M.Sc. from Ecole Polytechnique in Paris, France. Additionally, I worked as an intern at MIT CSAIL with Dr. Kalyan Veeramachaneni and Dr. Lalana Kagal.
Naila Murray obtained a BSE in electrical engineering from Princeton University in 2007. In 2012, she received her Ph.D. from the Universitat Autonoma de Barcelona, in affiliation with the Computer Vision Center. She joined Xerox Research Centre Europe in 2013 as a research scientist in the computer vision team, working on topics including fine-grained visual categorization, image retrieval, and visual attention. From 2015 to 2019, she led the computer vision team at Xerox Research Centre Europe and continued to serve in this role after its acquisition and transition to becoming NAVER LABS Europe. In 2019, she became the director of science at NAVER LABS Europe. In 2020, she joined Meta AI’s FAIR team, where she served as a senior research manager. She now leads as the director of AI research at Meta. She has served as area chair for ICLR 2018, ICCV 2019, ICLR 2019, CVPR 2020, ECCV 2020, and CVPR 2022, and program chair for ICLR 2021. Her current research interests include few-shot learning and domain adaptation.
I have joined the Computer Science department of the Swiss Federal Institute of Technology (EPFL) as of January 2020. Prior to EPFL, I spent time at Stanford, UC Berkeley, and UCF where I had the privilege of working with Silvio Savarese, Jitendra Malik, Mubarak Shah, and Leonidas Guibas. My research interests are broadly in computer vision, machine learning, and AI. The goal of my research is going beyond narrow and offline perception methods toward a general multi-task visual perception that operates as part of an active body in the real world. Here are some of my works on this topic: Taskonomy, Cross-Task Consistency, Mid-Level Vision, Gibson. I'm also interested in structured prediction, video understanding, and 3D vision. I'm a fan of Slow Science.
Dr. Torbjörn Nordling obtained both his Ph.D. in Automatic Control (2013) and his M.Sc. in Engineering Physics (2005) from KTH Royal Institute of Technology in Stockholm, Sweden. He joined the Department of Mechanical Engineering at the National Cheng Kung University in Taiwan as an Assistant Professor in September 2015 and the Department of Applied Physics and Electronics at Umeå University in Sweden as Senior Lecturer in January 2020. Previously, he completed a PostDoc at the Dept. of Immunology, Genetics and Pathology at Uppsala University in Sweden. He has been a visiting researcher at Telethon Institute of Genetics and Medicine in Naples, Italy and ERATO Kitano Symbiotic Systems Project at Japan Science and Technology Agency in Tokyo, Japan. He is the founder of Nordron AB, a startup specialised in data analysis, and a co-founder of Jagah Systems AB, an award winning indoor geolocalisation startup. He has co-authored more than 25 peer-reviewed full-length articles published in international journals (SCI IF 1.5-25) or conference proceedings and given more than 60 invited lectures, oral conference, or poster presentations. He has specialised in Data Science, Mathematical modeling, System identification, Machine learning, and Artificial Intelligence with applications in Biology and Medicine.
Sasha is a Postdoctoral Researcher working with Yoshua Bengio and others on a project that uses Artificial Intelligence to visualize the consequences of climate change. I am also a 2020 National Geographic Explorer, and hold an IVADO postdoctoral scholarship.
Dr. Sameer Singh is an Associate Professor of Computer Science at the University of California, Irvine (UCI). He is working primarily on robustness and interpretability of machine learning algorithms, along with models that reason with text and structure for natural language processing. Sameer was a postdoctoral researcher at the University of Washington and received his PhD from the University of Massachusetts, Amherst, during which he interned at Microsoft Research, Google Research, and Yahoo! Labs. He has received the NSF CAREER award, selected as a DARPA Riser, UCI ICS Mid-Career Excellence in research award, and the Hellman and the Noyce Faculty Fellowships. His group has received funding from Allen Institute for AI, Amazon, NSF, DARPA, Adobe Research, Hasso Plattner Institute, NEC, Base 11, and FICO. Sameer has published extensively at machine learning and natural language processing venues, including paper awards at KDD 2016, ACL 2018, EMNLP 2019, AKBC 2020, and ACL 2020.
Kamyar has been a Research Staff at Nvidia since the Summer of 2022. Prior to his role at Nvidia, he was an assistant professor at Purdue University, department of computer science, from Fall 2020 to Fall 2022. Before his faculty position, he was at the California Institute of Technology (Caltech) as a Postdoctoral Scholar in the Department of Computing + Mathematical Sciences. Before his postdoctoral position, he was appointed as a special student researcher at Caltech, working with ML and Control researchers at the CMS department and the Center for Autonomous Systems and Technologies. He is also a former visiting student researcher at Caltech. Kamyar Azizzadenesheli is a former visiting student researcher at Stanford University, and a researcher at Simons Institute, UC. Berkeley. In addition, he is a former guest researcher at INRIA France (SequeL team), as well as a visitor at Microsoft Research Lab, New England, and New York. He received his Ph.D. at the University of California, Irvine. Kamyar's research interest is mainly in the area of Machine Learning, from theory to practice. He works on topics including but not limited to Reinforcement Learning (Bandits to MPDs, POMDPs, etc), Applied Mathematics, Deep Learning, Neural Operators, Control Theory, Spectral Method, Optimization, High Dimensional Statistics, Risk Assessment, Online learning, Domain Shift, Active Learning, Safety, Adversarial Attacks, and Generative models through both learning theory and core scientific lenses.
Favour Nerrise is a student at the University of Maryland, College Park (Go Terps!), studying Computer Science and Mathematics with minors in Arabic and Global Engineering Leadership. Primarily identifying as Cameroonian, #237, Favour is an advocate of giving back to her community, currently by building her first STEM-focused school in Cameroon. She is currently conducting research with NASA Harvest on estimating cropland using ML techniques and with the Battle Data Lab on visualization recommender systems; she has previously applied AI and ML to research opportunities in novelty-based targeting with NASA JPL for the Mars 2020 Rover, integrated ML and agent-based modeling for hurricane forecast trajectories, and activation map analysis for filtering big data. She currently serves as the Region II Chairperson for the National Society of Black Engineers, is a member of the Women in Engineering Student Advisory Board at UMD, and a member of BlackinAI.
Deb Raji is a 2020 Mozilla Fellow who has worked closely with the Algorithmic Justice League initiative, founded by Joy Buolamwini of the MIT Media Lab, on several projects to highlight cases of bias in computer vision. Her first-author work with Joy has been featured in the New York Times, Washington Post, The Verge, Venture Beats, National Post, EnGadget, Toronto Star and won the Best Student Paper Award at the ACM/AAAI Conference for AI Ethics & Society. An enthusiastic student eager to make constructive contributions to society, while heavily investing in self improvement along the way. A self-starter, personally responsible for projects with a notable impact on the local, regional, national and global scale. A determined social entrepreneur and volunteer, committed to helping her surrounding community. Feel free to contact me about internship and full time opportunities : deborah.raji@mail.utoronto.ca
There are few things I love more than cuddling up with an exciting new book. There are always more things I want to learn than time I have in the day, and I think books are such a fun, long-form way of engaging (one where I won’t be tempted to check Twitter partway through). This book roundup is a selection from the last few years of TWIML guests, counting only the ones related to ML/AI published in the past 10 years. We hope that some of their insights are useful to you! If you liked their book or want to hear more about them before taking the leap into longform writing, check out the accompanying podcast episode (linked on the guest’s name). (Note: These links are affiliate links, which means that ordering through them helps support our show!) Adversarial ML Generative Adversarial Learning: Architectures and Applications (2022), Jürgen Schmidhuber AI Ethics Sex, Race, and Robots: How to Be Human in the Age of AI (2019), Ayanna Howard Ethics and Data Science (2018), Hilary Mason AI Sci-Fi AI 2041: Ten Visions for Our Future (2021), Kai-Fu Lee AI Analysis AI Superpowers: China, Silicon Valley, And The New World Order (2018), Kai-Fu Lee Rebooting AI: Building Artificial Intelligence We Can Trust (2019), Gary Marcus Artificial Unintelligence: How Computers Misunderstand the World (The MIT Press) (2019), Meredith Broussard Complexity: A Guided Tour (2011), Melanie Mitchell Artificial Intelligence: A Guide for Thinking Humans (2019), Melanie Mitchell Career Insights My Journey into AI (2018), Kai-Fu Lee Build a Career in Data Science (2020), Jacqueline Nolis Computational Neuroscience The Computational Brain (2016), Terrence Sejnowski Computer Vision Large-Scale Visual Geo-Localization (Advances in Computer Vision and Pattern Recognition) (2016), Amir Zamir Image Understanding using Sparse Representations (2014), Pavan Turaga Visual Attributes (Advances in Computer Vision and Pattern Recognition) (2017), Devi Parikh Crowdsourcing in Computer Vision (Foundations and Trends(r) in Computer Graphics and Vision) (2016), Adriana Kovashka Riemannian Computing in Computer Vision (2015), Pavan Turaga Databases Machine Knowledge: Creation and Curation of Comprehensive Knowledge Bases (2021), Xin Luna Dong Big Data Integration (Synthesis Lectures on Data Management) (2015), Xin Luna Dong Deep Learning The Deep Learning Revolution (2016), Terrence Sejnowski Dive into Deep Learning (2021), Zachary Lipton Introduction to Machine Learning A Course in Machine Learning (2020), Hal Daume III Approaching (Almost) Any Machine Learning Problem (2020), Abhishek Thakur Building Machine Learning Powered Applications: Going from Idea to Product (2020), Emmanuel Ameisen ML Organization Data Driven (2015), Hilary Mason The AI Organization: Learn from Real Companies and Microsoft’s Journey How to Redefine Your Organization with AI (2019), David Carmona MLOps Effective Data Science Infrastructure: How to make data scientists productive (2022), Ville Tuulos Model Specifics An Introduction to Variational Autoencoders (Foundations and Trends(r) in Machine Learning) (2019), Max Welling NLP Linguistic Fundamentals for Natural Language Processing II: 100 Essentials from Semantics and Pragmatics (2013), Emily M. Bender Robotics What to Expect When You’re Expecting Robots (2021), Julie Shah The New Breed: What Our History with Animals Reveals about Our Future with Robots (2021), Kate Darling Software How To Kernel-based Approximation Methods Using Matlab (2015), Michael McCourt
The nlp_embeddings group is dedicated to learning through sharing knowledge, code, and resources with a focus on embeddings, transformers, and other NLP technologies. The group meets every Thursday at 10 am PT. Each week we discuss our personal and professional NLP projects, providing advice and guidance and sharing code. We are also working towards contributing novel solutions to Project AIMS (Artificial Intelligence against Modern Slavery). The group is open to anyone interested in NLP, including beginners! All levels of experience are welcome. We look forward to having you join us.
Checkout our recent Causal Modeling in ML webinar with Robert Ness here We’re collaborating with research scientist and instructor Robert Ness to bring his course sequence, Causal Modeling in Machine Learning, to the TWIML Community. Causality has become a very hot topic in the ML/AI space. In fact, it’s come up in a good number of my recent conversations, like this one with Zach Lipton. One of the challenges facing those interested in learning about causality in ML is that most resources on the topic are geared towards the needs of statisticians or economists, versus those of data scientists and machine learning engineers. Robert, an ML research scientist at startup Gamalon and an instructor at Northeastern University, has developed a series of six course modules on Causal Modeling in Machine Learning that are designed to be more practical and accessible for data scientists and engineers. He is teaching the course live to graduate students at Northeastern University, and through our new partnership Robert will also be hosting a study group via the TWIML platform, i.e. Zoom and Slack. The study group will provide TWIML enrollees some of the benefits of taking the course live. Robert will hold a weekly review session after each week of study in the sequence, will be available to answer questions via Slack, will personally grade submitted assignments, and will be available to assist with course homework and projects. The previous cohort of this course received great feedback from students: "I liked the course very much. Robert did a great job of reaching out to students to understand their background and interest in the course. It was great how he then continued to use what he learned about the students to make the course relevant and engaging to everyone enrolled. I also like how he made a connection to new paradigms. It was really nice to feel that the course is up to date. There are a lot of machine learning courses but this course was really special." "I loved the course. I learned a ton and Robert was very available to students. When I think about how much I would have paid at my university for a similar course, TWIML is a great value." To learn more about the courses and study group please feel free to peruse the FAQ we've prepared below. Note, enrollment is open through Thursday, September 17th. When you're ready to enroll, you can do so at Robert's AltDeep.ai web site. Be sure to also join the TWIML Community and the #causality_course channel on our Slack. Frequently Asked Questions What are the courses? The course sequence consists of six modules as listed on the AltDeep web site. They are: Model-based Thinking in Machine Learning. Lay the foundation for causal models by deconstructing mental biases and acquiring new mental models for applied DS/ML. Do Causality like a Bayesian. Continue your “mental refactoring” by developing a Bayesian mental model for machine learning. How to Speak Graph; or DAG that's a Nice Model! Become fluent in directed graphs and graph algorithms as a language of probability. The Tao of Do; Modeling and Simulating Causal Interventions. Learn to build your first causal generative machine learning model using a deep learning framework Applied Causal Inference; Identification and Estimation of Causal Effects from Data. Gain mastery of programmatic causal effect estimation. Counterfactual Machine Learning. Implement counterfactual reasoning algorithms in automated decision-making settings in industry. Are the courses free? How much are they? How are they sold? These are paid courses. Robert has put a ton of work into this sequence and will be providing TWIML learners with human support as they take the courses. Rather than selling the modules individually, Robert offers enrollment in the full Causal Modeling in Machine Learning Track for $1,199. This course sequence is designed to take you deeper into the practice of causal ML. How long will each course run? What is the level of effort expected? The course will run from September 10th to December 10th. On time commitment, if you just want to go through lectures and videos, then the time commitment is akin to a deep read of one paper a week. If you wanted to work through code examples and assignments, then more. The course is designed to give you a level of depth that suits you. Is there a discount for TWIML participants? Glad you asked. Yes, to kick off this partnership, Robert has agreed to extend a 15% discount to TWIML community members who register using the links above. I suspect this is the lowest price these courses will ever be offered for. Please use the discount codes TWIML2020FALL or TWIML2020FALL (for the monthly payment plan) to get the TWIML participant discount. Is TWIML paid as part of this arrangement? Yes, we are an AltDeep / Teachable affiliate and get a commission as part of the partnership. Whatever we earn through this relationship will help support our broader community and educational efforts. That said, we would never recommend a course we didn’t think was a good use of your time and a good value. How long will students have access to the course materials? After you enroll, you will have access to the materials indefinitely. How long will the course be open? While the course itself is fundamentally designed for self-paced study, with Robert running a live weekly study group, enrollment will be closed on September 17th. Will the weekly study group sessions be open to anyone? Robert’s weekly study group sessions are intended for enrollees and will assume that learners have at least gone over that week’s lectures at a high level. Is there a detailed syllabus? Yes, the syllabus will roughly follow that of Robert’s Northeastern course, which you can find here. What programming languages/frameworks are used in the course? The courses incorporate probabilistic programming concepts and use Pyro. From the Pyro web site:Pyro is a universal probabilistic programming language (PPL) written in Python and supported by PyTorch on the backend. Pyro enables flexible and expressive deep probabilistic modeling, unifying the best of modern deep learning and Bayesian modeling. Where can I learn more about Robert and the course? Please check out my recent podcast, Causality 101, with Robert, here. What can I expect from the weekly online sessions? The weekly online sessions are live study group sessions presented by Robert, the instructor and author of the Causal Models in Machine Learning courses. At the sessions, Robert will present a summary of that week’s lecture and open the floor for student Q&A. What if I cannot participate in the weekly online sessions? The weekly study group sessions will be recorded and will be available to TWIML enrollees. What is the refund policy? There is a 30-day refund policy on the courses.
A couple of days ago was #GivingTuesday, a “global generosity movement” that aims to harness some of the energy of Black Friday and Cyber Monday in support of charitable organizations. Hopefully you took advantage of the opportunity to donate to organizations you believe in. While you’re in the giving spirit, don’t forget those organizations working to make the field of AI more accessible, responsible and diverse. We want to make this a thing in our community… we can call it #GivingTuesdAI! To kick this off, I’m happy to announce that we’ll match the first $5,000 in donations made to any of the below organizations, up to $500 per individual, through December 27th. To apply for the match, reply or tweet me (@samcharrington) a screenshot of your donation receipt with the tag #GivingTuesdAI. 1. AI Ethics Algorithmic Justice League (donate): Raises awareness about the impact of AI through research, art, and media to mitigate AI harms and biases. The Future Society (donate): A research, education, and advocacy organization that aims to advance the responsible adoption of AI. AI For Good (donate): A nonprofit that harnesses the power of AI to create positive social and environmental change. 2. Education Girls Who Code (donate): Creates programs that connect girls ages 8-22 with clubs, classes, and summer programs, and showcases female coders in order to close the gender gap in computer science. AI4All (donate): Connects educators, mentors, and students of diverse backgrounds to increase diversity and inclusion in AI education, research, development, and policy. KhanAcademy (donate): Provides free, high-quality educational resources for Kindergarten through college students and teachers. Sal Khan actually spoke about Khan Academy and the future of education at TWIMLfest last year. 3. Professional Organizations Black in AI (donate): Increases the inclusion of Black people in the field of AI by creating space for sharing ideas and fostering collaborations through mentorship and advocacy. Latinx in AI (donate): Drives and supports research, development, infrastructure, and mentoring programs to boost innovation and capabilities of LatinX professionals. Women in AI (donate): Empowers women and minorities to become AI & Data experts, innovators and leaders through mentorship and conferences. There are certainly other organizations worthy of our support. Comment any of your favorites that we’ve missed in this list!
A quick note: While they’re still early in their ML/AI journey, Stack Overflow is a company that is well-known and beloved in technical communities, and often one of the first places we look to solve problems big or small. For that reason I’m excited to share the conversation with you. That said, if you’re looking for an in-depth technical analysis, you might want to skip this one. Back to the Basics: Introducing Stack Overflow & Prashanth Chandraseka Stack Overflow was founded in 2008, and has been growing massively ever since. The site hosts almost 100 million monthly visitors from around the world, who can access 50 million questions, which amounts to 1 TB of publicly accessible data in the form of questions, answers, and comments. While most people recognize Stack Overflow for its public-facing Q&A site, the company is also developing products to help businesses foster collaboration by using private versions of Stack Overflow-like sites within their organizations. Prashanth became CEO of Stack Overflow in October 2019. Having spearheaded Rackspace’s pivot from hosting to managing cloud services, Prashanth had firsthand experience in growing a business in this direction quickly. Impact of COVID at Stack Overflow Just 6 months after becoming Stack Overflow’s CEO, COVID-19 revolutionized the way the company was run. The impact was mixed. On the one hand, the widespread transition to a remote workforce increased the number of people coming to Stack Overflow. When Prashant joined the company, there were about 150,000 monthly signups on the public platform, a number which has since increased to the 200,000-250,000 range. As technology was one of the few sectors whose revenue increased over the pandemic, the transition to remote work actually also allowed for increased adoption of Stack Overflow’s Teams integration and advertising products. The objective of the Stack Overflow Teams product is to minimize context switching and conserve the energy of technologists. In order to do this, the team created an asynchronous collaboration tool that automates some frequent responses, using common tools like Slack and Microsoft Teams. For example: imagine a new employee asks you where a specific repository is via Slack. The Stack Overflow integration then suggests similar questions and answers that have already been documented. The new employee gets an immediate response, and you don’t have to spend time explaining or tracking down information. Since technology companies are always looking for quality places to showcase their products, Stack Overflow was the perfect place to reach technical communities. Stack Overflow’s advertising product Reach and Relevance saw a significant increase in advertising revenue during 2020. Despite these positives, the same negative things that affected the world in 2020 deeply affected the personnel at Stack Overflow. Personal health issues, families that were affected by COVID, and the cultural reckoning around racism and inclusion, made it a challenging time. There were additional challenges, such as having to pivot away from certain products, like a job listing product that was too unpredictable as a steady revenue source. Recent Acquisition by Prosus Stack Overflow was recently acquired by Prosus, a move that Prashanth attributes to the companies’ similar missions and alignment of their long-term goals. While Stack Overflow had been approached many times over the years, Prashanth believed Prosus could help Stack Overflow think big and grow to its full potential in a global, community-oriented way. “Prosus builds leading consumer internet companies and empowers people and enriches communities.” When you think about that statement, that speaks immediately to the heart and soul of this company. Stack Overflow Community Philosophy Stack Overflow has a few principles that guide the way they approach building and managing communities. 1. Focus on a shared identity. Their mission is guided by focusing on elements Stack Overflow users share, like a love of problem solving. 2. Create sufficient incentives for people to contribute and participate in the system. At Stack Overflow, that incentive is helping out your fellow developer, showcasing your knowledge, and being recognized for it. 3. Build with the community instead of for the community. Prashanth sees it as “common sense” to recognize the community members as equal partners, and the team has built in modes of feedback and collaboration through moderator mechanisms and an email newsletter. 4. Break down silos and build bridges across various communities. One way Prashanth decided to do this was by hiring Felipe Baudette (formerly of Reddit) as the new VP of Community. “Ultimately, it’s all about making sure that we’re building thriving communities.” One distinction Prashanth mentions is that while Stack Overflow is a community space where members share information, it is not a social network. It’s less a matter of sharing opinions, and more a matter of right and wrong answers: either the code will work or it won’t. He reiterates that people appreciate Stack Overflow because it gives them the information they need to solve their problems. Data at Stack Overflow Stack Overflow is relatively early on in their adoption of AI. While there’s a ton of interesting data they have collected (70 TB worth!), they are currently building out a team who will be able to do creative things with it. One of the most interesting data sources they collect is a developer survey which polls about a hundred thousand people; this past year, 85,000 people responded. While this yearly survey has a lot of great insights, Prashanth would love to be able to generate valuable insights on a more frequent basis. Prashanth’s favorite finding was that 70% of developers reported they learn a new technology at least once a year. While he knew that developers liked to learn new things, 70% was a much higher number than expected. Prashanth also mentioned that 60% of these developers are learning to code from online resources. While there is some amount of structured data, the vast majority of what the team collects is unstructured data in the form of Q&A text, comments, and tags. They have also developed a network graph that can track a person’s workflow and help inform how users engage with the platform. Stack Overflow is currently establishing the groundwork for a machine learning and data science team– with a data lake, training spaces, and other necessities for AI. While they are still early in their journey, there are already two interesting endeavors they’ve created. Early ML Applications: The Unfriendly Robot and Teams Integration One of these projects involves making Stack Overflow a more amenable place for beginners. A common criticism of Stack Overflow is that it’s a very harsh environment for a new user, as more experienced Stack Overflow users will shoot down novice questions if they see them as “bad”. The Unfriendly Robot is a bot that scans all the public exchanges and flags inappropriate comments to a question, noting any “unfriendly” language. The team is continuing to evaluate the best way to build a softer “landing zone”, a place where people can onboard in a friendly environment and try out a few things before they actually go and ask that first question. The other area where ML is being used is in a bot in the Stack Overflow Teams product, the same product that’s integrated with Slack and Microsoft Teams. The bot helps suggest answers to FAQs. When someone asks a question on the Stack Overflow for Teams, the bot can provide the user with a few other questions that sound similar and offer to pull up the answer that was documented previously. This allows for better remote onboarding and less time spent responding to the same questions. The Future of Stack Overflow Two of the questions that drive future innovation at Stack Overflow are: How do you improve efficiency and how do you remove context switching? Moving forward, Prashanth hopes to automate parts of the Stack Overflow workflow that make it even easier and more efficient. Another goal Prashanth has is expanding their US-centric customer base internationally, in partnership with Prosus. Prashanth also hopes to integrate Stack Overflow even more deeply into the technologist workflow, ideally creating “connective tissue” between that public platform and the private teams version, and between Stack Overflow instances and other developer tools like IDEs. Ultimately, Prashant wants to build thriving communities and effective remote workflows that embrace a culture of learning. He imagines Stack Overflow integrations helping technologists work and scale more effectively, ultimately helping to build better technology for the future of humanity. To hear more about what’s going on at Stack Overflow, you can listen to the full episode here!
https://www.youtube.com/watch?v=eyGtJVVmVUM&t=180s Causality and causal modeling Causality and causal modeling is one of the hottest topics in machine learning. Earlier this year we launched a new causality course and study group with instructor Robert Osazuwa Ness, which received great feedback from students: “I liked the course very much. Robert did a great job of reaching out to students to understand their background and interest in the course. It was great how he then continued to use what he learned about the students to make the course relevant and engaging to everyone enrolled. I also like how he made a connection to new paradigms. It was really nice to feel that the course is up to date. There are a lot of machine learning courses but this course was really special.” “I loved the course. I learned a ton and Robert was very available to students. When I think about how much I would have paid at my university for a similar course, TWIML is a great value.” I’m happy to announce that the Fall 2020 course and study group will be starting on September 17th! Check out our recent webinar above to learn more about the course. Then, visit our Causal Modeling in Machine Learning course page to enroll.
Today we're joined by Drago Anguelov, Distinguished Scientist and Head of Research at Waymo. In our conversation, we explore the state of the autonomous vehicles space broadly and at Waymo, including how AV has improved in the last few years, their focus on level 4 driving, and Drago's thoughts on the direction of the industry going forward. Drago breaks down their core ML use cases, Perception, Prediction, Planning, and Simulation, and how their work has lead to a fully autonomous vehicle being deployed in Phoenix. We also discuss the socioeconomic and environmental impact of self-driving cars, a few research papers submitted to NeurIPS 2020, and if the sophistication of AV systems will lend themselves to the development of tomorrow's enterprise machine learning systems.
Friday’s TWIMLcon Executive Summit closed out a full first week at the conference! Speakers from BP, Walmart, Accenture, Qualcomm, Orangetheory Fitness, and more shared their experiences and insights on key issues faced by AI/ML leaders and teams. The day began with a keynote interview featuring Franziska Bell, VP of Data and Analytics at BP. Fran had some very strong advice on what it takes to ensure ML project success. Her principles include creating mutual partnership between the business and the data team early on in the process; working hard to ensure that the data team is actually solving the business need; and emphasizing the importance of empathy, understanding, and common goals and language among the cross-disciplinary teams building data products. The first panel of the day focused on Building the Business Case for ML Platforms and featured Divya Jain (Director of ML Platform, Adobe), Justin Norman (VP Data Science and Analytics at Yelp), and Kirk Borne (Principal Data Scientist and Executive Advisor, Booz Allen Hamilton). We discussed business value, measuring impact and ROI, build vs. buy, centralized vs. embedded teams, and standardization of infrastructure vs. flexibility. One attendee question prompted panelists to explore the topic of whether centralization was even a good thing. All panelists had strong opinions on this topic--not always in agreement--but Justin summarized it well with the following: “Businesses have many teams, those teams have requirements and those requirements should drive the platform choices. If it makes sense to centralize something... then do it. But if a team is doing something very unique with a different set of requirements than the other teams, they may need their own vertically integrated stack.” The next session had Adrian Cartier (VP of Data Science, Ocelot Consulting), Andy Minteer (Senior Director, Digital Transformation - Head AI Products, Walmart Global Tech), Jurgen Weichenberger (Data Science Senior Principal & Global AI Lead, Resources, Accenture) up to discuss Why ML projects Fail and How to Ensure Their Success. Right off the bat, Andrew challenged the idea of failure and had us rethink what failure even means. He asked: “What if the model is accurate but nobody adopts it? Isn’t that also failure?” Jurgen, who has worked with many customers in many industries, cautioned that it’s important to back up even further and to assess where the customer is on their maturity curve: Some industries are further ahead than others and that will drive a lot of what success and failure even mean to them. The panel closed with a discussion about the central role of people in the technology decisions leaders make. Jurgen offered: “It is our obligation to bring our customers on the journey with us. We need to be in the mindset that we are enabling people to do their jobs. You need to take the whole company on a journey with you... Bring them along, build trust and confidence, and show them how this can make their lives easier.” The fourth session of the day centered around what is required when Building Teams and Cultures that Support ML Innovation. For this discussion, we invited Ameen Kazerouni (Chief Analytics Officer, Orangetheory Fitness), Pardis Noorzad (Head of Data Science, Carbon Health), and Ziad Asghar (VP of AI at Qualcomm) to share their thoughts. The conversation included topics such as: what are the factors in building high-performance teams; how do we measure team success; and what is the role of culture in building teams. Sufficient budgets, common language, and shared rituals were all mentioned as key elements enabling effective teams. The impact of the pandemic on teams, namely the accelerated shift to remote work, was discussed as well. Ziad left us with this amusing thought on the topic: “If 2020 had a t-shirt, it would read: ‘Hey we can’t hear you, you’re on mute,’” illustrating how fundamental some of the challenges we face are. The final session of the day, the Executive Summit Roundtable Discussion, was a particularly animated and rich discussion. Ameen Kazerouni, Hussein Mehanna (VP, Head of ML/AI, Cruise), and Paul van der Boor (Senior Director of Data Science, Prosus Group), each shared their experiences on a topic relevant to leading ML teams and then Sam facilitated a great discussion afterwards with the attendees. A few of the many compelling ideas that came out of this section include: Ameen’s suggestion that: “The currency of an analytics team is trust, not data.” Paul’s insights from the experience of one of the teams at Prosus which has developed dashboards to granularly track the impact of every ML model they deliver on the business, and the need to understand whether a project’s key contribution is operational (improving what you already do) or innovation (doing new things). Hussein’s definition of “AI Native Products” as those that must leverage AI even at the MVP stage and his mind-blowing hypothesis (presented first at TWIMLcon!) that in order for organizations to create AI-Native Products that they need to organize internally like a neural network. We had a fun and engaging discussion after those three wrapped up and I think the summary was that we in the ML community have been spoiled with an explosion of new tools and techniques, and that at some point, there will likely be a “great reckoning” where the toolchain will all converge and MLOps will become more standardized. As one attendee, Gavin Bell, put it: “We used to have serial ports, parallel ports, printer ports, display ports, headphone jacks...and now it’s all USB-C. What is the respiration of this round of technology going to leave behind? “ Big thanks to Adrian, Ameen, Andy, Divya, Franziska, Jurgen, Justin, Kirk, Pardis, Paul, Ziad, Hussein and all of the Executive Summit attendees for a fun and stimulating day of discussion on these very important topics. And special thanks to Qualcomm, Executive Summit Platinum Sponsor. If you missed the session today, it’s not too late to register for TWIMLcon! There are still four more days of sessions next week. “Executive” tickets offer on-demand access to all of the Executive Summit sessions you missed, as well as the entirety of TWIMLcon. All tickets offer on-demand access to all regular conference sessions through the end of January, and “Pro Plus” and “Executive” tickets let you watch replay sessions whenever you like. You can check out the TWIMLcon agenda here and the speakers here. See you next week!
Day 2 (of 8!) of TWIMLcon: AI Platforms 2021 was a day of sharing hard-earned lessons. (The conference started yesterday and runs through January 29, 2021. It’s not too late to join in! Use discount code GREATCONTENT for 25% off registration.) We kicked off the day interviewing Faisal Saddiqi, Director of Engineering for Personalization Infrastructure at Netlix. Faisal has been at Netflix for a little over six years and he shared a ton of great lessons learned by him and his team while building out their internal ML platforms. This was a very dense discussion, full of hard lessons and good advice. Some key take-aways that stood out: Get clear on your internal users and what they need as your customer and then build systems that empower them to do their work with the tools they want to use. Be both opinionated AND flexible. Use prescriptive approaches and technologies lower in the stack and where you need to maintain control and provide more flexibility up at higher levels where people need the room to innovate. Overall the discussion on structure vs. flexibility was worth the price of admission as it ties into usability of the platforms we’re all building. Understand that components in your tech stack and MLOps platform are probably going to be mixed and matched between the four possibilities: build it (DIY), borrow it (from elsewhere in the company), use an open-source element, or use a commercial solution. He commented that Netflix and his team used all four options. There was so much more in this conversation and it was such a great start to the day. I highly recommend going back and catching the replay of this episode. Next up, we heard from Todd Underwood, an Engineering Director at Google. Todd walked us through how models fail and how to prevent it. He probably made a lot of people feel both better and worse by starting off saying that model quality is a common production problem and that it’s both an operational (systems) problem and also a human trust problem. Basically, if they fail (and they will) and you don’t know why they fail, people are less likely to trust them. From there, he walked through his lessons learned on how to think about failure as a gift, how to look past the obvious sources of failure to the more esoteric and boring causes, and how to learn from failures as an organization over time. From there, he walked through classes of failures and their frequent causes and then illustrated the principles he had laid out by walking through a particular story. While he had many interesting quotes in the presentation, I’ll put one of my favorite ones here: “Understand YOUR system, and your system’s failures. It is worth doing. It pays dividends in better models, more resilience...if you don’t monitor model quality yet, start. If you don’t write and track post mortems yet, start. When you have an outage, make sure you learn everything you can from it. You’ve already paid for it, so you should get the value out of it.”  - Todd Underwood, Engineering Director • Google Overall, his presentation was a call to action to embrace failure and accept it as a part of building complex systems generally, and AI/ML systems in particular. This talk is worth sharing with your whole team and taking action on. After that tough love talk, we got to hear from Ariel Biller, an Evangelist at ClearML and his customer Dotan Asselman, Co-Founder and CTO of theator. The talk continued on with the Build vs. Buy debate that Faisal touched on in the morning keynote. Spoiler alert: Both of them agreed on the core insight of this talk: “It’s not build vs. buy - it’s build AND buy and that golden ratio is use-case specific. ‘Buy’ also means open-source - remember that it may be ‘free’ but it has associated support costs.” Ariel Biller, Evangelist • ClearML What was really great about this talk was that they outlined the end-to-end ML platform system at theator and walked through which components were BUILT and which were BOUGHT. More importantly, they explained the thinking behind those decisions. I won’t get into the details here as you can check out the replay until the end of the conference (or after the conference for the Pro Plus Passes or Executive Summit pass holders). I encourage you to check out the full presentation. As we got into the thick of the day, Chip Huyen, the author of the excellent MLOps Tooling Landscape, let the audience know that ML is going real-time and that they’re probably not prepared for it. (What a day of tough love around here!) Chip’s core message was that organizations needed to move beyond thinking about real-time vs batch, but rather consider “online learning.” “Online learning is crucial for systems to adapt to rare events... Because Black Friday happens only once a year, there’s no way Amazon or other ecommerce sites can get enough historical data to learn how users are going to behave that day, so their systems need to continually learn on that day to adapt.” She then discussed how the two pipeline architecture that many systems have (a batch based pipeline for training and a streaming data pipeline for inference) is a common source of production failure and that teams should be looking at ways to unify those into a common pipeline that does both. Overall, she made a compelling case for rethinking the status quo of system architectures and considering whether online learning should be a goal for your system design. As with the others above, we can’t really do it justice here: check out the replay! To continue on with the themes of systems and their components, Monte Zweben, CEO of Splice Machine shared his thoughts on feature stores - what they are, what they do, and how they’re traditionally deployed in a three database architecture alongside scale-out operational databases and scale-out analytical data platforms. From there, he explained how Splice Machine has unified the three functions in one open-source system, to help customers deliver features much faster, and simplify the lifecycle of ML models. He made a argument for a database-centric approach to MLOps and I’d encourage any of you wrestling with the complexities feature management and delivery to go chat with Monte and his co-presenter Jack Ploshnick here at the conference this week to learn more. The second last session of the day was a fun panel discussion with a bunch of the Spotify ML team who shared their thoughts on how to drive platform adoption within their broader company. A key takeaway from this discussion was that “if you build it, they will come” is not enough at a certain level of scale. Spotify created a new “engagement manager” role on its platform team to address this, with a focus on evangelizing the platform to Spotify teams, and helping them be successful. There were lots of lessons in this chat for anybody building and evangelizing an internal ML platform. Finally, we closed out the day with a workshop presented by John Posada, a Partner Solutions Architect at Dataiku. Echoing what Ariel Biller discussed earlier in the day, John discussed how technical debt builds up, for example as regulatory frameworks change, and if you’re not agile enough your AI systems can fall afoul of the regulatory environment, causing customer and business harm. This is the stuff that keeps your risk management team up at night (and probably your CEO as well.) He suggested that the key is to use modular platforms that let you evolve the elements in the system while not changing the whole system, adding a layer of governance and building in guardrails to ensure fair and responsible use of AI. Dataiku’s answer to all of these requirements is their Data Science Software (DSS) platform and John presented a thorough walk through of how it can be used to create end-to-end MLOps pipelines. Tomorrow, we will be changing things up by shifting the focus to two major workshops, plus a networking session: David Hershey, a Solutions Architect for Tecton AI will walk through an entire case study in how to deploy a Fraud Detection Model with their Feature Store More networking (with a twist!) Kristopher Overholt, a Solution Engineer from Algorithmia, will demonstrate how to move models from training into production. Friday, our Executive Summit sessions will be happening, and then the regular mix of technical sessions will pick up again on Tuesday January 26th. If this sounds interesting, it’s not too late to register! There are still six more days of sessions, including Friday’s Executive Summit. Pro Plus and Executive passes provide ongoing access to the conference recordings so that you can catch up after the event. You can check out the agenda here and the speakers here. Thanks to all of today’s speakers Faisal Siddiqi, Todd Underwood, Dotan Asselman, Ariel Biller, Chip Huyen, Monte Zweben, Maya Hristakeva, Lex Beattie, Maisha Lopa, Samuel Ngahane, and John Posada for their time and contributions to a great day of learning.
Today we're back with the final episode of AI Rewind joined by Michael Bronstein, a professor at Imperial College London and the Head of Graph Machine Learning at Twitter. In our conversation with Michael, we touch on his thoughts about the year in Machine Learning overall, including GPT-3 and Implicit Neural Representations, but spend a major chunk of time on the sub-field of Graph Machine Learning. We talk through the application of Graph ML across domains like physics and bioinformatics, and the tools to look out for. Finally, we discuss what Michael thinks is in store for 2021, including graph ml applied to molecule discovery and non-human communication translation. We want to hear from you! Send your thoughts on the year that was 2020 below in the comments, or via Twitter at @samcharrington or @twimlai. To follow along with the 2020 AI Rewind Series, head over to the series page.
Today we continue the 2020 AI Rewind series, joined by friend of the show Sameer Singh, an Assistant Professor in the Department of Computer Science at UC Irvine. We last spoke with Sameer at our Natural Language Processing office hours back at TWIMLfest, and was the perfect person to help us break down 2020 in NLP. Sameer tackles the review in 4 main categories, Massive Language Modeling, Fundamental Problems with Language Models, Practical Vulnerabilities with Language Models, and Evaluation. We also explore the impact of GPT-3 and Transformer models, the intersection of vision and language models, and the injection of causal thinking and modeling into language models, and much more. We want to hear from you! Send your thoughts on the year that was 2020 below in the comments, or via Twitter at @samcharrington or @twimlai. To follow along with the 2020 AI Rewind Series, head over to the series page!
AI Rewind continues today as we're joined by Pavan Turaga, Associate Professor, in both the Departments of Arts, Media, and Engineering & Electrical Engineering, and the Interim Director of the School of Arts, Media, and Engineering at Arizona State University. Pavan, who joined us back in June to talk through his work from CVPR ‘20, Invariance, Geometry and Deep Neural Networks, is back to walk us through the trends he's seen in Computer Vision last year. We explore the revival of physics-based thinking about scenes, differential rendering, the best papers, and where the field is going in the near future. We want to hear from you! Send your thoughts on the year that was 2020 below in the comments, or via Twitter at @samcharrington or @twimlai. To follow along with the 2020 AI Rewind Series, head over to the series page!
We hope you’re taking some time to relax over the holidays and want to make sure you have plenty of content for when you’re in the mood for a technical binge! December is always slammed with end-of-year events, and this year is no different. We’ve covered some of the month’s events and innovations in our three series this month: NeurIPS, re:Invent, and AI Rewind! re:Invent As always, AWS packed a ton of innovations and announcements into this year’s re:Invent conference. This year, we caught up with none other than Swami Sivasubramanian, VP of AI at AWS, for a re:Invent Roundup to discuss them all. One new offering this year is Amazon SageMaker Feature Store. It turns out that the new service is modeled after the feature store developed by Intuit. In the second episode in this series, we speak with Srivathsan Canchi, head of engineering for the Machine Learning Platform team at Intuit, to dig into the details and backstory. This series also concludes a chat with Edgar Bahilo Rodriguez of Siemens Energy who joins us to talk about Productionalizing Time-Series Workloads, a topic he touched on in his re:Invent talk. NeurIPS As always, this year’s NeurIPS conference featured a ton of great innovation on the research front, and once again we bring those researchers to you with our NeurIPS series. This time around, we spoke with Taco Cohen, ML Researcher at Qualcomm to discuss his work and NeurIPS talk on Natural Graph Networks and video compression using generative models. Another 🔥 conversation in that series is with Charles Isbell, AI researcher and Dean of Engineering at Georgia Tech, who joins us to dive into ML as a Software Engineering Enterprise, the topic of his invited NeurIPS talk exploring the need for a systems approach to tackling big issues in ML like bias. Finally, Aravind Rajeswaran, PhD student at the University of Washington, joins us to talk about his paper, MOReL: Model-Based Offline Reinforcement Learning. AI Rewind Now in its third year, TWIML’s AI Rewind series has become a perennial crowd favorite. The series brings back friends of the show to discuss and explore the year’s most important trends in key topical areas like Machine Learning/Deep Learning, Computer Vision, Natural Language Processing, and Reinforcement Learning. For each topic, we discuss important research and commercial developments of the year and predictions for the year ahead. We just kicked off the AI Rewind series today! Our first episode features my conversation with Pablo Samuel Castro exploring Trends in Reinforcement Learning. Stay tuned for more great shows in this series! We wish you lots of binge-worthy content this holiday season and we’ll see you in the new year!
Chris Albon joined the Wikimedia Foundation in 2020 as director of machine learning with a mandate to empower the teams producing models for Wikipedia to move more quickly. In this interview, Sam and Chris discuss ML use cases at Wikimedia, the evolution of the organization’s ML infrastructure, their use of Kubernetes and Kubeflow to support the ML workflow, and their ultimate plans to make this infrastructure available to the broader Wikimedia community.
Every year we take time to recap the most important developments in the various disciplines under the broad machine learning umbrella. Even amongst the uncertainty that 2020 brought, the ML community still managed to produce some great breakthroughs. In these conversations, we brought back some of our friends from the podcast to cover the latest and greatest in Machine Learning & Deep Learning, Computer Vision, Natural Language Processing, and Reinforcement Learning research, tools, use cases, and even predictions for the year ahead. As always, we'd love to hear your thoughts on this series, including anything we might have missed. Drop your thoughts, feedback or favorite papers via twitter @samcharrington, or via a comment below.
Today we kick off our annual AI Rewind series joined by friend of the show Pablo Samuel Castro, a Staff Research Software Developer at Google Brain. Pablo joined us earlier this year for a discussion about Music & AI, and his Geometric Perspective on Reinforcement Learning, as well our RL office hours during the inaugural TWIMLfest. In today's conversation, we explore some of the latest and greatest RL advancements coming out of the major conferences this year, broken down into a few major themes, Metrics/Representations, Understanding and Evaluating Deep Reinforcement Learning, and RL in the Real World. This was a very fun conversation, and we encourage you to check out all the great papers and other resources available below. We want to hear from you! Send your thoughts on the year that was 2020 below in the comments, or via Twitter at @samcharrington or @twimlai. To follow along with the 2020 AI Rewind Series, head over to the series page!
This study group will follow the Deep Learning with Pytorch course at NYU. The course is about the latest techniques in deep learning in the areas of computer vision, NLP, and speech. The course also has dedicated lectures on graph neural networks and energy based models. Each lecture is followed by a practicum video that implements learnings in PyTorch. The study group will aim to cover all 31 videos in the series, one every week if possible. Each session will be discussion of the lecture/code led by a volunteer. This study group meets every Saturday at 8am PT starting on January 2, 2021 and will run through August 28th, 2021. The study group slack channel is #dl_pytorch (you can join our slack community by clicking on “join us” at twimlai.com/community). Course home page -> https://atcold.github.io/pytorch-Deep-Learning/ Lecture videos -> https://www.youtube.com/playlist?list=PLLHTzKZzVU9eaEyErdV26ikyolxOsz6mq
One of the silver linings to an otherwise crazy year was that I spent a lot of time at a desk that would have otherwise been spent in planes, trains, and automobiles. As a result, Team TWIML got quite a lot done this year! I thought I’d share a few of the accomplishments we’re most proud of this year. Expansion of Our Education & Community Programs Thinking back to our team retreat in January (good times!) a big part of what we wanted to accomplish this year was to broaden our education and community offerings and to further the vision of the TWIML Community as a place for those interested in ML and AI to advance their skills and knowledge. In January we partnered with instructor Robert Ness to launch his Causal Modeling in Machine Learning course to rave reviews. We’ve since delivered it to two additional cohorts, and we’re super excited to work with Robert to launch two new courses in January. We also worked with Luigi Patruno to launch his first course on Amazon SageMaker this year. Our community stepped up in a big way this year as well. This year we held more study groups than ever— sixteen in total—including standbys like the Fast.ai Practical Deep Learning for Coders course that we’ve run for several years now, as well as a host of new offerings like the aforementioned AI Enterprise Workflow group, and groups focused on topics like Deep Reinforcement Learning, Natural Language Processing, Kaggle, and more. Leaning into Video Early on in the pandemic, we realized that folks would be spending a lot more time in front of their computers than usual, and we decided to help them take advantage of this opportunity by leaning into video. We started with our first video interview back in April (Google’s Quoc Le) and at this point, I can’t remember the last audio-only interview I’ve done. We also added online panel discussions to our regular repertoire this year. In an effort to support our community during the pandemic we took on subjects like Responsible Data Science in the Fight Against COVID-19, and Advancing Your Data Science Career During the Pandemic. We’ve since broadened our coverage to include practical and technical topics like the ML Programming Language Un-Debate, the Model Explainability Forum, and Feature Stores for Accelerating AI Development. If you missed any of these, you can watch them on our YouTube channel. Please take a moment to subscribe here! TWIMLfest: A Virtual AI Festival We were really inspired by our community this year, and the passion and zeal they continued to bring to the study and advancement of ML and AI, even during a pandemic. To celebrate them and their accomplishments, we launched TWIMLfest, a global celebration of the TWIML Community. TWIMLfest was a virtual festival spanning three weeks, dedicated to connection, collaboration, fun, and learning. While we originally envisioned something much smaller, the event ultimately spanned 3 weeks and offered 40 sessions, hosted 70+ speakers, and saw over 1,700 members of our community register! Some of the highlights include my keynote Interview with Sal Khan, the panel discussion on Accessibility and Computer Vision, and the Coded Bias Film Screening + Director Q&A. Stacking the Bricks In addition to these programs, we also laid the groundwork for a lot of exciting work that will come to fruition in early 2021, including our AI Solutions Guide and the TWIMLcon conference, and we’re within spitting distance of achieving 10 million downloads, a milestone we expect to hit early next year. We’ve also been working behind the scenes to fine-tune a lot of what we do as a company, an ongoing effort that will allow us to elevate our for ML/AI programming and events once again. We can’t wait to see you all in 2021! From the entire TWIML team, we wish you happy holidays as we celebrate the end of this wild year. Cheers! Sam
As we continue our NeurIPS 2020 series, we're joined by friend-of-the-show Charles Isbell, Dean, John P. Imlay, Jr. Chair, and professor at the Georgia Tech College of Computing. This year Charles gave an Invited Talk at this year's conference, You Can't Escape Hyperparameters and Latent Variables: Machine Learning as a Software Engineering Enterprise. In our conversation, we explore the success of the Georgia Tech Online Masters program in CS, which now has over 11k students enrolled, and the importance of making the education accessible to as many people as possible. We spend quite a bit speaking about the impact machine learning is beginning to have on the world, and how we should move from thinking of ourselves as compiler hackers, and begin to see the possibilities and opportunities that have been ignored. We also touch on the fallout from Timnit Gebru being "resignated" and the importance of having diverse voices and different perspectives "in the room," and what the future holds for machine learning as a discipline.
https://www.youtube.com/watch?v=eyGtJVVmVUM&t=180s Causality and causal modeling is one of the hottest topics in machine learning. Earlier this year we launched a new causality course and study group with instructor Robert Osazuwa Ness, which received great feedback from students: “I liked the course very much. Robert did a great job of reaching out to students to understand their background and interest in the course. It was great how he then continued to use what he learned about the students to make the course relevant and engaging to everyone enrolled. I also like how he made a connection to new paradigms. It was really nice to feel that the course is up to date. There are a lot of machine learning courses but this course was really special.” “I loved the course. I learned a ton and Robert was very available to students. When I think about how much I would have paid at my university for a similar course, TWIML is a great value.” I’m happy to announce that the Fall 2020 course and study group will be starting on September 17th! Check out our recent webinar above to learn more about the course. Then, visit our Causal Modeling in Machine Learning course page to enroll.
Today we kick off our NeurIPS 2020 series joined by Taco Cohen, a Machine Learning Researcher at Qualcomm Technologies. In our conversation with Taco, we discuss his current research in equivariant networks and video compression using generative models, as well as his paper "Natural Graph Networks," which explores the concept of "naturality, a generalization of equivariance" which suggests that weaker constraints will allow for a "wider class of architectures." We also discuss some of Taco's recent research on neural compression and a very interesting visual demo for equivariance CNNs that Taco and the Qualcomm team released during the conference.
Today we're kicking off our annual re:invent series joined by Swami Sivasubramanian, VP of Artificial Intelligence, at AWS. During re:Invent last week, Amazon made a ton of announcements on the machine learning front, including quite a few advancements to SageMaker. In this roundup conversation, we discuss the motivation for hosting the first-ever machine learning keynote at the conference, a bunch of details surrounding tools like Pipelines for workflow management, Clarify for bias detection, and JumpStart for easy to use algorithms and notebooks, and many more. We also discuss the emphasis placed on DevOps and MLOps tools in these announcements, and how the tools are all interconnected. Finally, we briefly touch on the announcement of the AWS feature store, but be sure to check back later this week for a more in-depth discussion on that particular release!
Today we're joined by Sushil Thomas, VP of Engineering for Machine Learning at Cloudera. Over the summer, I had the pleasure of hosting Sushil and a handful of business leaders across industries at the Cloudera Virtual Roundtable. In this conversation with Sushil, we recap the roundtable, exploring some of the topics discussed and insights gained from those conversations. Sushil gives us a look at how COVID19 has impacted business throughout the year, and how the pandemic is shaping enterprise decision making moving forward. We also discuss some of the key trends he's seeing as organizations try to scale their machine learning and AI efforts, including understanding best practices, and learning how to hybridize the engineering side of ML with the scientific exploration of the tasks. Finally, we explore if organizational models like hub vs centralized are still organization-specific or if that's changed in recent years, as well as how to get and retain good ML talent with giant companies like Google and Microsoft looming large.
Today we're joined by Adina Trufinescu, Principal Program Manager at Microsoft, to discuss some of the computer vision updates announced at Ignite 2020. We focus on the technical innovations that went into their recently announced spatial analysis software, and the software's use cases including the movement of people within spaces, distance measurements (social distancing), and more. We also discuss the ‘responsible AI guidelines' put in place to curb bad actors potentially using this software for surveillance, what techniques are being used to do object detection and image classification, and the challenges to productizing this research. Get the transcript
Over the last few years, Deep Learning technologies have made a tremendous impact on Medical Imaging. In this session participants will hear about some of the latest advances in the following areas: medical image segmentation and classification and object detection in medical images. A review of these advances will be presented via a detailed analysis of research papers from prestigious journals such as Nature Methods. In-Depth Summary Details: Title: “Federated Learning: A privacy-preserving method for training deep learning models using healthcare data.” Speaker: Dr. Anthony Reina Federated learning is a distributed machine learning approach that enables organizations to collaborate without sharing sensitive data, such as, patient records, financial data, or classified secrets (McMahan, 2016; Sheller, Reina, Edwards, Martin, & Bakas, 2019; Yang, Liu, Chen, & Tong, 2019; Sheller et al., 2020). The basic premise behind federated learning is that the model moves to meet the data rather than the data moving to meet the model. Dr. Reina will discuss Intel’s work with the Federated Tumor Segmentation Initiative (FeTS) which uses Intel’s open-sourced federated learning framework across data from different hospitals to train an AI model for tumor segmentation from MRI scans of the brain. Title: Deep neural ensembles for improved pulmonary abnormality detection in chest radiographs Speaker: Sivaramakrishnan Rajaraman Cardiopulmonary diseases account for a significant proportion of deaths and disabilities across the world.Chest X-rays are a common diagnostic imaging modality for confirming intra-thoracic cardiopulmonary abnormalities. However, there remains an acute shortage of expert radiologists, particularly in under-resourced settings that results in interpretation delays and could have global health impact. These issues can be mitigated by an artificial intelligence (AI) powered computer-aided diagnostic (CADx) system.Such a system could help supplement decision-making and improve throughput while preserving and possibly improving the standard-of-care. A majority of such AI-based diagnostic tools at present use data-driven deep learning (DL) models that perform automated feature extraction and classification. Convolutional neural networks (CNN), a class of DL models, have gained significant research prominence in tasks related to image classification, detection, and localization. The literature reveals that they deliver promising results that scale impressively with an increasing number of training samples and computational resources. However, the techniques may be adversely impacted due to their sensitivity to high variance or fluctuations in training data. Ensemble learning helps mitigate these by combining predictions and blending intelligence from multiple learning algorithms. Complex non-linear functions constructed within ensembles help improve robustness and generalization. Empirical result predictions have demonstrated superiority over the conventional approach with stand-alone CNN models. In this talk, I will describe example work at the NLM that use model ensembles to improve pulmonary abnormality detection in chest radiographs.
Today we're joined by Nikos Athanasiou, Muhammed Kocabas, Ph.D. students, and Michael Black, Director of the Max Planck Institute for Intelligent Systems. We caught up with the group to explore their paper VIBE: Video Inference for Human Body Pose and Shape Estimation, which they submitted to CVPR 2020. In our conversation, we explore the problem that they're trying to solve through an adversarial learning framework, the datasets (AMASS) that they're building upon, the core elements that separate this work from its predecessors in this area of research, and the results they've seen through their experiments and testing.
https://www.youtube.com/watch?v=eyGtJVVmVUM&t=180s Causality and causal modeling is one of the hottest topics in machine learning. Earlier this year we launched a new causality course and study group with instructor Robert Osazuwa Ness, which received great feedback from students: “I liked the course very much. Robert did a great job of reaching out to students to understand their background and interest in the course. It was great how he then continued to use what he learned about the students to make the course relevant and engaging to everyone enrolled. I also like how he made a connection to new paradigms. It was really nice to feel that the course is up to date. There are a lot of machine learning courses but this course was really special.” “I loved the course. I learned a ton and Robert was very available to students. When I think about how much I would have paid at my university for a similar course, TWIML is a great value.” I’m happy to announce that the Fall 2020 course and study group will be starting on September 17th! Check out our recent webinar above to learn more about the course. Then, visit our Causal Modeling in Machine Learning course page to enroll.
A session exploring the possibilities of CheckList, the task-agnostic methodology for testing NLP models introduced in the paper, Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. This session is inspired by the recent TWIML interview with Sameer Singh, co-author of the paper. Read Beyond Accuracy: Behavioral Testing of NLP Models with CheckList in the ACL 2020 proceedings.
Today we're joined by the legendary Michael I. Jordan, Distinguished Professor in the Departments of EECS and Statistics at UC Berkeley. Michael was gracious enough to connect us all the way from Italy after being named IEEE's 2020 John von Neumann Medal recipient. In our conversation with Michael, we explore his career path, and how his influence from other fields like philosophy shaped his path. We spend quite a bit of time discussing his current exploration into the intersection of economics and AI, and how machine learning systems could be used to create value and empowerment across many industries through "markets." We also touch on the potential of "interacting learning systems" at scale, the valuation of data, the commoditization of human knowledge into computational systems, and much, much more.
Today we're joined by Sameer Singh, an assistant professor in the department of computer science at UC Irvine. Sameer's work centers on large-scale and interpretable machine learning applied to information extraction and natural language processing. We caught up with Sameer right after he was awarded the best paper award at ACL 2020 for his work on Beyond Accuracy: Behavioral Testing of NLP Models with CheckList. In our conversation, we explore CheckLists, the task-agnostic methodology for testing NLP models introduced in the paper. We also discuss how well we understand the cause of pitfalls or failure modes in deep learning models, Sameer's thoughts on embodied AI, and his work on the now famous LIME paper, which he co-authored alongside Carlos Guestrin.
This study group works through the fast.ai Practical Deep Learning for Coders course lectures and discusses additional resources for better understanding. This study group is for anyone who is interested in Deep Learning, including beginners! We try to discuss the lectures in as much detail as possible. Experience with working in Python is helpful. US based meetups take place every Tuesday starting September 8th at 5:30pm PT and will run through November 3, 2020. Join the conversation on slack at #fast_ai_practical_dl India based meetups take place every Sunday starting September 13th at 11:30am IST and will run through November 7, 2020. Join the conversation on slack at #fast_ai_practical_dl_ist Resources for the course can be found here. You can also find video recordings at the TWIML community Youtube channel. The playlist is updated as the course progresses.
Today we're joined by Devi Parikh, Associate Professor at the School of Interactive Computing at Georgia Tech, and research scientist at Facebook AI Research (FAIR). While Devi's work is more broadly focused on computer vision applications, we caught up to discuss her presentation on AI and Creativity at the CV for Fashion, Art and Design workshop at CVPR 2020. In our conversation, we touch on Devi's definition of creativity, explore multiple ways that AI could impact the creative process for artists, and help humans become more creative. We investigate tools like casual creator for preference prediction, neuro-symbolic generative art, and visual journaling.
Today we conclude our 2020 ICML coverage joined by Iordanis Kerenidis, Research Director at Centre National de la Recherche Scientifique (CNRS) in Paris, and Head of Quantum Algorithms at QC Ware. Iordanis' research centers around quantum algorithms of machine learning, and was an ICML main conference Keynote speaker on the topic! We focus our conversation on his presentation, exploring the prospects and challenges of quantum machine learning, as well as the field's history, evolution, and future. We'll also discuss the foundations of quantum computing, and some of the challenges to consider for breaking into the field.
Today we continue our ICML series with Elaine Nsoesie, assistant professor at Boston University. Elaine presented a keynote talk at the ML for Global Health workshop at ICML 2020, where she shared her research centered around data-driven epidemiology. In our conversation, we discuss the different ways that machine learning applications can be used to address global health issues, including use cases like infectious disease surveillance via hospital parking lot capacity, and tracking search data for changes in health behavior in African countries. We also discuss COVID-19 epidemiology, focusing on the importance of recognizing how the disease is affecting people of different races, ethnicities, and economic backgrounds.
The use of machine learning in business, government, and other settings that require users to understand the model’s predictions has exploded in recent years. This growth, combined with the increased popularity of opaque ML models like deep learning, has led to the development of a thriving field of model explainability research and practice. In this panel discussion, we bring together experts and researchers to explore the current state of explainability and some of the key emerging ideas shaping the field. Each guest will share their unique perspective and contributions to thinking about model explainability in a practical way. Join us as we explore concepts like stakeholder-driven explainability, adversarial attacks on explainability methods, counterfactual explanations, legal and policy implications, and more. We round out the session with an audience Q&A! Check out the list of resources below! https://www.youtube.com/embed/B2QBnVnbt7A Panelists: Rayid Ghani - Carnegie Mellon University Solon Barokas - Cornell, Microsoft Kush R. Varshney - IBM Alessya Labzhinova - Stealth Hima Lakkaraju - Harvard  Thank you to IBM for their support in helping to make this panel possible! IBM is committed to educating and supporting data scientists, and bringing them together to overcome technical, societal and career challenges. Through the IBM Data Science Community site, which has over 10,000 members, they provide a place for data scientists to collaborate, share knowledge, and support one another. IBM’s Data Science Community site is a great place to connect with other data scientists and to find information and resources to support your career. Join and get a free month of select IBM Programs on Coursera. Resources Rayid Ghani, Carnegie Mellon University - Professor in the Machine Learning Department (in the School of Computer Science) and the Heinz College of Information Systems and Public Policy Topic: Explainability Use Cases in Public Policy and Beyond Twitter: @rayidghani TWIML AI Podcast - #283 - Real World Model Explainability Solon Barocas, Cornell University - Assistant Professor, Department of Information Science, Principal Researcher at Microsoft Research Topic: Hidden Assumptions Behind Counterfactual Explanations Twitter: @s010n TWIML AI Podcast - #219 - Legal and Policy Implications of Model Interpretability: Resources: : The Hidden Assumptions Behind Counterfactual Explanations and Principal Reasons: Published at 2020 ACM Conference on Fairness, Accountability, and Transparency: Shorter version for the 2020 Workshop on Human Interpretability in Machine Learning (WHI) Additional References: Roles for Computing in Social Change. Published at the 2020 ACM Conference on Fairness, Accountability, and Transparency Textbook on Fairness and Machine Learning. Published by MIT Press. The Intuitive Appeal of Explainable Machines Kush R. Varshney, IBM, Distinguished Research Staff Member and Manager at IBM Thomas J. Watson Research Center Topic: Model Explainability as a Communications Challenge Twitter: @krvarshney Resources: IBM AI Fairness 360 IBM AI Explainability 360 IBM Adversarial Robustness 360 IBM AI FactSheets 360 Paper: On Mismatched Detection and Safe, Trustworthy Machine Learning Democast: Mitigating Discrimination and Bias with AI Fairness 360 Alessya Labzhinova, CEO of a stealth startup and former CTO in residence AI2 Topic: Stakeholder-Driven Explainability Resources: Explainable Machine Learning in Deployment, Bhatt et al. You Shouldn’t Trust Me: Learning Models Which Conceal Unfairness From Multiple Explanation Methods, Dimanov et al. Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods, Slack et al. - Causability and eExplainability of Artificial Intelligence in Medicine, Holzinger et al Getting a CLUE: A Method for Explaining Uncertainty Estimates, Antorán et al Hima Lakkaraju, Harvard  University Assistant Professor with appointments in Business School and Department of Computer Science Topic: Adversarial Attacks, Misleading Explanations, and Solutions Twitter: @hima_lakkaraju  TWIML AI Podcast - #387 - AI for High Stakes Decision Making Resources: Presentation Brief Slide Deck The slides also have references to these papers: Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods "How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations Robust and Stable Black Box Explanations
Today we're joined by Hal Daumé III, professor at the University of Maryland, Senior Principal Researcher at Microsoft Research, and Co-Chair of the 2020 ICML Conference. We had the pleasure of catching up with Hal ahead of this year's ICML to discuss his research at the intersection of bias, fairness, NLP, and the effects language has on machine learning models. We explore language in two categories as they appear in machine learning models and systems: (1) How we use language to interact with the world, and (2) how we "do" language. We also discuss ways to better incorporate domain experts into ML system development, and Hal's experience as ICML Co-Chair.
https://www.youtube.com/watch?v=_kaaJKOxAIo&t=142s We're collaborating with machine learning practitioner and instructor Luigi Patruno to bring his course and study group, Building, Deploying, and Monitoring Machine Learning Models with Amazon SageMaker, to the TWIML Community.   If you've been wanting to learn more about one of the hottest Machine Learning products of 2020, please register to join us for our live webinar on Saturday, August 1st, at 9 am PT/12pm ET. After registering, please visit our Building, Deploying, and Monitoring Machine Learning Models with Amazon SageMaker course page to learn more about the course, peruse FAQs, and enroll. Make sure to use the discount code TWIML to get a 10% discount. In addition, if you sign up before 11:59pm PT on August 1st, you will be eligible for an additional early bird discount! A six chapter pre-recorded version of the course with supporting lectures is available here for $199
This meetup has ended. Please contact team@twimlai.com if you’re interested in launching a new iteration of this group. This study group works through David Silver’s course on Reinforcement Learning. The group is for anyone who is interested in Reinforcement Learning, including beginners! Study groups take place every Sunday at 12pm PST respectively. Meetups will run through October 4, 2020. A detailed weekly schedule can be found here. Also, please join us on our TWIML community slack channel: #rl_meetup Resources: Reference book for the course Reinforcement Learning course page Deep Reinforcement Learning Study Group Github
Today we return to our coverage of the 2020 CVPR conference with a conversation with Julian Quiroga, a Computer Vision Team Lead at Genius Sports. Julian presented his recent paper "As Seen on TV: Automatic Basketball Video Production using Gaussian-based Actionness and Game States Recognition" at the CVSports workshop. We jump right into the paper, discussing details like camera setups and angles, detection and localization of the figures on the court (players, refs, and of course, the ball), and the role that deep learning plays in the process. We also break down how this work applies to different sports, and the ways that Julian is looking to improve on this work for better accuracy.
The issue of bias in AI was the subject of much discussion in the AI community last week. The publication of PULSE, a machine learning model by Duke University researchers, sparked a great deal of it. PULSE proposes a new approach to the image super-resolution problem, i.e. generating a faithful higher-resolution version of a low-resolution image. In short, PULSE works by using a novel technique to efficiently search space of high-resolution artificial images generated using a GAN and identify ones that are downscale to the low-resolution image. This is in contrast to previous approaches to solving this problem, which work by incrementally upscaling the low-resolution images and which are typically trained in a supervised manner with low- and high-resolution image pairs. The images identified by PULSE are higher resolution and more realistic than those produced by previous approaches, and without the latter’s characteristic blurring of detailed areas. However, what the community quickly identified was that the PULSE method didn’t work so well on non-white input images. An example using a low res image of President Obama was one of the first to make the rounds, and Robert Ness used a photo of me to create this example: I’m going to skip a recounting of the unfortunate Twitter firestorm that ensued following the model’s release. For that background, Khari Johnson provides a thoughtful recap over at VentureBeat, as does Andrey Kurenkov over at The Gradient. Rather, I’m going to riff a bit on the idea of where bias comes from in AI systems. Specifically, in today’s episode of the podcast featuring my discussion with AI Ethics researcher Deb Raji I note, “I don’t fully get why it’s so important to some people to distinguish between algorithms being biased and data sets being biased.” Bias in AI systems is a complex topic, and the idea that more diverse data sets are the only answer is an oversimplification. Even in the case of image super-resolution, one can imagine an approach based on the same underlying dataset that exhibits behavior that is less biased, such as by adding additional constraints to a loss or search function or otherwise weighing the types of errors we see here more heavily. See AI artist Mario Klingemann’s Twitter thread for his experiments in this direction. Not electing to consider robustness to dataset biases is a decision that the algorithm designer makes. All too often, the “decision” to trade accuracy with regards to a minority subgroup for better overall accuracy is an implicit one, made without sufficient consideration. But what if, as a community, our assessment of an AI system’s performance was expanded to consider notions of bias as a matter of course? Some in the research community choose to abdicate this responsibility, by taking the position that there is no inherent bias in AI algorithms and that it is the responsibility of the engineers who use these algorithms to collect better data. However, as a community, each of us, and especially those with influence, has a responsibility to ensure that technology is created mindfully, with an awareness of its impact. On this note, it’s important to ask the more fundamental question of whether a less biased version of a system like PULSE should even exist, and who might be harmed by its existence. See Meredith Whittaker’s tweet and my conversation with Abeba Birhane on Algorithmic Injustice and Relational Ethics for more on this. A full exploration of the many issues raised by the PULSE model is far beyond the scope of this article, but there are many great resources out there that might be helpful in better understanding these issues and confronting them in our work. First off there are the videos from the tutorial on Fairness Accountability Transparency and Ethics in Computer Vision presented by Timnit Gebru and Emily Denton. CVPR organizers regard this tutorial as “required viewing for us all.” Next, Rachel Thomas has composed a great list of AI ethics resources on the fast.ai blog. Check out her list and let us know what you find most helpful. Finally, there is our very own Ethics, Bias, and AI playlist of TWIML AI Podcast episodes. We’ll be adding my conversation with Deb to it, and it will continue to evolve as we explore these issues via the podcast. I'd love to hear your thoughts on this. (Thanks to Deb Raji for providing feedback and additional resources for this article!)
Causality and causal modeling is one of the hottest topics in machine learning. Earlier this year we launched a new causality course and study group with instructor Robert Osazuwa Ness, which received great feedback from students: "I liked the course very much. Robert did a great job of reaching out to students to understand their background and interest in the course. It was great how he then continued to use what he learned about the students to make the course relevant and engaging to everyone enrolled. I also like how he made a connection to new paradigms. It was really nice to feel that the course is up to date. There are a lot of machine learning courses but this course was really special." "I loved the course. I learned a ton and Robert was very available to students. When I think about how much I would have paid at my university for a similar course, TWIML is a great value." I’m happy to announce that the Summer 2020 course and study group will be starting on June 27th! Register below to join us for a live webinar on Thursday, June 25th, at 10 am PT to learn more. After registering, please visit our Causal Modeling in Machine Learning course page to learn more about the course and peruse FAQs. Robert will introduce the topic of causal modeling in machine learning and review the details of the course, including the many enhancements he’s made this time around.
Thank you for your interest in TWIML's ML Pulse 2020: ML Development, Deployment and Operations Survey. The data collection phase of the survey is now complete, and we are deep in the data analysis process. The resulting report will be available soon, and will cover: The State of the Market: A high-level overview of industry progress. Organizational Process and Platform Maturity: How are organizations meeting the challenge of scaling ML. Drivers and Barriers: What factors are accelerating and inhibiting progress. Tools and Capabilities: How are organizations investing to support data acquisition and management, experimentation and model development, and model deployment and management. Register now to be notified when the report is available.
Today we're joined by Rumman Chowdhury, Managing Director and Global Lead of Responsible Artificial Intelligence at Accenture. In our conversation with Rumman, we explored questions like: Why is now such a critical inflection point in the application of responsible AI? How should engineers and practitioners think about AI ethics and responsible AI? Why is AI ethics inherently personal and how can you define your own personal approach? Is the implementation of AI governance necessarily authoritarian? How do we balance idealism and pragmatism in the application of AI ethics? We also cover practical topics like how and where you should implement responsible AI in your organization, and building the teams and processes capable of taking on critical ethics and governance questions. This was a very fun interview, and for those interested in catching the video, visit our YouTube channel!
In a message last week, I addressed the recent death of George Floyd, the protests, and the future we are working towards. While we all have a responsibility to engage in the fight against racism, the ML/AI community has a unique responsibility to ensure that the technologies we produce are fair and responsible and don’t reinforce racial and socioeconomic biases. We discuss bias, ethics, and fairness in ML and AI frequently on the podcast. We’ve highlighted some of the episodes focused on these topics below. I hope these episodes help you engage in conversations about these issues with your colleagues and friends. We will also be hosting an interactive viewing session of my interview with Rumman Chowdhury, Global Lead of AI Responsibility at Accenture on Monday at 2 PM Pacific. Rumman and I will be live in the chat taking audience questions. Please join us by registering here. We’re looking forward to your questions in the chat. In the meantime, take a look at the shows: AI for Social Good: Why “Good” isn’t Enough with Ben Green - Does political orientation have a place in building technology? Ben comments on the controversial topic and shares how the notion of “good” is often elusive, and lacking in rigorous political or social depth despite enthusiasm from computer scientists to integrate these concepts into their work. The Measure and Mismeasure of Fairness with Sharad Goel - Sharad shares how machine learning can be used to expose unregulated police behavior, and why mathematical definitions are not sufficient for determining bias in algorithms. We also discuss The Stanford Open Policing Project, a data-gathering and analysis initiative started by Sharad. Algorithmic Injustices and Relational Ethics with Abeba Birhane - Abeba wants to shift our focus away from one that’s fundamentally technology-first (explainability, transparency) to one that reframes ethical questions from the perspective of the vulnerable communities our technologies put at risk. AI is just the latest in a series of technological disruptions, and as Abeba notes, one with the potential to negatively impact disadvantaged groups in significant ways. Trends in Fairness and AI Ethics with Timnit Gebru - Timnit provides an overview of the ethics and fairness landscape currently surrounding AI. She shares a ton of insights on diversification, and how groups like Black in AI, and WiML are helping make huge strides towards trends in the fairness communities. Operationalizing Responsible AI - This panel from TWIMLcon: AI Platforms features experts discussing the tools, approaches, and methods that have found useful for teams to implement responsible AI practices. Responsible AI in Practice with Sarah Bird - Sarah focuses on bringing machine learning research responsibly into production, as well as work on differential privacy. She walks through Microsoft's interpretability platform, Azure, and discusses the idea of going from "Black-Box models" to "Glass-Box models." The Ethics of AI-Enabled Surveillance with Karen Levy - Karen discusses how rules and technologies interact to regulate behavior, especially the legal, organizational, and social aspects of surveillance and monitoring. And how these data tracking and surveillance methods are often exploited in ways that impact marginalized groups. Fairness in Machine Learning with Hanna Wallach - Hanna shares how lack of interpretability and transparency show up across machine learning. We discuss the role that inadvertent human biases can impact machine learning. Along the way, Hanna points us to a TON of papers and resources to further explore the topic of fairness in ML. We have more coming your way! Subscribe to our newsletter to keep up to date.
Hi everyone! Unless you’re Jared Leto, or a contestant on Big Brother Germany, you’ve undoubtedly seen your world shift dramatically over the last few weeks. For most of us, this is uncharted territory and unfortunately, we’re not quite able to see the finish line just yet. Now, more than ever before, is the time for us all to lean into our various communities. We encourage you to take this opportunity to connect with your fellow human, albeit virtually! And please know that the TWIML Community, which is over 3,500 strong on Slack, is here for you. We welcome you to join us. Join a study group! Work on a kaggle competition with us! Share your thoughts on a paper you’re reading or the latest podcast episode! We’d love to hear what you’re working on and offer our support. For those who might just need a friendly hello, I’m here for that too. If you’re so inclined, just leave a comment below. Please be safe and take care of your health and that of those around you. Peace and love! Sam
Recently, the fine folks at Technology Review released their annual list of “Breakthrough Technologies,” highlighting 10 areas of tech they see having a real impact on the world in the near future. Well, we see your list, and we raise you one of our own! We took a closer look at the topics as they relate to ML and AI, since that’s kind of our thing. While we haven’t specifically touched each of their topics, we’ve definitely come close. Check out our list below. Unhackable Internet. TR posits that the next form of the internet could be based on quantum physics. This would enable “inherently secure communication,” with messages sent over said network being unhackable. While we cover quantum extensively below (#7), cybersecurity is inseparable from the subject at hand, and we’ve covered that a bit on the pod. Hyper-Personalized Medicine. TR has boiled this down to “genetic medicine tailored to a single patient.” Outside of patient-specific medicine, machine learning will affect several other medical use cases, including preventing genetic diseases before they ever occur. Check out these in-depth discussions spanning precision medicine, genomics, tools, and more. Digital Money This segment of the article focuses on the emergence of crypto- and other digital currencies as disrupters to global finance–a space that will eventually impact investments, trading, stock prices, etc. In the episodes below, we explore a few machine learning use cases around financial trading. Anti-aging drugs Slowing the aging process is touted as a potential solution to treat diseases such as cancer, heart disease, and dementia. Fortunately, the fountain of youth might not be the only way. In the interviews below, we explore the potential future of drug discovery. AI-discovered molecules While Tech Review highlighted AI-discovered molecules as a means of drug discovery, that’s not the only use case for these molecular revelations. In the conversations below, our guests describe how they’re using machine learning to identify a variety of new molecules and materials. Really interesting stuff. Satellite mega-constellations This one is a little tricky for us, and it seemed to be for TR as well. Due to the reduced cost of launching a satellite into orbit, we now have the ability to send multi-satellite networks into orbit, but does it come at the cost of astronomy research? If that turns out to be the case, then we’ll no longer be able to explore things like the mapping of dark matter and dark energy with machine learning, like we did in the interviews below. Quantum supremacy Uncrackable equations seem increasingly in reach with the emergence of usable quantum supercomputers. In the conversations below, we cover the gamut of all things quantum⁠—What is it? How is it measured? What could/should we use it for? And of course, why we’re already doing quantum machine learning all wrong. Tiny AI This topic is the most accessible, as an estimated 3.2 billion people across the globe have tiny AI algorithms and chips already in their hands (smartphones). But the next step is disconnecting these devices from the cloud. With help from a few friends at chip manufacturer Qualcomm, amongst others, we explore just how those tiny AI devices are shrinking and eventually deployed in the conversations below. Differential Privacy Differential privacy is an increasingly important topic, especially with the US government beginning to incorporate differential privacy into the way 2020 Census results are reported. TR defines differential privacy as “a mathematical technique that ... measures how much privacy increases when noise is added” to personal data. We talk a bunch about differential privacy on the pod, mostly due to the overwhelming importance of privacy in the transfer of said data. There is a bunch of compelling content in this list, and our interview with Aaron Roth is a great place to start if you’re unfamiliar with the topic. Climate change attribution Climate change is the existential crisis of our time, and with ever-improving tools, we’re able to get a better idea about what impact these changes are having on our weather, and how to prepare. In the conversations below, we explore some of the projects and research dedicated to not only tracking the impact of the damage being done, but also exploring solutions.
Renowned 19th-century biologist Louis Pasteur once said, "Science knows no country, because knowledge belongs to humanity, and is the torch which illuminates the world." Yet, we can learn much about the evolution of science by exploring its local origins. And with the help of AI, we can use this local knowledge to predict its future. Predicting the future of science, particularly physics, is the task that Matteo Chinazzi, an associate research scientist at Northeastern University focused on in his paper Mapping the Physics Research Space: a Machine Learning Approach, along with co-authors including former TWIML AI Podcast guest Bruno Gonçalves. "The idea is essentially to look under the microscope of how science works, meaning for example, how it evolves over time, how collaboration occurs between different scientists, in between different fields. How scientists pick their research problems, how they, for example, move across different institutions, how nations develop expertise in different fields of research and so on." In addition to predicting the trajectory of physics research, Matteo is also active in the computational epidemiology field. His work in that area involves building simulators that can model the spread of diseases like Zika or the seasonal flu at a global scale. Science of Science Matteo's background in economics and his interest in human behavior sparked his desire to explore the "science of science." Physics was the natural starting point since he already worked with many individuals in the field. To build his models, Matteo uses a core data set of papers published in the journals of The American Physical Society. This dataset was chosen in part because of the robustness of its classification scheme, the Physics and Astronomy Classification Scheme (PACS), which provides references to affiliated topics, authors and publications for each of the papers in the archive. PACS also provides a consistent set of keywords for each of the papers. These keywords are used to relate the various physics researchers to one another using an embedding model. In Matteo's case, the model they use is StarSpace, developed by Facebook AI Research. As Matteo puts it, "We are treating each author as a bag of topics, a bag of research fields in which that author has worked. Then we use this bag of topics to infer the embeddings for each specific research sub-area." Having created an embedding that relates the various research topics to one another, Matteo and his co-authors then use it to create what they call the Research Space Network (RSN). The RSN is a "mapping of the research space [created] by essentially looking at the expertise of authors to guide us on what it means for two topics to be similar to each other." Principle of Relatedness One of the main findings from the research so far is what Matteo refers to as a "fingerprint" of the scientific production of cities. The work is based on the idea of The Principle of Relatedness, an economics term that aims to measure the relationship between a nation's overall production, exports, expertise, and trade partners to predict what items the country should export next. In applying this idea to their research, Matteo would look at all of the scientific publications from a city and use the embedding space to measure the level of relatedness, and predict the direction of the city's scientific knowledge. You can use a network to visually show the interactions between different vectors (science topics) and rank the probability that a city will enter a specific field. That ranking becomes your "classifier" and allows you to determine where that field will or will not be developed next. If you were to plot out the topics of existing research in a city, you could see where the "knowledge density" collects, and note where the density is high, to predict the trajectory of research. If a country is in an intermediate stage of development, there's a higher chance of "jumping" to a different space. Focus and Limitations The focus, for now, is to find the best way of creating embeddings for a very specific problem, not for a variety of tasks. For example, there is no weighting of a researcher's volume of work or its relative importance--the associations include anything they've been active in. Likewise, for some analyses, you might want to identify where the scientist is most active and remove any side projects or abandoned subjects. None of these are considered in this paper. Rather, Matteo approaches the problem from the simplest possible scenario, effectively asking "What if we are blind?" "We...get a big pile of papers from an author. We just list all the topics in which he has worked on and train on that." They want to prove that you do not need to perform manual checks and optimizations to get useful results. Performance Metrics Matteo tested the results using a couple of different validations: One approach was to visualize the RSN and regional fingerprints for assessment. This made it easy to see the macro areas where the PACS classification distinguishes the different subfields of physics. This hierarchy was not used at training time and the algorithm was able to determine the right classification. The second method was to measure the predictive power of the algorithm by looking at each city at a given time period and listing the topics where they had a competitive advantage. Then they compared them using a standard metric like an ROC curve to see if the model was performing better than a random model. What's Next? While the goal is to eventually expand and apply these techniques to entire papers (vs just the PACS keywords), having a predetermined taxonomy and hierarchical structure laid out gives them a benchmark to validate their own observations. Scaling this approach to other fields is something they are starting to work on. They've made some progress using the Microsoft Academic Graph which includes all the different fields in science. As of now, they can't replicate the results they get when they apply the algorithm to physics, but the potential for the embedding space can be evolved for tracking things like the semantics of a term over time, or how authors tend to move in this space. There's also the possibility of finding gaps in the science and making connections that the field might not know to make.
The unfortunate reality is that many of the most commonly used machine learning metrics don't account for the complex trade-offs that come with real-world decision making. This is one of the challenges that Sanmi Koyejo has dedicated his research to addressing. Sanmi is an assistant professor at the Department of Computer Science at the University of Illinois where he applies his background in cognitive science, probabilistic modeling, and Bayesian inference to pursue his research which focuses broadly on "adaptive and robust machine learning." Constructing ML Models that Optimize Complex Metrics As an example of the disconnect between simple and complex machine learning metrics, think about an information retrieval problem, like search or document classification. For these types of problems, it's common to use a metric known as the F-measure to assess your model's performance. F-measure is preferred to simpler metrics like accuracy because it produces a more balanced result by looking at the model's precision and recall. Before Sanmi began his research in this area, there wasn't a good understanding of how to build a machine learning system that was specifically good at optimizing F-measure. Sanmi and his collaborators explored this area through a series of papers including Online Classification with Complex Metrics on making models that optimize complex, non-decomposable metrics. (Non-decomposable here means you can't write the metric as an average, which would allow you to apply existing tools like gradient descent.) Scaling up to More Complex Measures To generalize this idea beyond simple binary classifiers, we have to think about the confusion matrix, which is a key statistical tool used in assessing classifiers. The confusion matrix measures the distribution of predictions that a classifier makes given an input with a certain label. Sanmi's research provided guidance for building models that optimized arbitrary metrics based on the confusion matrix. "Initially we work[ed out] linear weighted combinations. Eventually, we got to ratios of linear things, which captures things like F-measure. Now we're at the point where we can pretty much do any function of the confusion matrix." Domain Experts and Metric Elicitation Having developed a framework for optimizing classifiers against complex performance metrics, the next question Sanmi asked (because it was the next question asked of him), is which one should you choose for a particular problem? This is where metric elicitation comes in. The idea is to flip the question around and try to determine good metrics for a particular problem by interacting with experts or users to determine which of the metrics we can now optimize for best approximate how the experts are making trade-offs against various types of predictions or classification errors. For example, a doctor understands the costs associated with diagnosing or misdiagnosing someone with a disease. The trade-off factors could include treatment prices or side effects--factors that can be compressed to the pros/cons of predicting a diagnosis or not. Building a trade-off function for these decisions is difficult. Metric elicitation allows us to identify the preferences of doctors through a series of interactions with them, and to identify the trade-offs that should correspond to their preferences." Once we know these trade-offs, we can build a metric that captures them, which allows you to optimize those preferences directly in your models using the techniques Sanmi developed earlier. In research developed with Gaurush Hiranandani and other colleagues at the University of Illinois, Performance Metric Elicitation from Pairwise Classifier Comparisons proposes a system of asking experts to rank pairs of preferences, kind of like an eye exam for machine learning metrics. Metric Elicitation and Inverse Reinforcement Learning Sanmi notes that learning metrics in this manner is similar to inverse reinforcement learning, where reward functions are being learned, often by interaction with humans. However, the fields differ in that RL is more focused on replicating behavior rather than getting the reward function correct. Metric elicitation, on the other hand, is focused on replicating the same decision-making reward function as the human expert. Matching the model's reward function, as opposed to the model's behavior, has the benefit of greater generalizability, which allows metrics that are agnostic to data distribution or the specific learner you're using. Sanmi mentions another interesting area of application around fairness and bias, where you have different measures of fairness that correspond to different notions of trade-offs. Upcoming research is focused on finding "elicitation procedures that build context-specific notions of metrics or statistics" that should be normalized across groups to reach a fairness goal in a specific setting. Robust Distributed Learning This interview also covers Sanmi's research into robust distributed learning, which aims to harden distributed machine learning systems against adversarial attacks. Be sure to check out the full interview for the interesting discussion Sam and Sanmi had on both metric elicitation and robust distributed learning. The latter discussion starts about 33 minutes into the interview.
Kamran Khan, founder of BlueDot, recently found his company the subject of attention for being among the first to publicly warn about the coronavirus (COVID-19) that initially appeared in the Chinese city of Wuhan. How did the company's system of data gathering techniques and algorithms help flag the potential dangers of the disease? In this interview, Kamran shares how they use a variety of machine learning techniques to track, analyze and predict infectious disease outbreaks. As a practicing physician based in Toronto, Kamran was directly impacted by the SARS outbreak in 2003. "We saw our hospitals completely overwhelmed. They went into lockdown. All elective procedures were canceled...even the city took on a different feel...there were billions of financial losses...and Toronto was just one of dozens." In the wake of that crisis, governments have been slow to act. Efforts like the International Health Regulations Treaty (2005), which aims to standardize communication about diseases, help but are not well enforced. It doesn't help that these nations are often unaware of the severity of an outbreak, or are hesitant to report a threat because of potential economic consequences. Ultimately, his experience with the SARS crisis led Kamran to explore the role technology might play in anticipating outbreaks and predicting how they might spread. Kamran's insight ultimately lead to the creation of BlueDot, which applies machine learning to four main challenges in infectious disease tracking: Surveillance, Dispersion, Impact, and Communication. Surveillance The BlueDot engine gathers data on over 150 diseases and syndromes around the world, looking at over 100,000 online articles each day spanning 65 languages, searching every 15 minutes, 24 hours a day. This includes official data from organizations like the Center for Disease Control or the World Health Organization, but also counts on less structured, local information from journalists and healthcare workers. BlueDot's epidemiologists and physicians manually classified the data and developed a taxonomy so relevant keywords could be scanned efficiently. They later applied ML and NLP to train the system. Kamran points out that the algorithms in place perform "relatively low-complexity tasks, but they're incredibly high volume and there's an enormous amount of them, so we can simply train a machine to replicate our judgment [for classifying]". As a result of their system's algorithms, only a handful of cases are flagged for human experts to analyze. In the case of COVOID-19, the system highlighted articles in Chinese that reported 27 pneumonia cases associated with a market that had seafood and live animals in Wuhan. Dispersion Recognizing the role that travel plays in disease dispersion—especially in the age of air travel—BlueDot uses geographic information system (GIS) data and flight ticket sales to create a dispersion graph for each disease based on the airports connected to a city and where passengers are likely to fly. Not everyone travels by air, so they also use anonymized location data from 400 million mobile devices to track flows from outbreak epicenters to other parts of the region or world. The locations receiving the highest volume of travelers are identified and diligently evaluated for what the impact of the disease could be in the area. For COVOID-19, BlueDot applied this methodology to identify many of the cities among the first to receive the coronavirus, including Tokyo, Bangkok, Hong Kong, Seoul, and Taipei. Impact Once a virus leaves its region of origin, a wide variety of factors determine whether it will ultimately die out or grow into a full-fledged outbreak: A region may have better or worse public health infrastructure, hospitable or inhospitable climates, or varying economic resources. BlueData's systems consider factors such as these to predict the potential impact on an identified area. For example, if a virus is being spread by ticks, and Vancouver is in the middle of winter snow, the likelihood of an outbreak is very low because ticks would not survive that climate. However, the same virus might thrive in a humid environment like Florida, making the region at-risk for an outbreak. Communication If an area is determined to be at-risk, the focus shifts to providing early warnings to health officials, hospitals, airlines, and government agencies in public health, national defense, national security and even agriculture. Kamran reiterates the importance of providing only the most relevant information to those who need it, referencing the ideas Clay Shirky and his 2008 talk], "It's Not Information Overload. It's Filter Failure. BlueDot first became aware of the pneumonia cases in Wuhan on December 31st, and in addition to notifying their clients and government stakeholders directly, they publicly released their findings in the Journal of Travel Medicine on January 14th. Criticism and Limitations These are incredibly difficult predictions to make, and the science behind the transmission of infectious diseases is complex and evolving every day. So, what is the proper role of technology? Kamran asserts that "by no means would [they] claim that AI has got this problem solved. It's just one of the tools in the toolbox." In some cases, Kamran and his team may lack sufficient observations to develop a machine learning model for a particular disease. For this and other reasons, the company relies on a combination of approaches and a diverse team of specialists in their work. With coronavirus already in full swing, BlueDot is looking more heavily at analyzing location data from mobile devices to provide a real-time understanding of how people are moving around. However, Kamran compares this to predicting the weather—the further ahead you're looking, the less accurate your prediction. Despite the limitations, Kamran reinforces the value of the work by acknowledging that "Manually, it would take a hundred people around the clock [to process the data], and we have four people and a machine."
We're excited to share our third annual Black in AI series! When you're done with this year's series, make sure you check out the previous BAI series: Black in AI 2019 - Black in AI 2018.
How does LinkedIn allow its data scientists to access aggregate user data for exploratory analytics while maintaining its users' privacy? That was the question at the heart of our recent conversation with Ryan Rogers, a senior software engineer in data science at the company. The answer, it turns out, is through differential privacy, a topic we've covered here on the show quite extensively over the years. Differential privacy is a system for publicly sharing information about a dataset by describing patterns of groups within the dataset, the catch is you have to do this without revealing information about individuals in the dataset (privacy). Ryan currently applies differential privacy at LinkedIn, but he has worked in the field, and on the related topic of federated learning, for quite some time. He was introduced to the subject as a PhD student at the University of Pennsylvania, where he worked closely with Aaron Roth, who we had the pleasure of interviewing back in 2018. Ryan later worked at Apple, where he focused on the local model of differential privacy, meaning differential privacy is performed on individual users' local devices before being collected for analysis. (Apple uses this, for example, to better understand our favorite emojis 🤯 👍👏). Not surprisingly, they do things a bit differently at LinkedIn. They utilize a central model, where the user's actual data is stored in a central database, with differential privacy applied before the data is made available for analysis. (Another interesting use case that Ryan mentioned in the interview: the U.S. Census Bureau has announced plans to publish 2020 census data using differential privacy.) Ryan recently put together a research paper with his LinkedIn colleague, David Durfee, that they presented as a spotlight talk at NeurIPS in Vancouver. The title of the paper is a bit daunting, but we break it down in the interview. You can check out the paper here: Practical Differentially Private Top-k Selection with Pay-what-you-get Composition. There are two major components to the paper. First, they wanted to offer practical algorithms that you can layer on top of existing systems to achieve differential privacy for a very common type of query: the "Top-k" query, which means helping answer questions like "what are the top 10 articles that members are engaging with across LinkedIn?" Secondly, because privacy is reduced when users are allowed to make multiple queries of a differentially private system, Ryan's team developed an innovative way to ensure that their systems accurately account for the information the system returns to users over the course of a session. It's called Pay-what-you-get Composition. One of the big innovations of the paper is discovering the connection between a common algorithm for implementing differential privacy, the exponential mechanism, and Gumbel noise, which is commonly used in machine learning. One of the really nice connections that we made in our paper was that actually the exponential mechanism can be implemented by adding something called Gumbel noise, rather than Laplace noise. Gumbel noise actually pops up in machine learning. It's something that you would do to report the category that has the highest weight, [using what is] called the Gumbel Max Noise Trick. It turned out that we could use that with the exponential mechanism to get a differentially private algorithm. [...] Typically, to solve top-k, you would use the exponential mechanism k different times⁠ —you can now do this in one shot by just adding Gumbel noise to [existing algorithms] and report the k values that are in the the top […]which made it a lot more efficient and practical. When asked what he was most excited about for the future of differential privacy Ryan cited the progress in open source projects. This is the future of private data analytics. It's really important to be transparent with how you're doing things, otherwise if you're just touting that you're private and you're not revealing what it is, then is it really private? He pointed out the open-source collaboration between Microsoft and Harvard's Institute for Quantitative Social Sciences. The project aims to create an open-source platform that allows researchers to share datasets containing personal information while preserving the privacy of individuals. Ryan expects such efforts to bring more people to the field, encouraging applications of differential privacy that work in practice and at scale. Listen to the interview with Ryan to get the full scope! And if you want to go deeper into differential privacy check out our series of interviews on the topic from 2018. Thanks to LinkedIn for sponsoring today's show! LinkedIn Engineering solves complex problems at scale to create economic opportunity for every member of the global workforce. AI and ML are integral aspects of almost every product the company builds for its members and customers. LinkedIn's highly structured dataset gives their data scientists and researchers the ability to conduct applied research to improve member experiences. To learn more about the work of LinkedIn Engineering, please visit engineering.linkedin.com/blog.
Hi Everyone, Happy New Year! TL;DR: I’m excited to announce a couple of new study group offerings in conjunction with our ever-expanding array of TWIML Community programs. One is focused on causality and the other on enterprise AI workflows. We’re collaborating with research scientist and instructor Robert Ness to bring his course sequence, Causal Modeling in Machine Learning, to the TWIML Community. Causality has become a very hot topic in the ML/AI space. In fact, it’s come up in a good number of my recent conversations, like this one with Zach Lipton. One of the challenges facing those interested in learning about causality in ML is that most resources on the topic are geared towards the needs of statisticians or economists, versus those of data scientists and machine learning engineers. Robert, a machine learning research scientist at ML startup Gamalon and an instructor at Northeastern University, designed his new causality courses to be more practical and accessible for data scientists and engineers. The first course in sequence is called Model-Based Thinking in Machine Learning and aims to help students develop universal mental model for data science and ML problem-solving while leaving them with a high-level understanding of causality in the context of ML. Robert is simultaneously teaching these courses at the graduate level at Northeastern University and through his virtual AltDeep School of AI, and has agreed to host a study group for the course for the TWIML Community as well. The study group will meet at 8 am US Pacific Time on Saturdays. To get things kicked off, Robert and I will host an overview session on the course on Saturday, February 1st at that time. ML and AI Platforms, and more broadly, strategies for developing and deploying machine learning and deep learning models in the enterprise, is another hot topic in our community and the broader industry. Heck, we held a whole conference on the topic. Until now, there have been few formal courses for learning how to deploy real-world AI and ML workflows in the enterprise. IBM is has addressed this gap with the new IBM AI Enterprise Workflow Specialization it has recently published on Coursera. The specialization consists of six courses which aim to progressively walk the learner through the experience of building and deploying a real-world enterprise AI solution, from establishing business priorities and a data pipeline through to deploying and managing your model in production. I’ll personally be taking this six course sequence and hosting a study group for those of you who would like to join me. If you're doing, interested, or would like to learn more about how to do real-world machine learning in an enterprise environment, I’d encourage you to join me in taking this sequence. I’ll be hosting an informational session on the course and a study group in early February. To join either of these study group or sign up for the respective overview sessions, first join the TWIML Community. Then, after joining our Slack via the invitation you're receive via email, join the #ai_enterprise_workflow or #causality_course channels and say hi! Cheers, Sam
This course works through the Stanford CS224 Winter 2019 course lectures, and discusses additional resources for better understanding. This study group is for anyone who is interested in NLP, including beginners! We try to discuss the lectures in as much detail as possible. Experience with working in Python and Neural Networks is helpful. Meetups take place every Saturday at 8am PT and will run through 11/21/2020. A detailed weekly schedule can be found here. Resources for the course can be found here: Course home page, Winter 2019 Video Playlist Final Project Reports, Winter 2019 You can also find video recordings of the study group sessions here. The playlist is updated as the course progresses.
Sam Charrington: Hey, what's up everyone? This is Sam. A quick reminder that we've got a bunch of newly formed or forming study groups, including groups focused on Kaggle competitions and the fast.ai NLP and Deep Learning for Coders part one courses. It's not too late to join us, which you can do by visiting twimlai.com/community. Also, this week I'm at re:Invent and next week I'll be at NeurIPS. If you're at either event, please reach out. I'd love to connect. All right. This week on the podcast, I'm excited to share a series of shows recorded in Orlando during the Microsoft Ignite conference. Before we jump in, I'd like to thank Microsoft for their support of the show and their sponsorship of this series. Thanks to decades of breakthrough research and technology, Microsoft is making AI real for businesses with Azure AI, a set of services that span vision, speech, language processing, custom machine learning, and more. Millions of developers and data scientists around the world are using Azure AI to build innovative applications and machine learning models for their organizations, including 85% of the Fortune 100. Microsoft customers like Spotify, Lexmark, and Airbus, choose Azure AI because of its proven enterprise grade capabilities and innovations, wide range of developer tools and services and trusted approach. Stay tuned to learn how Microsoft is enabling developers, data scientists and MLOps and DevOps professionals across all skill levels to increase productivity, operationalize models at scale and innovate faster and more responsibly with Azure machine learning. Learn more at aka.ms/azureml. All right, onto the show. Sam Charrington: [00:01:52] All right everyone, I am here in Sunny Orlando, actually it's not all that sunny today, it's kind of gray and gray and rainy but it is still Sunny Orlando, right? How could it not be? At Microsoft Ignite, and I've got the wonderful pleasure of being seated with Sarah Bird. Sarah is a principal program manager for Azure Machine Learning platform. Sarah, welcome to the TWIML AI Podcast. Sarah Bird: [00:02:15] Thank you, I'm excited to be here. Sam Charrington: [00:02:17] Absolutely. I am really excited about this conversation we're about to have on responsible AI. But before we do that, I'd love to hear a little bit more about your background. You've got a very enviable position kind of at the nexus of research and product and tech strategy how did you create that? Sarah Bird: [00:02:37] Well I started my career in research. I did my PhD in machine learning systems at Berkeley and I loved creating the basic technology, but then I wanted to take it to the next step and I wanted to have people who really used it. And I found that when you take research into production, there's a lot more innovation that happens. So since graduating I have styled my career around living at that intersection of research and product, and taking some of the great cutting edge ideas and figuring out how we can get them in the hands of people as soon as possible. And so my role now is specifically focused on trying to do this for Azure Machine Learning and responsible AI is one of the great new areas where there's a ton of innovation and research, and people need it right now. And so we're working to try to make that possible. Sam Charrington: [00:03:33] Oh, that's fantastic. And so between your grad work at Berkeley and Microsoft, what was the path? Sarah Bird: [00:03:42] So I was in John Lankford's group in Microsoft research and was working on a system for contextual bandits and trying to make it easier for people to use those in practice, because a lot of the times when people were trying to deploy that type of algorithm, the system infrastructure would get in the way. You wouldn't be able to get the features to the point of decision or the logging would not work and it would break the algorithm. And so we designed a system that made it correct by construction, so it was easy for people to go and plug it in, and this has actually turned into the Personalizer cognitive service now. But through that experience, I learned a lot about actually working with customers and doing this in production, and so I decided that I wanted to have more of that in my career. And so I spent a year as a technical advisor which is a great role in Microsoft where you work for an executive and advise them and help work on special projects. And it enables you to see both the business and the strategy side of things as well as all the operational things, how you run orgs and then of course the technical things. And I realized that I think that mix is very interesting. And so after that I joined Facebook and my role was at the intersection of FAIR, Facebook AI Research and AML which was the applied machine learning group with this role of specifically trying to take research into production and accelerate the rate of innovation. So I started the Onyx Project as a part of that, enabling us to solve a tooling gap where it was difficult to get models from one framework to another. And then also worked on PyTorch and enabling us to make that more production ready. And since then I've been working in AI ethics. Sam Charrington: [00:05:34] Yeah. If we weren't going to be focused on AI ethics and responsible AI today, we would be going deep into Personalizer, what was Microsoft Decision Service  and this whole contextual bandits thing. Really interesting topic, not the least of which because we talk a lot about reinforcement learning and if it's useful, and while it's not this deep reinforcement learning game playing thing, it's reinforcement learning and people are getting a lot of use out of it in a lot of different contexts. Sarah Bird: [00:06:05] Yeah. When it works, right? It doesn't work in all cases, but when it works, it works really well. It's the kind of thing where you get the numbers back and you're like, can this be true? And so I think it's a really exciting technology going forward and there's a lot of cases where people are using it successfully now, but I think though there'll be a lot more in the future. Sam Charrington: [00:06:25] Awesome. I'll have to take a rain check on that aspect of the conversation and kind of segue over to the responsible AI piece. And I've been thinking a lot about a a tweet that I saw by Rachel Thomas who is a former guest of the podcast, long time friend of the show and currently the UCSF Center for Applied Data Ethics head. And she was kind of lamenting that there are a lot of people out there talking about AI ethics like it's a solved problem. Do you think it's a solved problem? Sarah Bird: [00:06:58] No, absolutely not. I think there are, are fundamentally hard and difficult problems when we have a new technology, and so I think we're always going to be having the AI ethics conversation, this is not something that we're going to solve and go away. But what I do think we have now is a lot more tools and techniques and best practices to help people start the journey of doing things responsibly. And so I think the reality is there are many things people could be doing right now that they're not. And so I, I feel like there's an urgency date to get some of these tools into people's hands so that we can do that. So I `think we can quickly go a lot farther than we have right now. Sam Charrington: [00:07:41] In my conversations with folks that are working on this and thinking about the role that responsible AI plays and the way they "do AI," do machine learning. A lot of people get stopped at the very beginning like: Who should own this? Where does it live? Is it a research kind of function or is it a product function, or is it more of a compliancy thing for a chief data officer or a chief security officer? [Is it] one of those executive functions and oversight, or compliance is the better word? What do you see folks doing and do you have any thoughts on successful patterns of where it should live? Sarah Bird: [00:08:33] Yeah, I think the models that we've been using and are thinking a lot about... the transition  to security, for example. And I think the reality is it's not one person's job or one function. Everybody now has to think about security, even your basic software developers have to know and think about it when they're designing. However, there are people who are experts in it and handle the really challenging problems. There is of course legal and compliance pieces in there as well. And so I think we're seeing the same thing where we really need every role to come together and do this. And so one of the patterns we are seeing is part of the challenge with responsible AI and technology is that we've designed technology to abstract away things and enable you to just focus on your little problem, and this has led to a ton of innovation. However, the whole idea of responsible AI is actually, you need to pick your head up, you need to have this larger context, you need to think about the application in the real world, you need to think about the implications. And so we have to break a little bit of our patterns of 'my problem is just this little box,' and so we're finding that user research and design, for example, is already trained and equipped to think about the people element in that. And so it's really great to bring them into more conversations as we're developing the technology. So that's one pattern that we're finding adds a lot  of value. Sam Charrington: [00:10:07] In my conversation with with Jordan Edwards, your colleague, many of his answers were all of the above. And it sounds like this one is an "all of the above" response as well. Sarah Bird: [00:10:19] Yeah. I think doing machine learning in practice takes a lot of different roles, as Jordan was talking about, in operationalizing things, and then responsible AI just adds an extra layer of more roles on top of that. Sam Charrington: [00:10:32] Yeah. I guess one of the challenges that kind of naturally evolves when everyone has to be thinking about something is that it's a lot  harder, right? The developer is trained as a developer and now they have to start thinking about this security thing, and it's changing so quickly and the best practices are evolving all the time, and it's hard to stay on top of that. If we're to replicate that same kind of model in responsible AI, what sounds like the right thing to do? How do we support the people that are on the ground trying to do this? Sarah Bird: [00:11:07] Yeah. And I think it's definitely a challenge because the end result can't be that every individual person has to know the state of the art in every area in responsible AI. And so one of the ways that we're trying to do this is, as much as possible, build it into our processes and our tooling. So that you can say, okay, well you should have a fairness metric for your model and you can talk to experts about what that fairness metric should be, but you should know the requirement that you should have a fairness metric, for example. And so we first are starting with that process layer and then in Azure Machine Learning, we've built tools that enable you to easily enact that process. And so the foundational piece is the MLOps story that Jordan was talking about where we actually enable you to have a process that's reproducible, that's repeatable. So you can say, before this model goes into production, I know that it's passed these validation tests and I know that a human looked at it and said, it looks good. And if it's out in production and there's an error or there's some sort of issue that arises, you can go back, you can recreate that model, you can debug the error. And so that's the real foundational piece for all of it. And then on top of that, we're trying to give data scientists more tools to analyze the models themselves. And there's no magic button here. It's not just, Oh, we can run a test and we can tell you everything you want to know. But there's lots of great algorithms out there and research that help you better understand your model. Like SHAP or LIME are common interpretability ones. And so we've created a toolkit called Interpret ML, this is an open source toolkit you can use it anywhere. But it enables you to easily use a variety of these algorithms to explain your model behavior and explore it and see if there are any issues. And so we've also built that into our machine learning process so that if I build a model, I can easily generate explanations for that model. And when I've deployed it in production, I can also deploy and explain her with it so individual predictions can be explained while it's running so I can understand if I think it's doing the right thing and if I want to trust it, for example. Sam Charrington: [00:13:35] It strikes me that there's a bit of a catch 22 here, in the sense that the only way we could possibly do this is by putting tools in the hands of the folks that are working data scientists and machine learning engineers that are working on these problems. But the tools in their very nature kind of abstract them away from the problem and allow them, if not, encourage them to think less deeply about what's going on underneath. Right? How do we address that? Do you agree with that first of all? Sarah Bird: [00:14:09] No, I completely agree with that and it's a challenge that we have in all of these cases where we want to give the tool to help them and to have more insight but it's easy for people to just use it as a shortcut. And so in a lot of cases, we're being very thoughtful about the design of the tool and making sure that it is helping you surface insights. But it's not saying this is the answer because I think when you start doing that where you have something that flags and says this is a problem, then people really start relying on that. And maybe someday we will have the techniques where we have that level of confidence and we can do it. But right now we really don't, and so I think a lot of it is making sure that we designed the tools that encourages this mindset of exploration and deeper understanding of your models and what's going on. And not just, Oh, this is just another compliance tests I have to pass I just run this test and it says green. And I go. Sam Charrington: [00:15:12] You alluded to this earlier in the conversation, but it seems appropriate here as well, and it's maybe a bit of a tangent, but so much of pulling all these pieces together is kind of a user experience and design. Any thoughts on that? Is that something that you've kind of dug into and studied a lot? Or are the other folks worry about that here? Sarah Bird: [00:15:36] It's not in my background, but to me it's an essential part of the function of actually making these technologies usable. And particularly when you take something that as complex as an algorithm and you're trying to make that abstracted and usable for people, the design is a huge part of the story. And so what we're finding in responsible AI is that we need to think about this even more. And a lot of the guidelines are saying be more thoughtful and include sort of more careful design. For example, people are very tempted to say, well, this is the data I have so this is the model I can build and so I'm going to put it in my application that way. And then if it has too much inaccuracy, then you spend a lot of resources to try and make the model more accurate where you could have just had a more elegant UI design, for example, where you actually get better feedback based on the UI design or the design can tolerate more errors and you don't need that higher model accuracy. So we're really encouraging people to co-design the application in the model and not just take it for granted that this is what the model does and that's the thing we're gonna focus on. Sam Charrington: [00:16:53] With the Interpret ML tool, what's the user experience like? Sarah Bird: [00:17:01] It depends on what you're trying to do, there's two types of interpretability that people think about. One is what we call Glass-Box models. And the idea there is I want my model to be inherently interpretable. So I'm gonna pick something like a linear model or decision trees where I can actually inspect the model and enable you to to build a model of that, that you can actually understand. And so we support a bunch of different Glass-Box explainer or models. So then you can actually use it to train your own model. And the other part is Black-Box explainers where I have a model that I is a black box and I can't actually inspect it, but I can use these different algorithms to explain the behavior of the model. And so in that case what we've done is made it easy for you to just call and explain and ask for global explanations and ask for local explanations and ask for feature importance. And then all of those are brought together in an interactive dashboard where you can actually explore the explanations and try to understand the model behavior. So a lot of the experience it's an SDK and so it's all easy calls to ask for explanations, but then we expect a lot of people to spend their time in that dashboard exploring and understanding. Sam Charrington: [00:18:32] I did a really interesting interview with Cynthia Rudin who you may know she's a Duke professor and the interview was focused on her research that essentially says that we should not be using black box models in, I forget the terminology that she used, but something like mission critical scenarios or something along those lines where we're talking about someone's life or Liberty that kind of thing. Does providing interpretability tools that work with black box models, like encourage their use in scenarios that they shouldn't really be used in? And are there ways that you advise folks when and when not they should be using those types of models? Sarah Bird: [00:19:19] So we have people who do publish best practices for interpretability and  it's a very active area of work for the company. And we work with the partnership on AI to try to make industry-wide recommendations for that. I don't think it's completely decided on this idea that models should be interpretable in these settings versus, well, we want other mechanisms to make sure that they're doing the right thing. Interpretability is one way that we could be sure that they're doing the right thing, but we also could have more robust testing regimes. Right? There's a lot of technologies where we don't understand every detail of the technology, but we've been able to build safety critical systems on top of it, for example. And so yeah as a company we do try to provide guidance, but I don't think the industry has really decided the final word on this. And so the mindset of the toolkit is enabling you to use these techniques if it's right for you. But that doesn't specifically say that you should go use a neural net in a particular setting. Sam Charrington: [00:20:27] So in addition to the Interpret ML toolkit you also announced this week here from Ignite, a Fair Learn toolkit. What's that all about? Sarah Bird: [00:20:39] So it's the same spirit as Interpret ML where we want to bring together a collection of fairness techniques that have been published in research and make it easy for people to use them all in one toolkit with the same spirit that you want to be able to analyze your model and understand how it's working so that you could make decisions around fairness. And so there's famously, many different fairness metrics published. I think there was a paper cataloging 21 different fairness metrics. And so we've built many of these common ones into the toolkit and then it makes it easy for you to compare how well your model works for different groups of people in your data set. So for example, I could say does this model have the same accuracy for men and women? Does this model have the same outcomes for men and women? And so we have an interactive dashboard that allows you to explore these differences between groups and your model performance through a variety of these metrics that have been published in research. Then we've also built in several mitigation techniques so that if you want to do mitigation via post-processing and your model, then you can do that. For example, setting thresholds per group. And in a lot of cases it might be that you actually want to go and fix the underlying data or you wanting to make some different decisions. So the mitigation techniques aren't always what you would want to do, but they're available if you want to do that. And so the name of the toolkit actually comes from one of these mitigation techniques from Microsoft research where the algorithm was originally called Fair Learn. And the idea is that you say, I wanna reduce the difference between two groups on a particular dimension. So you pick the metric and you pick the groups and the algorithm actually retrains your model by re-wading data and iteratively retraining to try to reduce that disparity. So we've built that into the toolkit. So now you can actually look at a variety of your versions of your model and see if one of them has properties that works better for what you're looking for, to deploy. Sam Charrington: [00:22:59] Again, I'm curious about the user experience in, in doing this. How much knob turning and tuning does the user need to do when applying that technique you were describing? Or is it more, I'm envisioning something like contextual bandage reinforcement learning where it's kind of tooling the knobs for you. Sarah Bird: [00:23:18] Yeah, it is doing the knobs and the retraining, but what you have to pick is which metric you're trying to minimize. Do I want to reduce the disparity between the outcomes or do I want to reduce the disparity and accuracy or some other there's many different metrics you could pick, but you have to know the metric that's right for your problem. And then you also need to select the groups that you want to do. So it can work in a single dimension like as we were saying making men and women more more equal, but then it would be a totally separate thing to do it for age, for example. So you have to pick both the sensitive attribute that you are trying to reduce disparity and you have to pick the metric for disparity. Sam Charrington: [00:24:10] Were you saying that you're able to do multiple metrics in parallel or you're doing them serially? Sarah Bird: [00:24:17] Right now the techniques work for one, for just one metric. So it will produce a series of models, and if you look at the graph, you can actually plot disparity by accuracy and you'll have models that are on that Pareto optimal curve to look at. But then if you said, okay, well now I want to look at that same chart for age, the models might be all over the place in the space of disparity and accuracy. So it's not a perfect technique, but there are some settings where it's quite useful. Sam Charrington: [00:24:48] So going back to this idea of abstraction and tools versus deeply understanding the problem domain and how to think about it in the context of your problem domain. I guess the challenge domain or your problem domain, I don't know what the right terms are. But you mentioned that paper with all of the different disparity metrics and the like. Is that the best way for folks to get up to speed on this or are there other resources that you've come across that are useful? Sarah Bird: [00:25:23] Yeah, I think for fairness in particular it's better to start with your application domain and understand, for example, if you're working in an employment setting, how do we think about fairness and what are the cases and so in that case we actually recommend that you talk to domain experts, even your legal department to understand what fairness means in that setting. And then you can go to the academic literature and start saying, okay, well, which metrics line up with that higher level concept of fairness for my setting. But if you start with the metrics I think it can be very overwhelming and there's just many different metrics and a lot of them are quite different and in other ways they're very similar with each other. And so I find it much easier to start with the domain expertise and know what you're trying to achieve in fairness and then start finding the metrics that line up with that. Sam Charrington: [00:26:22] You're also starting to do some work in the differential privacy domain. Tell me a little bit about that. Sarah Bird: [00:26:27] Yeah, we announced a couple of weeks ago that we are building an open source privacy platform with Harvard and differential privacy is a really fascinating technology. It was first published in Microsoft Research in 2006 and it was a very interesting idea, but it has taken a while for it, as an idea, to mature and develop and actually be able to be used in practice. However, now we're seeing several different companies who are using it in production. But in every case the deployment was a very bespoke deployment with experts involved. And so we're trying to make a platform that makes it much easier for people to use these techniques without having to understand them as much. And so the idea is the open source platform can go on top of a data store, enable you to do queries in a differentially private way, which means that actually it adds noise to the results so that you can't reconstruct the underlying data and also then potentially use the same techniques to build simple machine learning models. And so we think this is particularly important for some of our really societaly valuable datasets. For example, there are data sets where people would like to do medical research, but because we're worried about the privacy of individuals, there's limits to what they can actually do. And if we use differential private interface on that, we have a lot more privacy guarantees and so we can unlock a new type of innovation and research in understanding our data. So I think we're really excited and think this could be the future of privacy in certain applications, but the tooling just isn't there, and so we're working on trying to make it easier for people to do that. We're building it in the open source because it's important that people can actually ... It's very easy to get the implementation of these algorithms wrong and so we want the community and the privacy experts to be able to inspect and test the implementations and have the confidence that it's there. And also we think this is such an important problem for the community. We would like anybody who wants to, to be joining in and working on this. This is not something that we can solve on our own. Sam Charrington: [00:28:58] Yeah, differential privacy in general and differentially private machine learning are fascinating topics and ones that we've covered fairly extensively in the podcast. We did a series on differential privacy a couple of years ago maybe and it's continuing to be an interesting topic. At the Census Bureau I think is using differential privacy for the first time next year and it's both providing the anticipated benefits but also raising some interesting concerns about an increased opacity on the part of researchers to the data that they wanna get access to. Are you familiar with that challenge? Sarah Bird: [00:29:41] Yeah, absolutely. So the reality is people always want the most accurate data, right? It doesn't sound great to say, well, we're adding noise and the data is less accurate. But, in a lot of cases it is accurate enough for the tasks that you want to accomplish. And I think we have to recognize that, privacy is one of the sort of, fundamental values that we want to uphold, and so in some cases it's worth the cost. For the census in particular, to motivate the decision to start using this for the 2020 census they did a study where they took the reports from the 1940 census and they were able to recreate something like 40% of Americans' data with the result of just the outputs from the census. Sam Charrington: [00:30:33] Meaning personally identify 40% of Americans? Sarah Bird: [00:30:37] Yeah, he talks about this in his ICML keynote from last year. So if you want to learn more you can watch the keynote. But yeah, basically they took all the reports and they used some of these privacy attacks and they could basically recreate a bunch of the underlying data. And this is a real risk, and so we have to recognize that yes, the census results are incredibly important and they help us make many different decisions, but also protecting people's data is important. And so some of it is education and changing our thinking and some of it is making sure that we use the techniques in the right way in that domain where you're not losing what you were trying to achieve in the first place, but you are adding these privacy benefits. Sam Charrington: [00:31:21] There are a couple of different ways that people have been applying differential privacy one is a, a more centralized way where you're applying it to a data store. It sounds a little bit like that's where your focus is. Others like Apple's a noted use case where they're applying differential privacy in a distributed manner at the handset to keep user data on the iPhone, but still provide information centrally for analysis. Am I correct that your focus is on the centralized use case? Or does the toolkit also support the distributed use case? Sarah Bird: [00:32:02] We are focusing on the global model. The local model works really well, and particularly in some of these user telemetry settings, but it limits what you can do. You need much larger volume to actually get the accuracy for a lot of the queries that you need, and there aren't as many queries that you can do. And so the global model, on the other hand, there's a lot more that you can do and still have reasonable privacy guarantees. And so as I was saying, we were motivated by these cases where we have the data sets. Like somebody is trusted to have the data sets but we can't really use them. And so that looks like a global setting. And so to start, we're focused on, on the global piece, but there are many cases where the local is promising and there are cases where we are doing that in our products. And so it's certainly a direction that things could go. Sam Charrington: [00:32:58] And differential privacy from a data perspective doesn't necessarily get you to differentially private machine learning. Are you doing anything in particular on the differentially private ML side of things? Sarah Bird: [00:33:11] The plan is to do that but the project is pretty new so we haven't built it yet. Sam Charrington: [00:33:19] And before we wrap up, you're involved in a bunch of industry and research initiatives in the space that you've mentioned, MLSys, a bunch of other things. Can you talk a little bit about some of the broader things that you're doing? Sarah Bird: [00:33:38] Yeah, so I helped found the, now I think named MLSys systems and machine learning research conference. And that was specifically because I've been working at this intersection for a while and there were some dark days where it was very hard to publish work because the machine learning community was like, this is a systems result. And the systems community was like, this doesn't seem like a systems result and so we started the conference about two years ago and apparently many other people were feeling the same pain because even from the first conference, we got excellent work. People's top work, which is always a challenge with research conferences because people don't want to submit their best work to an unnamed conference. Right? But there was such a gap for the community. So it's been really exciting  to see that community form more  and now have a home where they can put their work and connect.  I've also been running the machine learning systems workshops at NeurIPS for several years now. And that's been a really fun place because it really has helped us form the community, particularly before we started the conference. But it's also a place where you can explore new ideas. This last year we're starting to see a lot more innovation at the intersection of programming languages and machine learning. And so in the workshop format we can have several of those talks highlighted, and have a dialogue, and show some of the emerging trends so that's been a really fun thing to be involved in. Sam Charrington: [00:35:13] Awesome. Yeah, was it last year that there was both the SysML workshop and the ML for systems workshop and it got really confusing? Sarah Bird: [00:35:24] Yeah. This year too. We have both. And I think that's a sign that the field is growing that it used to be that it felt like we didn't even have enough people for one room at the Intersection of Machine Learning and Systems. And I think this last year there was maybe four or 500 people in our workshop alone. And so that's great. Now, there's definitely room to have workshops on more focused topics. Right? And so I think machine learning for systems is an area that people are really excited about now that we have more depth in understanding the intersection. For me, it's very funny because that is really kind of the flavor of my thesis which was a  while ago. And so it's a fun to see it now starting to become an area that people are excited about. Sam Charrington: [00:36:16] The other conference that we didn't talk about, ML for Systems is all about using machine learning within computational systems, networking systems as a way to optimize them. So for example, ML to do database query optimization. Also a super interesting topic. Sarah Bird: [00:36:36] Yeah, I know it absolutely is. And I really believe in that, and I think for several years people were just trying to replace kind of all of the systems intelligent with one machine learning algorithm and it was not working very well. And I think what we're seeing now is recognizing that a lot of the algorithms that we used to control systems were designed for that way and  they work, actually, pretty well. But on the other hand, there's something that's dynamic about the world or the workload. And so you do want this prediction capability built in. And so a lot of the work now has a more intelligent way of plugging the algorithms into the system. And so now we're starting to see promising results at this intersection. So my thesis work was a resource allocation that built models in real time in the operating system and allocated resources. And it was exactly this piece where there was a modeling and a prediction piece, but, the final resource allocation algorithm was not purely machine learning. Sam Charrington: [00:37:43] Awesome. Wonderful conversation, looking forward to catching up with you at NeurIPS, hopefully. thanks so much for taking the time to chat with us. Sarah Bird: [00:37:52] Yes, thanks for having me. And I look forward to seeing you at NeurIPS. Sam Charrington: [00:37:56] Thank you.
Bits & Bytes Google scraps controversial AI ethics council days after it was announced. Google has shuttered its new artificial intelligence ethics council, a little more than a week after announcing it, in large part due to employee reactions to the appointment of a conservative think tank leader to the group. WIPO launches AI-based image search tool for brands. The World Intellectual Property Organization (WIPO) has launched a new AI-powered image search technology that uses deep learning to identify combinations of concepts within an image, thus making it faster and easier to establish a trademark’s distinctiveness. Tim Apple poaches Ian GANfellow. Apple has poached another of Google’s top AI researchers. This time it’s Ian Goodfellow, best known as the creator of GANs, or generative adversarial networks, who has joined Apple in a director role. FDA Proposes Regulatory Framework for AI- and Machine Learning-Driven SaMD. US Food and Drug Administration (FDA) requested feedback on a new discussion paper that proposes applying a “focused review” approach to premarket assessments of software as a medical device (SaMD) technologies that are powered by AI and ML. [Paper] Qualcomm Reveals “Cloud AI 100” AI Inference Accelerator. Qualcomm announced their first discrete dedicated AI processors, the Qualcomm Cloud AI 100 family. The chip, which it expects to begin producing in 2020, is designed for use in datacenters to meet increasing demand for AI inference processing. Google launches an end-to-end AI platform. At Google Next ’19, Google announced the beta version of its end-to-end AI platform, which aims to offer developers and data scientists a one-stop shop for building, testing and deploying models. At Next, the company made several additional announcements as well, including updates to its suite of Cloud AI Solutions, AutoML, BigQuery ML, its pre-trained ML APIs, and more. Petuum Unveils Industrial AI Product for Complex Manufacturing Operations. Petuum announced the Petuum Industrial AI Autopilot product, which enables optimization of complex manufacturing operations with modules that continuously learn and adapt. Dollars & Sense Intel Capital announced investments in five AI companies at its annual Global Summit. IntelCapital led a $13M round in AI chip startup Untether AI and a $150M round in AI systems company SambaNova Systems. In addition, the company invested undisclosed amounts in Andrew Ng’s Landing AI and China-based CloudPick and Zhuhai Eeasy Technology. Run.AI, a startup building a new virtualization and acceleration platform for deep learning, announced that it has raised $13M Enlitic, a San Francisco, CA-based company leveraging AI for medical imaging raised $15M in Series B financing Boston Dynamics announced that it is acquiring California-based startup Kinema Systems, which makes computer-vision and ML systems for warehouse robots Evolve IP, announced that it has acquired Jog.ai, a speech analytics and natural language technology firm based in Austin, Texas Rasa, San Francisco-based an open source company that enables developers to build contextual AI assistants, has secured $13M in Series A funding Labelbox, a collaborative training data platform for machine learning applications, has raised a $10M Series A funding Observe.AI has secured $8M in a Series A funding To receive the Bits & Bytes to your inbox, subscribe to our Newsletter.
Bits & Bytes Google introduces Feast: an open source feature store for ML. GO-JEK and Google announced the release of Feast that allows teams to manage, store, and discover features to use for ML projects. Amazon CEO, Jeff Bezos, is launching a new conference dedicated to AI. The new AI specific conference, re:MARS, will be held in Las Vegas between June 4th and 7th this year. Should be an interesting event. Mayo Clinic research uses AI for early detection of silent heart disease. Mayo Clinic study finds that applying AI to electrocardiogram (EKG) test results offers a simple, affordable early indicator of asymptomatic left ventricular dysfunction, a precursor to heart failure. Microsoft announces ML.NET 0.9. Microsoft’s open-source and cross-platform ML framework, ML.NET, was updated to version 0.9. New and updated features focus on expanded model interpretability capabilities, GPU support for ONNX models, new Visual Studio project templates in preview, and more. Intel and Alibaba team up on new AI-powered 3D athlete tracking technology. At CES 2019, Intel and Alibaba announced the new collaboration to develop AI-powered 3D athlete tracking technology to be deployed at the 2020 Olympic Games. Baidu unveils open source edge computing platform and AI boards. OpenEdge, an open source computing platform enables developers to build edge applications with more flexibility. The company also announced new AI hardware development platforms BIE-AI-Box with Intel for in-car video analysis, and BIE-AI-Board, co-developed with NXP, for object classification. Qualcomm shows off an AI-equipped car cockpit at CES 2019. At CES, Qualcomm introduced the third generation of its Snapdragon Automotive Cockpit Platforms. The upgraded version covers various aspects of the in-car experience from voice-activated interfaces to traditional navigation systems. Their keynote featured a nice demo of “pedestrian intent prediction” based on various computer vision techniques including object detection and pose estimation. Dollars & Sense Fractal Analytics, an AI firm based in India which, among other things, owns Qure.ai (see my interview with CEO Prashant Warier), raised $200M from private equity investor Apax Standard has acquired Explorer.ai, a mapping and computer vision start-up Israeli AI-based object recognition company, AnyVision, has raised $15M from Lightspeed Venture Partners Spell, an NYC-based AI and ML platform startup raised $15M HyperScience, an edge ML company has raised $30M WeRide.ai, a Chinese autonomous driving technology specialist raised series A funding from SenseTime Technology and ABC International in series A funding UK-based Exscientia, AI-driven drug discovery company, has raised $26 million CrowdAnalytix raises $40 million in strategic investment for crowdsourced AI algorithms To receive the Bits & Bytes to your inbox, subscribe to our Newsletter.