We could not locate the page you were looking for.

Below we have generated a list of search results based on the page you were trying to reach.

404 Error
Today we kick off our annual coverage of the CVPR conference joined by Fatih Porikli, Senior Director of Engineering at Qualcomm AI Research. In our conversation with Fatih, we explore a trio of CVPR-accepted papers, as well as a pair of upcoming workshops at the event. The first paper, Panoptic, Instance and Semantic Relations: A Relational Context Encoder to Enhance Panoptic Segmentation, presents a novel framework to integrate semantic and instance contexts for panoptic segmentation. Next up, we discuss Imposing Consistency for Optical Flow Estimation, a paper that introduces novel and effective consistency strategies for optical flow estimation. The final paper we discuss is IRISformer: Dense Vision Transformers for Single-Image Inverse Rendering in Indoor Scenes, which proposes a transformer architecture to simultaneously estimate depths, normals, spatially-varying albedo, roughness, and lighting from a single image of an indoor scene. For each paper, we explore the motivations and challenges and get concrete examples to demonstrate each problem and solution presented.
Dr. Torbjörn Nordling obtained both his Ph.D. in Automatic Control (2013) and his M.Sc. in Engineering Physics (2005) from KTH Royal Institute of Technology in Stockholm, Sweden. He joined the Department of Mechanical Engineering at the National Cheng Kung University in Taiwan as an Assistant Professor in September 2015 and the Department of Applied Physics and Electronics at Umeå University in Sweden as Senior Lecturer in January 2020. Previously, he completed a PostDoc at the Dept. of Immunology, Genetics and Pathology at Uppsala University in Sweden. He has been a visiting researcher at Telethon Institute of Genetics and Medicine in Naples, Italy and ERATO Kitano Symbiotic Systems Project at Japan Science and Technology Agency in Tokyo, Japan. He is the founder of Nordron AB, a startup specialised in data analysis, and a co-founder of Jagah Systems AB, an award winning indoor geolocalisation startup. He has co-authored more than 25 peer-reviewed full-length articles published in international journals (SCI IF 1.5-25) or conference proceedings and given more than 60 invited lectures, oral conference, or poster presentations. He has specialised in Data Science, Mathematical modeling, System identification, Machine learning, and Artificial Intelligence with applications in Biology and Medicine.
There are few things I love more than cuddling up with an exciting new book. There are always more things I want to learn than time I have in the day, and I think books are such a fun, long-form way of engaging (one where I won’t be tempted to check Twitter partway through). This book roundup is a selection from the last few years of TWIML guests, counting only the ones related to ML/AI published in the past 10 years. We hope that some of their insights are useful to you! If you liked their book or want to hear more about them before taking the leap into longform writing, check out the accompanying podcast episode (linked on the guest’s name). (Note: These links are affiliate links, which means that ordering through them helps support our show!) Adversarial ML Generative Adversarial Learning: Architectures and Applications (2022), Jürgen Schmidhuber AI Ethics Sex, Race, and Robots: How to Be Human in the Age of AI (2019), Ayanna Howard Ethics and Data Science (2018), Hilary Mason AI Sci-Fi AI 2041: Ten Visions for Our Future (2021), Kai-Fu Lee AI Analysis AI Superpowers: China, Silicon Valley, And The New World Order (2018), Kai-Fu Lee Rebooting AI: Building Artificial Intelligence We Can Trust (2019), Gary Marcus Artificial Unintelligence: How Computers Misunderstand the World (The MIT Press) (2019), Meredith Broussard Complexity: A Guided Tour (2011), Melanie Mitchell Artificial Intelligence: A Guide for Thinking Humans (2019), Melanie Mitchell Career Insights My Journey into AI (2018), Kai-Fu Lee Build a Career in Data Science (2020), Jacqueline Nolis Computational Neuroscience The Computational Brain (2016), Terrence Sejnowski Computer Vision Large-Scale Visual Geo-Localization (Advances in Computer Vision and Pattern Recognition) (2016), Amir Zamir Image Understanding using Sparse Representations (2014), Pavan Turaga Visual Attributes (Advances in Computer Vision and Pattern Recognition) (2017), Devi Parikh Crowdsourcing in Computer Vision (Foundations and Trends(r) in Computer Graphics and Vision) (2016), Adriana Kovashka Riemannian Computing in Computer Vision (2015), Pavan Turaga Databases Machine Knowledge: Creation and Curation of Comprehensive Knowledge Bases (2021), Xin Luna Dong Big Data Integration (Synthesis Lectures on Data Management) (2015), Xin Luna Dong Deep Learning The Deep Learning Revolution (2016), Terrence Sejnowski Dive into Deep Learning (2021), Zachary Lipton Introduction to Machine Learning A Course in Machine Learning (2020), Hal Daume III Approaching (Almost) Any Machine Learning Problem (2020), Abhishek Thakur Building Machine Learning Powered Applications: Going from Idea to Product (2020), Emmanuel Ameisen ML Organization Data Driven (2015), Hilary Mason The AI Organization: Learn from Real Companies and Microsoft’s Journey How to Redefine Your Organization with AI (2019), David Carmona MLOps Effective Data Science Infrastructure: How to make data scientists productive (2022), Ville Tuulos Model Specifics An Introduction to Variational Autoencoders (Foundations and Trends(r) in Machine Learning) (2019), Max Welling NLP Linguistic Fundamentals for Natural Language Processing II: 100 Essentials from Semantics and Pragmatics (2013), Emily M. Bender Robotics What to Expect When You’re Expecting Robots (2021), Julie Shah The New Breed: What Our History with Animals Reveals about Our Future with Robots (2021), Kate Darling Software How To Kernel-based Approximation Methods Using Matlab (2015), Michael McCourt
Checkout our recent Causal Modeling in ML webinar with Robert Ness here We’re collaborating with research scientist and instructor Robert Ness to bring his course sequence, Causal Modeling in Machine Learning, to the TWIML Community. Causality has become a very hot topic in the ML/AI space. In fact, it’s come up in a good number of my recent conversations, like this one with Zach Lipton. One of the challenges facing those interested in learning about causality in ML is that most resources on the topic are geared towards the needs of statisticians or economists, versus those of data scientists and machine learning engineers. Robert, an ML research scientist at startup Gamalon and an instructor at Northeastern University, has developed a series of six course modules on Causal Modeling in Machine Learning that are designed to be more practical and accessible for data scientists and engineers. He is teaching the course live to graduate students at Northeastern University, and through our new partnership Robert will also be hosting a study group via the TWIML platform, i.e. Zoom and Slack. The study group will provide TWIML enrollees some of the benefits of taking the course live. Robert will hold a weekly review session after each week of study in the sequence, will be available to answer questions via Slack, will personally grade submitted assignments, and will be available to assist with course homework and projects. The previous cohort of this course received great feedback from students: "I liked the course very much. Robert did a great job of reaching out to students to understand their background and interest in the course. It was great how he then continued to use what he learned about the students to make the course relevant and engaging to everyone enrolled. I also like how he made a connection to new paradigms. It was really nice to feel that the course is up to date. There are a lot of machine learning courses but this course was really special." "I loved the course. I learned a ton and Robert was very available to students. When I think about how much I would have paid at my university for a similar course, TWIML is a great value." To learn more about the courses and study group please feel free to peruse the FAQ we've prepared below. Note, enrollment is open through Thursday, September 17th. When you're ready to enroll, you can do so at Robert's AltDeep.ai web site. Be sure to also join the TWIML Community and the #causality_course channel on our Slack. Frequently Asked Questions What are the courses? The course sequence consists of six modules as listed on the AltDeep web site. They are: Model-based Thinking in Machine Learning. Lay the foundation for causal models by deconstructing mental biases and acquiring new mental models for applied DS/ML. Do Causality like a Bayesian. Continue your “mental refactoring” by developing a Bayesian mental model for machine learning. How to Speak Graph; or DAG that's a Nice Model! Become fluent in directed graphs and graph algorithms as a language of probability. The Tao of Do; Modeling and Simulating Causal Interventions. Learn to build your first causal generative machine learning model using a deep learning framework Applied Causal Inference; Identification and Estimation of Causal Effects from Data. Gain mastery of programmatic causal effect estimation. Counterfactual Machine Learning. Implement counterfactual reasoning algorithms in automated decision-making settings in industry. Are the courses free? How much are they? How are they sold? These are paid courses. Robert has put a ton of work into this sequence and will be providing TWIML learners with human support as they take the courses. Rather than selling the modules individually, Robert offers enrollment in the full Causal Modeling in Machine Learning Track for $1,199. This course sequence is designed to take you deeper into the practice of causal ML. How long will each course run? What is the level of effort expected? The course will run from September 10th to December 10th. On time commitment, if you just want to go through lectures and videos, then the time commitment is akin to a deep read of one paper a week. If you wanted to work through code examples and assignments, then more. The course is designed to give you a level of depth that suits you. Is there a discount for TWIML participants? Glad you asked. Yes, to kick off this partnership, Robert has agreed to extend a 15% discount to TWIML community members who register using the links above. I suspect this is the lowest price these courses will ever be offered for. Please use the discount codes TWIML2020FALL or TWIML2020FALL (for the monthly payment plan) to get the TWIML participant discount. Is TWIML paid as part of this arrangement? Yes, we are an AltDeep / Teachable affiliate and get a commission as part of the partnership. Whatever we earn through this relationship will help support our broader community and educational efforts. That said, we would never recommend a course we didn’t think was a good use of your time and a good value. How long will students have access to the course materials? After you enroll, you will have access to the materials indefinitely. How long will the course be open? While the course itself is fundamentally designed for self-paced study, with Robert running a live weekly study group, enrollment will be closed on September 17th. Will the weekly study group sessions be open to anyone? Robert’s weekly study group sessions are intended for enrollees and will assume that learners have at least gone over that week’s lectures at a high level. Is there a detailed syllabus? Yes, the syllabus will roughly follow that of Robert’s Northeastern course, which you can find here. What programming languages/frameworks are used in the course? The courses incorporate probabilistic programming concepts and use Pyro. From the Pyro web site:Pyro is a universal probabilistic programming language (PPL) written in Python and supported by PyTorch on the backend. Pyro enables flexible and expressive deep probabilistic modeling, unifying the best of modern deep learning and Bayesian modeling. Where can I learn more about Robert and the course? Please check out my recent podcast, Causality 101, with Robert, here. What can I expect from the weekly online sessions? The weekly online sessions are live study group sessions presented by Robert, the instructor and author of the Causal Models in Machine Learning courses. At the sessions, Robert will present a summary of that week’s lecture and open the floor for student Q&A. What if I cannot participate in the weekly online sessions? The weekly study group sessions will be recorded and will be available to TWIML enrollees. What is the refund policy? There is a 30-day refund policy on the courses.
Today we're joined by Sushil Thomas, VP of Engineering for Machine Learning at Cloudera. Over the summer, I had the pleasure of hosting Sushil and a handful of business leaders across industries at the Cloudera Virtual Roundtable. In this conversation with Sushil, we recap the roundtable, exploring some of the topics discussed and insights gained from those conversations. Sushil gives us a look at how COVID19 has impacted business throughout the year, and how the pandemic is shaping enterprise decision making moving forward. We also discuss some of the key trends he's seeing as organizations try to scale their machine learning and AI efforts, including understanding best practices, and learning how to hybridize the engineering side of ML with the scientific exploration of the tasks. Finally, we explore if organizational models like hub vs centralized are still organization-specific or if that's changed in recent years, as well as how to get and retain good ML talent with giant companies like Google and Microsoft looming large.
Kamran Khan, founder of BlueDot, recently found his company the subject of attention for being among the first to publicly warn about the coronavirus (COVID-19) that initially appeared in the Chinese city of Wuhan. How did the company's system of data gathering techniques and algorithms help flag the potential dangers of the disease? In this interview, Kamran shares how they use a variety of machine learning techniques to track, analyze and predict infectious disease outbreaks. As a practicing physician based in Toronto, Kamran was directly impacted by the SARS outbreak in 2003. "We saw our hospitals completely overwhelmed. They went into lockdown. All elective procedures were canceled...even the city took on a different feel...there were billions of financial losses...and Toronto was just one of dozens." In the wake of that crisis, governments have been slow to act. Efforts like the International Health Regulations Treaty (2005), which aims to standardize communication about diseases, help but are not well enforced. It doesn't help that these nations are often unaware of the severity of an outbreak, or are hesitant to report a threat because of potential economic consequences. Ultimately, his experience with the SARS crisis led Kamran to explore the role technology might play in anticipating outbreaks and predicting how they might spread. Kamran's insight ultimately lead to the creation of BlueDot, which applies machine learning to four main challenges in infectious disease tracking: Surveillance, Dispersion, Impact, and Communication. Surveillance The BlueDot engine gathers data on over 150 diseases and syndromes around the world, looking at over 100,000 online articles each day spanning 65 languages, searching every 15 minutes, 24 hours a day. This includes official data from organizations like the Center for Disease Control or the World Health Organization, but also counts on less structured, local information from journalists and healthcare workers. BlueDot's epidemiologists and physicians manually classified the data and developed a taxonomy so relevant keywords could be scanned efficiently. They later applied ML and NLP to train the system. Kamran points out that the algorithms in place perform "relatively low-complexity tasks, but they're incredibly high volume and there's an enormous amount of them, so we can simply train a machine to replicate our judgment [for classifying]". As a result of their system's algorithms, only a handful of cases are flagged for human experts to analyze. In the case of COVOID-19, the system highlighted articles in Chinese that reported 27 pneumonia cases associated with a market that had seafood and live animals in Wuhan. Dispersion Recognizing the role that travel plays in disease dispersion—especially in the age of air travel—BlueDot uses geographic information system (GIS) data and flight ticket sales to create a dispersion graph for each disease based on the airports connected to a city and where passengers are likely to fly. Not everyone travels by air, so they also use anonymized location data from 400 million mobile devices to track flows from outbreak epicenters to other parts of the region or world. The locations receiving the highest volume of travelers are identified and diligently evaluated for what the impact of the disease could be in the area. For COVOID-19, BlueDot applied this methodology to identify many of the cities among the first to receive the coronavirus, including Tokyo, Bangkok, Hong Kong, Seoul, and Taipei. Impact Once a virus leaves its region of origin, a wide variety of factors determine whether it will ultimately die out or grow into a full-fledged outbreak: A region may have better or worse public health infrastructure, hospitable or inhospitable climates, or varying economic resources. BlueData's systems consider factors such as these to predict the potential impact on an identified area. For example, if a virus is being spread by ticks, and Vancouver is in the middle of winter snow, the likelihood of an outbreak is very low because ticks would not survive that climate. However, the same virus might thrive in a humid environment like Florida, making the region at-risk for an outbreak. Communication If an area is determined to be at-risk, the focus shifts to providing early warnings to health officials, hospitals, airlines, and government agencies in public health, national defense, national security and even agriculture. Kamran reiterates the importance of providing only the most relevant information to those who need it, referencing the ideas Clay Shirky and his 2008 talk], "It's Not Information Overload. It's Filter Failure. BlueDot first became aware of the pneumonia cases in Wuhan on December 31st, and in addition to notifying their clients and government stakeholders directly, they publicly released their findings in the Journal of Travel Medicine on January 14th. Criticism and Limitations These are incredibly difficult predictions to make, and the science behind the transmission of infectious diseases is complex and evolving every day. So, what is the proper role of technology? Kamran asserts that "by no means would [they] claim that AI has got this problem solved. It's just one of the tools in the toolbox." In some cases, Kamran and his team may lack sufficient observations to develop a machine learning model for a particular disease. For this and other reasons, the company relies on a combination of approaches and a diverse team of specialists in their work. With coronavirus already in full swing, BlueDot is looking more heavily at analyzing location data from mobile devices to provide a real-time understanding of how people are moving around. However, Kamran compares this to predicting the weather—the further ahead you're looking, the less accurate your prediction. Despite the limitations, Kamran reinforces the value of the work by acknowledging that "Manually, it would take a hundred people around the clock [to process the data], and we have four people and a machine."
How does LinkedIn allow its data scientists to access aggregate user data for exploratory analytics while maintaining its users' privacy? That was the question at the heart of our recent conversation with Ryan Rogers, a senior software engineer in data science at the company. The answer, it turns out, is through differential privacy, a topic we've covered here on the show quite extensively over the years. Differential privacy is a system for publicly sharing information about a dataset by describing patterns of groups within the dataset, the catch is you have to do this without revealing information about individuals in the dataset (privacy). Ryan currently applies differential privacy at LinkedIn, but he has worked in the field, and on the related topic of federated learning, for quite some time. He was introduced to the subject as a PhD student at the University of Pennsylvania, where he worked closely with Aaron Roth, who we had the pleasure of interviewing back in 2018. Ryan later worked at Apple, where he focused on the local model of differential privacy, meaning differential privacy is performed on individual users' local devices before being collected for analysis. (Apple uses this, for example, to better understand our favorite emojis 🤯 👍👏). Not surprisingly, they do things a bit differently at LinkedIn. They utilize a central model, where the user's actual data is stored in a central database, with differential privacy applied before the data is made available for analysis. (Another interesting use case that Ryan mentioned in the interview: the U.S. Census Bureau has announced plans to publish 2020 census data using differential privacy.) Ryan recently put together a research paper with his LinkedIn colleague, David Durfee, that they presented as a spotlight talk at NeurIPS in Vancouver. The title of the paper is a bit daunting, but we break it down in the interview. You can check out the paper here: Practical Differentially Private Top-k Selection with Pay-what-you-get Composition. There are two major components to the paper. First, they wanted to offer practical algorithms that you can layer on top of existing systems to achieve differential privacy for a very common type of query: the "Top-k" query, which means helping answer questions like "what are the top 10 articles that members are engaging with across LinkedIn?" Secondly, because privacy is reduced when users are allowed to make multiple queries of a differentially private system, Ryan's team developed an innovative way to ensure that their systems accurately account for the information the system returns to users over the course of a session. It's called Pay-what-you-get Composition. One of the big innovations of the paper is discovering the connection between a common algorithm for implementing differential privacy, the exponential mechanism, and Gumbel noise, which is commonly used in machine learning. One of the really nice connections that we made in our paper was that actually the exponential mechanism can be implemented by adding something called Gumbel noise, rather than Laplace noise. Gumbel noise actually pops up in machine learning. It's something that you would do to report the category that has the highest weight, [using what is] called the Gumbel Max Noise Trick. It turned out that we could use that with the exponential mechanism to get a differentially private algorithm. [...] Typically, to solve top-k, you would use the exponential mechanism k different times⁠ —you can now do this in one shot by just adding Gumbel noise to [existing algorithms] and report the k values that are in the the top […]which made it a lot more efficient and practical. When asked what he was most excited about for the future of differential privacy Ryan cited the progress in open source projects. This is the future of private data analytics. It's really important to be transparent with how you're doing things, otherwise if you're just touting that you're private and you're not revealing what it is, then is it really private? He pointed out the open-source collaboration between Microsoft and Harvard's Institute for Quantitative Social Sciences. The project aims to create an open-source platform that allows researchers to share datasets containing personal information while preserving the privacy of individuals. Ryan expects such efforts to bring more people to the field, encouraging applications of differential privacy that work in practice and at scale. Listen to the interview with Ryan to get the full scope! And if you want to go deeper into differential privacy check out our series of interviews on the topic from 2018. Thanks to LinkedIn for sponsoring today's show! LinkedIn Engineering solves complex problems at scale to create economic opportunity for every member of the global workforce. AI and ML are integral aspects of almost every product the company builds for its members and customers. LinkedIn's highly structured dataset gives their data scientists and researchers the ability to conduct applied research to improve member experiences. To learn more about the work of LinkedIn Engineering, please visit engineering.linkedin.com/blog.
Sam Charrington: Hey, what's up everyone? This is Sam. A quick reminder that we've got a bunch of newly formed or forming study groups, including groups focused on Kaggle competitions and the fast.ai NLP and Deep Learning for Coders part one courses. It's not too late to join us, which you can do by visiting twimlai.com/community. Also, this week I'm at re:Invent and next week I'll be at NeurIPS. If you're at either event, please reach out. I'd love to connect. All right. This week on the podcast, I'm excited to share a series of shows recorded in Orlando during the Microsoft Ignite conference. Before we jump in, I'd like to thank Microsoft for their support of the show and their sponsorship of this series. Thanks to decades of breakthrough research and technology, Microsoft is making AI real for businesses with Azure AI, a set of services that span vision, speech, language processing, custom machine learning, and more. Millions of developers and data scientists around the world are using Azure AI to build innovative applications and machine learning models for their organizations, including 85% of the Fortune 100. Microsoft customers like Spotify, Lexmark, and Airbus, choose Azure AI because of its proven enterprise grade capabilities and innovations, wide range of developer tools and services and trusted approach. Stay tuned to learn how Microsoft is enabling developers, data scientists and MLOps and DevOps professionals across all skill levels to increase productivity, operationalize models at scale and innovate faster and more responsibly with Azure machine learning. Learn more at aka.ms/azureml. All right, onto the show. Sam Charrington: [00:01:52] All right everyone, I am here in Sunny Orlando, actually it's not all that sunny today, it's kind of gray and gray and rainy but it is still Sunny Orlando, right? How could it not be? At Microsoft Ignite, and I've got the wonderful pleasure of being seated with Sarah Bird. Sarah is a principal program manager for Azure Machine Learning platform. Sarah, welcome to the TWIML AI Podcast. Sarah Bird: [00:02:15] Thank you, I'm excited to be here. Sam Charrington: [00:02:17] Absolutely. I am really excited about this conversation we're about to have on responsible AI. But before we do that, I'd love to hear a little bit more about your background. You've got a very enviable position kind of at the nexus of research and product and tech strategy how did you create that? Sarah Bird: [00:02:37] Well I started my career in research. I did my PhD in machine learning systems at Berkeley and I loved creating the basic technology, but then I wanted to take it to the next step and I wanted to have people who really used it. And I found that when you take research into production, there's a lot more innovation that happens. So since graduating I have styled my career around living at that intersection of research and product, and taking some of the great cutting edge ideas and figuring out how we can get them in the hands of people as soon as possible. And so my role now is specifically focused on trying to do this for Azure Machine Learning and responsible AI is one of the great new areas where there's a ton of innovation and research, and people need it right now. And so we're working to try to make that possible. Sam Charrington: [00:03:33] Oh, that's fantastic. And so between your grad work at Berkeley and Microsoft, what was the path? Sarah Bird: [00:03:42] So I was in John Lankford's group in Microsoft research and was working on a system for contextual bandits and trying to make it easier for people to use those in practice, because a lot of the times when people were trying to deploy that type of algorithm, the system infrastructure would get in the way. You wouldn't be able to get the features to the point of decision or the logging would not work and it would break the algorithm. And so we designed a system that made it correct by construction, so it was easy for people to go and plug it in, and this has actually turned into the Personalizer cognitive service now. But through that experience, I learned a lot about actually working with customers and doing this in production, and so I decided that I wanted to have more of that in my career. And so I spent a year as a technical advisor which is a great role in Microsoft where you work for an executive and advise them and help work on special projects. And it enables you to see both the business and the strategy side of things as well as all the operational things, how you run orgs and then of course the technical things. And I realized that I think that mix is very interesting. And so after that I joined Facebook and my role was at the intersection of FAIR, Facebook AI Research and AML which was the applied machine learning group with this role of specifically trying to take research into production and accelerate the rate of innovation. So I started the Onyx Project as a part of that, enabling us to solve a tooling gap where it was difficult to get models from one framework to another. And then also worked on PyTorch and enabling us to make that more production ready. And since then I've been working in AI ethics. Sam Charrington: [00:05:34] Yeah. If we weren't going to be focused on AI ethics and responsible AI today, we would be going deep into Personalizer, what was Microsoft Decision Service  and this whole contextual bandits thing. Really interesting topic, not the least of which because we talk a lot about reinforcement learning and if it's useful, and while it's not this deep reinforcement learning game playing thing, it's reinforcement learning and people are getting a lot of use out of it in a lot of different contexts. Sarah Bird: [00:06:05] Yeah. When it works, right? It doesn't work in all cases, but when it works, it works really well. It's the kind of thing where you get the numbers back and you're like, can this be true? And so I think it's a really exciting technology going forward and there's a lot of cases where people are using it successfully now, but I think though there'll be a lot more in the future. Sam Charrington: [00:06:25] Awesome. I'll have to take a rain check on that aspect of the conversation and kind of segue over to the responsible AI piece. And I've been thinking a lot about a a tweet that I saw by Rachel Thomas who is a former guest of the podcast, long time friend of the show and currently the UCSF Center for Applied Data Ethics head. And she was kind of lamenting that there are a lot of people out there talking about AI ethics like it's a solved problem. Do you think it's a solved problem? Sarah Bird: [00:06:58] No, absolutely not. I think there are, are fundamentally hard and difficult problems when we have a new technology, and so I think we're always going to be having the AI ethics conversation, this is not something that we're going to solve and go away. But what I do think we have now is a lot more tools and techniques and best practices to help people start the journey of doing things responsibly. And so I think the reality is there are many things people could be doing right now that they're not. And so I, I feel like there's an urgency date to get some of these tools into people's hands so that we can do that. So I `think we can quickly go a lot farther than we have right now. Sam Charrington: [00:07:41] In my conversations with folks that are working on this and thinking about the role that responsible AI plays and the way they "do AI," do machine learning. A lot of people get stopped at the very beginning like: Who should own this? Where does it live? Is it a research kind of function or is it a product function, or is it more of a compliancy thing for a chief data officer or a chief security officer? [Is it] one of those executive functions and oversight, or compliance is the better word? What do you see folks doing and do you have any thoughts on successful patterns of where it should live? Sarah Bird: [00:08:33] Yeah, I think the models that we've been using and are thinking a lot about... the transition  to security, for example. And I think the reality is it's not one person's job or one function. Everybody now has to think about security, even your basic software developers have to know and think about it when they're designing. However, there are people who are experts in it and handle the really challenging problems. There is of course legal and compliance pieces in there as well. And so I think we're seeing the same thing where we really need every role to come together and do this. And so one of the patterns we are seeing is part of the challenge with responsible AI and technology is that we've designed technology to abstract away things and enable you to just focus on your little problem, and this has led to a ton of innovation. However, the whole idea of responsible AI is actually, you need to pick your head up, you need to have this larger context, you need to think about the application in the real world, you need to think about the implications. And so we have to break a little bit of our patterns of 'my problem is just this little box,' and so we're finding that user research and design, for example, is already trained and equipped to think about the people element in that. And so it's really great to bring them into more conversations as we're developing the technology. So that's one pattern that we're finding adds a lot  of value. Sam Charrington: [00:10:07] In my conversation with with Jordan Edwards, your colleague, many of his answers were all of the above. And it sounds like this one is an "all of the above" response as well. Sarah Bird: [00:10:19] Yeah. I think doing machine learning in practice takes a lot of different roles, as Jordan was talking about, in operationalizing things, and then responsible AI just adds an extra layer of more roles on top of that. Sam Charrington: [00:10:32] Yeah. I guess one of the challenges that kind of naturally evolves when everyone has to be thinking about something is that it's a lot  harder, right? The developer is trained as a developer and now they have to start thinking about this security thing, and it's changing so quickly and the best practices are evolving all the time, and it's hard to stay on top of that. If we're to replicate that same kind of model in responsible AI, what sounds like the right thing to do? How do we support the people that are on the ground trying to do this? Sarah Bird: [00:11:07] Yeah. And I think it's definitely a challenge because the end result can't be that every individual person has to know the state of the art in every area in responsible AI. And so one of the ways that we're trying to do this is, as much as possible, build it into our processes and our tooling. So that you can say, okay, well you should have a fairness metric for your model and you can talk to experts about what that fairness metric should be, but you should know the requirement that you should have a fairness metric, for example. And so we first are starting with that process layer and then in Azure Machine Learning, we've built tools that enable you to easily enact that process. And so the foundational piece is the MLOps story that Jordan was talking about where we actually enable you to have a process that's reproducible, that's repeatable. So you can say, before this model goes into production, I know that it's passed these validation tests and I know that a human looked at it and said, it looks good. And if it's out in production and there's an error or there's some sort of issue that arises, you can go back, you can recreate that model, you can debug the error. And so that's the real foundational piece for all of it. And then on top of that, we're trying to give data scientists more tools to analyze the models themselves. And there's no magic button here. It's not just, Oh, we can run a test and we can tell you everything you want to know. But there's lots of great algorithms out there and research that help you better understand your model. Like SHAP or LIME are common interpretability ones. And so we've created a toolkit called Interpret ML, this is an open source toolkit you can use it anywhere. But it enables you to easily use a variety of these algorithms to explain your model behavior and explore it and see if there are any issues. And so we've also built that into our machine learning process so that if I build a model, I can easily generate explanations for that model. And when I've deployed it in production, I can also deploy and explain her with it so individual predictions can be explained while it's running so I can understand if I think it's doing the right thing and if I want to trust it, for example. Sam Charrington: [00:13:35] It strikes me that there's a bit of a catch 22 here, in the sense that the only way we could possibly do this is by putting tools in the hands of the folks that are working data scientists and machine learning engineers that are working on these problems. But the tools in their very nature kind of abstract them away from the problem and allow them, if not, encourage them to think less deeply about what's going on underneath. Right? How do we address that? Do you agree with that first of all? Sarah Bird: [00:14:09] No, I completely agree with that and it's a challenge that we have in all of these cases where we want to give the tool to help them and to have more insight but it's easy for people to just use it as a shortcut. And so in a lot of cases, we're being very thoughtful about the design of the tool and making sure that it is helping you surface insights. But it's not saying this is the answer because I think when you start doing that where you have something that flags and says this is a problem, then people really start relying on that. And maybe someday we will have the techniques where we have that level of confidence and we can do it. But right now we really don't, and so I think a lot of it is making sure that we designed the tools that encourages this mindset of exploration and deeper understanding of your models and what's going on. And not just, Oh, this is just another compliance tests I have to pass I just run this test and it says green. And I go. Sam Charrington: [00:15:12] You alluded to this earlier in the conversation, but it seems appropriate here as well, and it's maybe a bit of a tangent, but so much of pulling all these pieces together is kind of a user experience and design. Any thoughts on that? Is that something that you've kind of dug into and studied a lot? Or are the other folks worry about that here? Sarah Bird: [00:15:36] It's not in my background, but to me it's an essential part of the function of actually making these technologies usable. And particularly when you take something that as complex as an algorithm and you're trying to make that abstracted and usable for people, the design is a huge part of the story. And so what we're finding in responsible AI is that we need to think about this even more. And a lot of the guidelines are saying be more thoughtful and include sort of more careful design. For example, people are very tempted to say, well, this is the data I have so this is the model I can build and so I'm going to put it in my application that way. And then if it has too much inaccuracy, then you spend a lot of resources to try and make the model more accurate where you could have just had a more elegant UI design, for example, where you actually get better feedback based on the UI design or the design can tolerate more errors and you don't need that higher model accuracy. So we're really encouraging people to co-design the application in the model and not just take it for granted that this is what the model does and that's the thing we're gonna focus on. Sam Charrington: [00:16:53] With the Interpret ML tool, what's the user experience like? Sarah Bird: [00:17:01] It depends on what you're trying to do, there's two types of interpretability that people think about. One is what we call Glass-Box models. And the idea there is I want my model to be inherently interpretable. So I'm gonna pick something like a linear model or decision trees where I can actually inspect the model and enable you to to build a model of that, that you can actually understand. And so we support a bunch of different Glass-Box explainer or models. So then you can actually use it to train your own model. And the other part is Black-Box explainers where I have a model that I is a black box and I can't actually inspect it, but I can use these different algorithms to explain the behavior of the model. And so in that case what we've done is made it easy for you to just call and explain and ask for global explanations and ask for local explanations and ask for feature importance. And then all of those are brought together in an interactive dashboard where you can actually explore the explanations and try to understand the model behavior. So a lot of the experience it's an SDK and so it's all easy calls to ask for explanations, but then we expect a lot of people to spend their time in that dashboard exploring and understanding. Sam Charrington: [00:18:32] I did a really interesting interview with Cynthia Rudin who you may know she's a Duke professor and the interview was focused on her research that essentially says that we should not be using black box models in, I forget the terminology that she used, but something like mission critical scenarios or something along those lines where we're talking about someone's life or Liberty that kind of thing. Does providing interpretability tools that work with black box models, like encourage their use in scenarios that they shouldn't really be used in? And are there ways that you advise folks when and when not they should be using those types of models? Sarah Bird: [00:19:19] So we have people who do publish best practices for interpretability and  it's a very active area of work for the company. And we work with the partnership on AI to try to make industry-wide recommendations for that. I don't think it's completely decided on this idea that models should be interpretable in these settings versus, well, we want other mechanisms to make sure that they're doing the right thing. Interpretability is one way that we could be sure that they're doing the right thing, but we also could have more robust testing regimes. Right? There's a lot of technologies where we don't understand every detail of the technology, but we've been able to build safety critical systems on top of it, for example. And so yeah as a company we do try to provide guidance, but I don't think the industry has really decided the final word on this. And so the mindset of the toolkit is enabling you to use these techniques if it's right for you. But that doesn't specifically say that you should go use a neural net in a particular setting. Sam Charrington: [00:20:27] So in addition to the Interpret ML toolkit you also announced this week here from Ignite, a Fair Learn toolkit. What's that all about? Sarah Bird: [00:20:39] So it's the same spirit as Interpret ML where we want to bring together a collection of fairness techniques that have been published in research and make it easy for people to use them all in one toolkit with the same spirit that you want to be able to analyze your model and understand how it's working so that you could make decisions around fairness. And so there's famously, many different fairness metrics published. I think there was a paper cataloging 21 different fairness metrics. And so we've built many of these common ones into the toolkit and then it makes it easy for you to compare how well your model works for different groups of people in your data set. So for example, I could say does this model have the same accuracy for men and women? Does this model have the same outcomes for men and women? And so we have an interactive dashboard that allows you to explore these differences between groups and your model performance through a variety of these metrics that have been published in research. Then we've also built in several mitigation techniques so that if you want to do mitigation via post-processing and your model, then you can do that. For example, setting thresholds per group. And in a lot of cases it might be that you actually want to go and fix the underlying data or you wanting to make some different decisions. So the mitigation techniques aren't always what you would want to do, but they're available if you want to do that. And so the name of the toolkit actually comes from one of these mitigation techniques from Microsoft research where the algorithm was originally called Fair Learn. And the idea is that you say, I wanna reduce the difference between two groups on a particular dimension. So you pick the metric and you pick the groups and the algorithm actually retrains your model by re-wading data and iteratively retraining to try to reduce that disparity. So we've built that into the toolkit. So now you can actually look at a variety of your versions of your model and see if one of them has properties that works better for what you're looking for, to deploy. Sam Charrington: [00:22:59] Again, I'm curious about the user experience in, in doing this. How much knob turning and tuning does the user need to do when applying that technique you were describing? Or is it more, I'm envisioning something like contextual bandage reinforcement learning where it's kind of tooling the knobs for you. Sarah Bird: [00:23:18] Yeah, it is doing the knobs and the retraining, but what you have to pick is which metric you're trying to minimize. Do I want to reduce the disparity between the outcomes or do I want to reduce the disparity and accuracy or some other there's many different metrics you could pick, but you have to know the metric that's right for your problem. And then you also need to select the groups that you want to do. So it can work in a single dimension like as we were saying making men and women more more equal, but then it would be a totally separate thing to do it for age, for example. So you have to pick both the sensitive attribute that you are trying to reduce disparity and you have to pick the metric for disparity. Sam Charrington: [00:24:10] Were you saying that you're able to do multiple metrics in parallel or you're doing them serially? Sarah Bird: [00:24:17] Right now the techniques work for one, for just one metric. So it will produce a series of models, and if you look at the graph, you can actually plot disparity by accuracy and you'll have models that are on that Pareto optimal curve to look at. But then if you said, okay, well now I want to look at that same chart for age, the models might be all over the place in the space of disparity and accuracy. So it's not a perfect technique, but there are some settings where it's quite useful. Sam Charrington: [00:24:48] So going back to this idea of abstraction and tools versus deeply understanding the problem domain and how to think about it in the context of your problem domain. I guess the challenge domain or your problem domain, I don't know what the right terms are. But you mentioned that paper with all of the different disparity metrics and the like. Is that the best way for folks to get up to speed on this or are there other resources that you've come across that are useful? Sarah Bird: [00:25:23] Yeah, I think for fairness in particular it's better to start with your application domain and understand, for example, if you're working in an employment setting, how do we think about fairness and what are the cases and so in that case we actually recommend that you talk to domain experts, even your legal department to understand what fairness means in that setting. And then you can go to the academic literature and start saying, okay, well, which metrics line up with that higher level concept of fairness for my setting. But if you start with the metrics I think it can be very overwhelming and there's just many different metrics and a lot of them are quite different and in other ways they're very similar with each other. And so I find it much easier to start with the domain expertise and know what you're trying to achieve in fairness and then start finding the metrics that line up with that. Sam Charrington: [00:26:22] You're also starting to do some work in the differential privacy domain. Tell me a little bit about that. Sarah Bird: [00:26:27] Yeah, we announced a couple of weeks ago that we are building an open source privacy platform with Harvard and differential privacy is a really fascinating technology. It was first published in Microsoft Research in 2006 and it was a very interesting idea, but it has taken a while for it, as an idea, to mature and develop and actually be able to be used in practice. However, now we're seeing several different companies who are using it in production. But in every case the deployment was a very bespoke deployment with experts involved. And so we're trying to make a platform that makes it much easier for people to use these techniques without having to understand them as much. And so the idea is the open source platform can go on top of a data store, enable you to do queries in a differentially private way, which means that actually it adds noise to the results so that you can't reconstruct the underlying data and also then potentially use the same techniques to build simple machine learning models. And so we think this is particularly important for some of our really societaly valuable datasets. For example, there are data sets where people would like to do medical research, but because we're worried about the privacy of individuals, there's limits to what they can actually do. And if we use differential private interface on that, we have a lot more privacy guarantees and so we can unlock a new type of innovation and research in understanding our data. So I think we're really excited and think this could be the future of privacy in certain applications, but the tooling just isn't there, and so we're working on trying to make it easier for people to do that. We're building it in the open source because it's important that people can actually ... It's very easy to get the implementation of these algorithms wrong and so we want the community and the privacy experts to be able to inspect and test the implementations and have the confidence that it's there. And also we think this is such an important problem for the community. We would like anybody who wants to, to be joining in and working on this. This is not something that we can solve on our own. Sam Charrington: [00:28:58] Yeah, differential privacy in general and differentially private machine learning are fascinating topics and ones that we've covered fairly extensively in the podcast. We did a series on differential privacy a couple of years ago maybe and it's continuing to be an interesting topic. At the Census Bureau I think is using differential privacy for the first time next year and it's both providing the anticipated benefits but also raising some interesting concerns about an increased opacity on the part of researchers to the data that they wanna get access to. Are you familiar with that challenge? Sarah Bird: [00:29:41] Yeah, absolutely. So the reality is people always want the most accurate data, right? It doesn't sound great to say, well, we're adding noise and the data is less accurate. But, in a lot of cases it is accurate enough for the tasks that you want to accomplish. And I think we have to recognize that, privacy is one of the sort of, fundamental values that we want to uphold, and so in some cases it's worth the cost. For the census in particular, to motivate the decision to start using this for the 2020 census they did a study where they took the reports from the 1940 census and they were able to recreate something like 40% of Americans' data with the result of just the outputs from the census. Sam Charrington: [00:30:33] Meaning personally identify 40% of Americans? Sarah Bird: [00:30:37] Yeah, he talks about this in his ICML keynote from last year. So if you want to learn more you can watch the keynote. But yeah, basically they took all the reports and they used some of these privacy attacks and they could basically recreate a bunch of the underlying data. And this is a real risk, and so we have to recognize that yes, the census results are incredibly important and they help us make many different decisions, but also protecting people's data is important. And so some of it is education and changing our thinking and some of it is making sure that we use the techniques in the right way in that domain where you're not losing what you were trying to achieve in the first place, but you are adding these privacy benefits. Sam Charrington: [00:31:21] There are a couple of different ways that people have been applying differential privacy one is a, a more centralized way where you're applying it to a data store. It sounds a little bit like that's where your focus is. Others like Apple's a noted use case where they're applying differential privacy in a distributed manner at the handset to keep user data on the iPhone, but still provide information centrally for analysis. Am I correct that your focus is on the centralized use case? Or does the toolkit also support the distributed use case? Sarah Bird: [00:32:02] We are focusing on the global model. The local model works really well, and particularly in some of these user telemetry settings, but it limits what you can do. You need much larger volume to actually get the accuracy for a lot of the queries that you need, and there aren't as many queries that you can do. And so the global model, on the other hand, there's a lot more that you can do and still have reasonable privacy guarantees. And so as I was saying, we were motivated by these cases where we have the data sets. Like somebody is trusted to have the data sets but we can't really use them. And so that looks like a global setting. And so to start, we're focused on, on the global piece, but there are many cases where the local is promising and there are cases where we are doing that in our products. And so it's certainly a direction that things could go. Sam Charrington: [00:32:58] And differential privacy from a data perspective doesn't necessarily get you to differentially private machine learning. Are you doing anything in particular on the differentially private ML side of things? Sarah Bird: [00:33:11] The plan is to do that but the project is pretty new so we haven't built it yet. Sam Charrington: [00:33:19] And before we wrap up, you're involved in a bunch of industry and research initiatives in the space that you've mentioned, MLSys, a bunch of other things. Can you talk a little bit about some of the broader things that you're doing? Sarah Bird: [00:33:38] Yeah, so I helped found the, now I think named MLSys systems and machine learning research conference. And that was specifically because I've been working at this intersection for a while and there were some dark days where it was very hard to publish work because the machine learning community was like, this is a systems result. And the systems community was like, this doesn't seem like a systems result and so we started the conference about two years ago and apparently many other people were feeling the same pain because even from the first conference, we got excellent work. People's top work, which is always a challenge with research conferences because people don't want to submit their best work to an unnamed conference. Right? But there was such a gap for the community. So it's been really exciting  to see that community form more  and now have a home where they can put their work and connect.  I've also been running the machine learning systems workshops at NeurIPS for several years now. And that's been a really fun place because it really has helped us form the community, particularly before we started the conference. But it's also a place where you can explore new ideas. This last year we're starting to see a lot more innovation at the intersection of programming languages and machine learning. And so in the workshop format we can have several of those talks highlighted, and have a dialogue, and show some of the emerging trends so that's been a really fun thing to be involved in. Sam Charrington: [00:35:13] Awesome. Yeah, was it last year that there was both the SysML workshop and the ML for systems workshop and it got really confusing? Sarah Bird: [00:35:24] Yeah. This year too. We have both. And I think that's a sign that the field is growing that it used to be that it felt like we didn't even have enough people for one room at the Intersection of Machine Learning and Systems. And I think this last year there was maybe four or 500 people in our workshop alone. And so that's great. Now, there's definitely room to have workshops on more focused topics. Right? And so I think machine learning for systems is an area that people are really excited about now that we have more depth in understanding the intersection. For me, it's very funny because that is really kind of the flavor of my thesis which was a  while ago. And so it's a fun to see it now starting to become an area that people are excited about. Sam Charrington: [00:36:16] The other conference that we didn't talk about, ML for Systems is all about using machine learning within computational systems, networking systems as a way to optimize them. So for example, ML to do database query optimization. Also a super interesting topic. Sarah Bird: [00:36:36] Yeah, I know it absolutely is. And I really believe in that, and I think for several years people were just trying to replace kind of all of the systems intelligent with one machine learning algorithm and it was not working very well. And I think what we're seeing now is recognizing that a lot of the algorithms that we used to control systems were designed for that way and  they work, actually, pretty well. But on the other hand, there's something that's dynamic about the world or the workload. And so you do want this prediction capability built in. And so a lot of the work now has a more intelligent way of plugging the algorithms into the system. And so now we're starting to see promising results at this intersection. So my thesis work was a resource allocation that built models in real time in the operating system and allocated resources. And it was exactly this piece where there was a modeling and a prediction piece, but, the final resource allocation algorithm was not purely machine learning. Sam Charrington: [00:37:43] Awesome. Wonderful conversation, looking forward to catching up with you at NeurIPS, hopefully. thanks so much for taking the time to chat with us. Sarah Bird: [00:37:52] Yes, thanks for having me. And I look forward to seeing you at NeurIPS. Sam Charrington: [00:37:56] Thank you.
Bits & Bytes Google scraps controversial AI ethics council days after it was announced. Google has shuttered its new artificial intelligence ethics council, a little more than a week after announcing it, in large part due to employee reactions to the appointment of a conservative think tank leader to the group. WIPO launches AI-based image search tool for brands. The World Intellectual Property Organization (WIPO) has launched a new AI-powered image search technology that uses deep learning to identify combinations of concepts within an image, thus making it faster and easier to establish a trademark’s distinctiveness. Tim Apple poaches Ian GANfellow. Apple has poached another of Google’s top AI researchers. This time it’s Ian Goodfellow, best known as the creator of GANs, or generative adversarial networks, who has joined Apple in a director role. FDA Proposes Regulatory Framework for AI- and Machine Learning-Driven SaMD. US Food and Drug Administration (FDA) requested feedback on a new discussion paper that proposes applying a “focused review” approach to premarket assessments of software as a medical device (SaMD) technologies that are powered by AI and ML. [Paper] Qualcomm Reveals “Cloud AI 100” AI Inference Accelerator. Qualcomm announced their first discrete dedicated AI processors, the Qualcomm Cloud AI 100 family. The chip, which it expects to begin producing in 2020, is designed for use in datacenters to meet increasing demand for AI inference processing. Google launches an end-to-end AI platform. At Google Next ’19, Google announced the beta version of its end-to-end AI platform, which aims to offer developers and data scientists a one-stop shop for building, testing and deploying models. At Next, the company made several additional announcements as well, including updates to its suite of Cloud AI Solutions, AutoML, BigQuery ML, its pre-trained ML APIs, and more. Petuum Unveils Industrial AI Product for Complex Manufacturing Operations. Petuum announced the Petuum Industrial AI Autopilot product, which enables optimization of complex manufacturing operations with modules that continuously learn and adapt. Dollars & Sense Intel Capital announced investments in five AI companies at its annual Global Summit. IntelCapital led a $13M round in AI chip startup Untether AI and a $150M round in AI systems company SambaNova Systems. In addition, the company invested undisclosed amounts in Andrew Ng’s Landing AI and China-based CloudPick and Zhuhai Eeasy Technology. Run.AI, a startup building a new virtualization and acceleration platform for deep learning, announced that it has raised $13M Enlitic, a San Francisco, CA-based company leveraging AI for medical imaging raised $15M in Series B financing Boston Dynamics announced that it is acquiring California-based startup Kinema Systems, which makes computer-vision and ML systems for warehouse robots Evolve IP, announced that it has acquired Jog.ai, a speech analytics and natural language technology firm based in Austin, Texas Rasa, San Francisco-based an open source company that enables developers to build contextual AI assistants, has secured $13M in Series A funding Labelbox, a collaborative training data platform for machine learning applications, has raised a $10M Series A funding Observe.AI has secured $8M in a Series A funding To receive the Bits & Bytes to your inbox, subscribe to our Newsletter.