Arthur AI

Take Control of your AI Operations June 2020

The world is run by AI It always works flawlessly!*

Leadership Team Adam Wenchel Liz O’Sullivan John Dickerson Priscilla Keegan Hines CEO VP Commercial Chief Scientist Alexander VP Machine Learning Founder, Capital One Head of partnerships & ML Faculty at UMD VP Engineering Director of AI Research, Center for Machine customer success at PhD from Carnegie Director, Capital One Capital One Center for Learning and VP for Data Clarifai Mellon Center for Machine Machine Learning Innovation Leading voice on Learning PhD in Computational 20 years AI experience, Responsible AI Work with Facebook and Neuroscience, University Google VP of Engineering at starting at DARPA 10 years commercial AI Learning Objects of Texas Previous company, Anax, experience Noted ML fairness expert Lead Data Scientist at NSF CAREER award Lead architect at Red Hat acquired by Capital One MS in CS from UNC IronNet winner Adjunct faculty member, DARPA PI Georgetown University

The Three Pillars of Model Observability Performance Explainability Bias Detection

ARTIFICIAL INTELLIGENCE CORONAVIRUS GOING OUTSIDE SAFELY CHILDCARE & SUMMER CAMPS CAN I TRAVEL? Free Resources Pandemic-Related Disruptions Hurt Reliability of Some AI Models Humana and JPMorgan are among the companies watching for AI ‘model drift’ as lockdowns and social- distancing rules bring about unforeseen trends Heather Carroll Cox, Humana's chief digital health and analytics officer. PHOTO: HUMANA INC. ByBy SaSarraa CCaasstteellllaannooss June 10, 2020 5:30 am ET | WSJ PRO SAVE SHARE TEXT Some companies are monitoring and retraining artiﬁcial intelligence models that were developed using data from before the coronavirus pandemic because their predictions may no longer be accurate. Lockdowns, social-distancing rules, unemployment rates and supply-chain disruptions world-wide have led to changes in ARTIFICIAL INTELLIGENCE customer behavior and new economic trends. That has caused some AI models to become less reliable because of so-called model drift, analysts and executives say. Drift is common and happens when new data looks less similar to the historical data the model was originally trained on. For CORONAVIRUS example, models that take into account historical data about GOING OUTSIDE SAFELY CHILDCARE & SUMMER CAMPS CAN I TRAVEL? Free Resources unemployment rates have changed drastically during the pandemic, as the unemployment rate surged. Pandemic-Related Disruptions“They’re all out of whack now,” said Mike Gualtieri, vice Hurt Reliability of Some AI president and principal analyst at Forrester Research Inc. The majority of models in deployment at corporations are Models trained on data from many months and in some cases years before the pandemic. Industries where AI models are likely Humana and JPMorgan are among the companies most susceptible to drift over the past three months include watching for AI ‘model drift’ as lockdowns and social- retail, e-commerce, ﬁnance, travel and health care, experts say. distancing rules bring about unforeseen trends “The pandemic has introduced a whole bunch of new conditions that have never existed,” and AI models now have to take them into account, said Swami Chandrasekaran, head of solution architecture and digital solutions at professional Performance services ﬁrm KPMG LLP. Heather Carroll Cox, Humana's chief digital health and analytics officer. PHOTO: HUMANA INC. ByBy SaSarraa CCaasstteellllaannooss June 10, 2020 5:30 am ET | WSJ PRO Apoorv Saxena, global head of AI technology at JPMorgan Chase. PHOTO: JPMORGAN CHASE & CO. SAVE SHARE TEXT Investment banks, for example, may use an AI model to Some companies are monitoring and retraining artiﬁcial recommend investment products to prospective and existing intelligence models that were developed using data from customers, and now need to factor in those individuals’ risks of contracting Covid-19, Mr. Chandrasekaran said. The model can before the coronavirus pandemic because their predictions be retrained and updated by data scientists by factoring in that may no longer be accurate. new risk parameter. The new model can then recommend less risky investment products to customers more likely to contract Lockdowns, social-distancing rules, unemployment rates and Covid-19, which could help protect their retirement savings, he supply-chain disruptions world-wide have led to changes in said. customer behavior and new economic trends. That has caused Many AI models need to be monitored for deterioration of some AI models to become less reliable because of so-called performance, said Apoorv Saxena, global head of AI technology model drift, analysts and executives say. at JPMorgan Chase & Co. JPMorgan and other companies use software that monitors AI models to catch those that are deteriorating in accuracy, combined with data scientists who Drift is common and happens when new data looks less similar determine what has changed in a model and how to retrain it to to the historical data the model was originally trained on. For improve predictions. During times of economic distress and example, models that take into account historical data about volatility, such as the pandemic, it is critical to halt some AI unemployment rates have changed drastically during the models or retrain them, otherwise they may make wrong predictions, Mr. Saxena said. pandemic, as the unemployment rate surged. “Every model, depending on the domain, is trained on a “They’re all out of whack now,” said Mike Gualtieri, vice particular assumption of the world. [When] those assumptions president and principal analyst at Forrester Research Inc. change signiﬁcantly, then you need to update and make sure the model is still valid,” said Mr. Saxena. He declined to The majority of models in deployment at corporations are disclose speciﬁcs on which models have been retrained at JPMorgan because of the pandemic. trained on data from many months and in some cases years before the pandemic. Industries where AI models are likely Arthur AI Inc., which sells AI-monitoring technology and other services, has seen some customers’ AI models drift at a rate of most susceptible to drift over the past three months include about 20 times more than usual during the past three months, retail, e-commerce, ﬁnance, travel and health care, experts say. said Adam Wenchel, co-founder and chief executive. Arthur’s software detects anomalies in machine-learning models “The pandemic has introduced a whole bunch of new deployed by its customers. An anomaly is detected when the conditions that have never existed,” and AI models now have model’s predictions have shifted signiﬁcantly, by comparing to take them into account, said Swami Chandrasekaran, head the model’s results over time, or when there is a shift in the data that the model is using, Mr. Wenchel said. The New York- of solution architecture and digital solutions at professional based startup has about 20 business customers. services ﬁrm KPMG LLP. The unpredictable shifts in people’s behavior as a result of the pandemic will continue causing AI models to drift for the foreseeable future, Mr. Wenchel said. Louisville, Ky.-based health insurer Humana Inc. started a pilot program with Arthur last month to help better detect and prevent drift for models trained on large amounts of health- care data that ﬂuctuate during events such as the pandemic, said Slawek Kierner, Humana’s senior vice president of data and analytics. Arthur will help Humana’s data scientists identify which models are experiencing the most drift and then coders can tweak the algorithms manually, Mr. Kierner said. MORE IN ARTIFICIAL INTELLIGENCE • New Computer Chips Could Power AI to Next Level June 2, 2020 • Businesses Expected to Lean on Automation for Recovery May 20, 2020 • Ahold Delhaize Accelerates Automation as Coronavirus Pressures Workforce May 15, 2020 Two models that will be monitored can help how Humana Apoorv Saxena, global head of AI technology at JPMorgan Chase. supports its customers during the pandemic. One is used to PHOTO: JPMORGAN CHASE & CO. predict customers who may be facing food shortages because of factors including supply-chain disruptions caused by the pandemic, he said. Humana uses that information to send Investment banks, for example, may use an AI model to packages of food to its members. Arthur also will monitor an AI recommend investment products to prospective and existing model that can predict people who may be suﬀering from customers, and now need to factor in those individuals’ risks ofloneliness because of the pandemic and may need contracting Covid-19, Mr. Chandrasekaran said. The model can psychological support, Mr. Kierner said. be retrained and updated by data scientists by factoring in that “We have to make sure we’re constantly monitoring, assessing new risk parameter. The new model can then recommend less and reassessing so there is no drift. It’s going to happen inadvertently, so we need to be able to, in the moment, make risky investment products to customers more likely to contractthe pivot,” said Heather Carroll Cox, Humana’s chief digital Covid-19, which could help protect their retirement savings, he health and analytics oﬃcer. said. Write to Sara Castellanos at [email protected] Many AI models need to be monitored for deterioration of performance, said Apoorv Saxena, global head of AI technology at JPMorgan Chase & Co. JPMorgan and other companies use software that monitors AI models to catch those that are deteriorating in accuracy, combined with data scientists who determine what has changed in a model and how to retrain it to improve predictions. During times of economic distress and volatility, such as the pandemic, it is critical to halt some AI models or retrain them, otherwise they may make wrong predictions, Mr. Saxena said. NEWS FROM PRO AI “Every model, depending on the domain, is trained on a Pandemic-Related Disruptions Hurt Reliability of Some AI Models particular assumption of the world. [When] those assumptions change signiﬁcantly, then you need to update and make sure the model is still valid,” said Mr. Saxena. He declined to MORE NEWS disclose speciﬁcs on which models have been retrained at Pool Owners Take Up AI to Prevent Drownings JPMorgan because of the pandemic. Arthur AI Inc., which sells AI-monitoring technology and other AI Job Market Remains Strong Amid Coronavirus services, has seen some customers’ AI models drift at a rate of Slowdown about 20 times more than usual during the past three months, said Adam Wenchel, co-founder and chief executive. Arthur’s Black Technologists Hope New Conversations About software detects anomalies in machine-learning models Race Spark Overdue Change deployed by its customers. An anomaly is detected when the model’s predictions have shifted signiﬁcantly, by comparing the model’s results over time, or when there is a shift in the New Computer Chips Could Power AI to Next Level data that the model is using, Mr. Wenchel said. The New York- based startup has about 20 business customers. The unpredictable shifts in people’s behavior as a result of the pandemic will continue causing AI models to drift for the BACK TO TOP foreseeable future, Mr. Wenchel said. Louisville, Ky.-based health insurer Humana Inc. started a pilot PROFESSIONAL RESOURCES program with Arthur last month to help better detect and WSJ Conferences prevent drift for models trained on large amounts of health- Factiva care data that ﬂuctuate during events such as the pandemic, Risk & Compliance Journal Dow Jones Risk & Compliance said Slawek Kierner, Humana’s senior vice president of data Dow Jones Newswires and analytics. CFO Journal CIO Journal CMO Arthur will help Humana’s data scientists identify which Logistics models are experiencing the most drift and then coders can tweak the algorithms manually, Mr. Kierner said. Send us your feedback: [email protected] MORE IN ARTIFICIAL INTELLIGENCE • New Computer Chips Could Power AI to Next Level June 2, 2020 Subscriber Agreement & Terms of Use • Businesses Expected to Lean on Automation for Recovery May 20, Privacy Notice 2020 Cookie Notice • Ahold Delhaize Accelerates Automation as Coronavirus Pressures Copyright Policy Workforce May 15, 2020 Data Policy SIGN OUT Two models that will be monitored can help how Humana © 2020 Dow Jones & Company, Inc. supports its customers during the pandemic. One is used to All Rights Reserved predict customers who may be facing food shortages because of factors including supply-chain disruptions caused by the pandemic, he said. Humana uses that information to send packages of food to its members. Arthur also will monitor an AI model that can predict people who may be suﬀering from loneliness because of the pandemic and may need psychological support, Mr. Kierner said. “We have to make sure we’re constantly monitoring, assessing and reassessing so there is no drift. It’s going to happen inadvertently, so we need to be able to, in the moment, make the pivot,” said Heather Carroll Cox, Humana’s chief digital health and analytics oﬃcer. Write to Sara Castellanos at [email protected] NEWS FROM PRO AI Pandemic-Related Disruptions Hurt Reliability of Some AI Models MORE NEWS Pool Owners Take Up AI to Prevent Drownings AI Job Market Remains Strong Amid Coronavirus Slowdown Black Technologists Hope New Conversations About Race Spark Overdue Change New Computer Chips Could Power AI to Next Level BACK TO TOP PROFESSIONAL RESOURCES WSJ Conferences Factiva Risk & Compliance Journal Dow Jones Risk & Compliance Dow Jones Newswires CFO Journal CIO Journal CMO Logistics Send us your feedback: [email protected] Subscriber Agreement & Terms of Use Privacy Notice Cookie Notice Copyright Policy Data Policy SIGN OUT © 2020 Dow Jones & Company, Inc. All Rights Reserved

Explainability

Bias Detection

Production Models Arthur Platform Organization Operations Models • Better model performance ($$$) • Faster incident resolution Application 1 Arthur • Continuous Accuracy • Data Drift Data Science Models • Prediction Stability • Feedback on model • Explanations at Scale design • Fairness Analytics • Tools to debug production issues Application 2 • Audit Logging • Anomaly Detection Compliance • Alerting Models • Oversight • Auditability Available On-prem, Private Cloud, Public • Explainability Application 3 Cloud, or SaaS • Fairness Deploying Arthur is similar to application and infrastructure monitoring solutions: an agent gets deployed with your models and sends telemetry data back to the central platform for analysis. Data is available both via API for programmatic consumption and interactively through our user Interface.

Arthur is built for… • Data Scientists and Engineers • AND operations teams, risk functions, business leads and key stakeholders

The only model monitoring platform that is fully enterprise ready • Scale • Security • Ease of deployment • Serves Enterprise Stakeholders

Get in touch! [email protected]