![\"\"](\"https:\/\/sites.mitre.org\/aifails\/wp-content\/uploads\/sites\/15\/2020\/02\/icon_developing-002.png\"){kind=icon}
We’re Not Done Yet: After Developing the AI<\/h1>\n[et_pb_section fb_built=”1″ specialty=”on” _builder_version=”4.2.2″][et_pb_column type=”1_4″ _builder_version=”3.25″ custom_padding=”|||” custom_padding__hover=”|||”][et_pb_sidebar area=”et_pb_widget_area_19″ admin_label=”After Developing the AI” _builder_version=”4.4.1″ min_height=”200px” custom_margin=”||0px||false|false” custom_padding=”||0px||false|false”][\/et_pb_sidebar][\/et_pb_column][et_pb_column type=”3_4″ specialty_columns=”3″ _builder_version=”3.25″ custom_padding=”|||” custom_padding__hover=”|||”][et_pb_row_inner _builder_version=”4.0.3″ custom_padding=”||0px||false|false”][et_pb_column_inner saved_specialty_column_type=”3_4″ _builder_version=”4.0.3″][et_pb_text admin_label=”We’re Not Done Yet: After Developing the AI” _builder_version=”4.4.8″ custom_margin=”||0px||false|false” custom_padding=”5px||0px||false|false”]<\/p>\n
We’re Not Done Yet: After Developing the AI<\/h1>\nDeveloping AI is a dynamic, multifaceted process. Even if an AI performs optimally from a technical standpoint, other constraining factors could limit its overall performance and acceptance. Developing an AI to be safe and dependable means stakeholders must learn more about how the AI functions as the risks from its use increase. This section details factors that make that understanding challenging to achieve, and describes how proper documentation, explanations of intent, and user education can improve outcomes.<\/p>\n
<\/p>\n
[\/et_pb_text][et_pb_text admin_label=”Fails” _builder_version=”4.4.8″ custom_margin=”||0px||false|false” custom_padding=”5px||0px||false|false”]<\/p>\n
[\/et_pb_text][et_pb_tabs active_tab_background_color=”#a0ddf3″ inactive_tab_background_color=”#e5e7e8″ active_tab_text_color=”#000000″ admin_label=”Fails and Lessons Learned” module_class=”icon-tabs” _builder_version=”4.4.8″ tab_text_color=”#000000″ body_text_color=”#000000″ tab_font_size=”15px” tab_line_height=”1.3em” custom_margin=”|||0px|false|false” custom_padding=”|||0px|false|false”][et_pb_tab title=”Testing in the Wild” _builder_version=”4.4.8″]<\/p>\n
Test and evaluation (T&E) teams work with algorithm developers to outline criteria for quality control, and of course they can\u2019t anticipate all algorithmic outcomes. But the consequences (and even blame) for the unexpected results are sometimes transferred onto groups who are unaware of these limitations or have not consented to being test subjects.<\/p>\n
<\/p>\n<\/div>\n
![\"\"](\"https:\/\/sites.mitre.org\/aifails\/wp-content\/uploads\/sites\/15\/2020\/03\/icon_example_robot-002.png\"){kind=icon}
<\/p>\n
Boeing initially blamed foreign pilots for the 737 MAX crashes, even though a sensor malfunction, faulty software, lack of pilot training, making a safety feature an optional purchase, and not mentioning the software in the pilot manual were all contributory causes.1<\/a><\/p>\n In 2014, UK immigration rules required some foreigners to pass an English proficiency test. A voice recognition system was used as part of the exam to detect fraud (e.g., if an applicant took the test multiple times under different names, or if a native speaker took the oral test posing as the applicant). But because the government did not understand how high the algorithm\u2019s error rate was, and each flagged recording was checked by undertrained employees, the UK cancelled thousands of visas and deported people in error.2,<\/a>3<\/a> Thus, applicants who had followed the rules suffered the consequences of the shortcomings in the algorithm.<\/p>\n <\/a><\/p>\n<\/div>\n T&E of AI algorithms is hard. Even for AI models that aren\u2019t entirely black boxes we have only limited T&E tools4,<\/a>5<\/a> (though resources are emerging6,<\/a>7,<\/a>8<\/a>). Difficulties for T&E result from:<\/p>\n <\/a>Uncertain outcomes:<\/em> Many AI models are complex, not fully explainable, and potentially non-linear (meaning they behave in unexpected ways in response to unexpected inputs), and we don\u2019t have great tools to help us understand their decisions and limitations.9,<\/a>10,<\/a>11<\/a><\/p>\n Model drift: <\/em>Due to changes in data, the environment, or people\u2019s behavior an AI\u2019s performance will drift, or become outdated, over time.12,<\/a>13<\/a><\/p>\n Unanticipated use: <\/em>Because AI interacts with people who probably do not share our skills or understanding of the system, and who may not share our goals, the AI will be used in unanticipated ways.<\/p>\n Pressures to move quickly:<\/em> There is a tension between resolving to develop and deploy automated products quickly and taking time to test, understand, and address the limitations of those products.14<\/a><\/p>\n Because all these difficulties, deployers and consumers of AI models often don\u2019t know the range or severity of consequences of the AI\u2019s application.15<\/a><\/p>\n Jonathan Zittrain, Harvard Law School professor, describes how the issues that emerge from an unpredictable system will become problematic as the number of systems increases. He introduces the concept of \u201cintellectual debt,\u201d which applies to many fields, not only AI. For example, in medicine some drugs are approved for wide use even when \u201cno one knows exactly how they work,\u201d16<\/a> but they may still have value. If the unknowns were limited to only a single AI (or drug), then causes and effects might be isolated and mitigated. But as the number of AIs and their interactions with humans grows, performing the number of tests required to uncover potential consequences becomes logistically impossible.<\/p>\n <\/p>\n Users are held responsible for bad AI outcomes even if those outcomes aren\u2019t entirely (or at all) their fault. A lack of laws defining accountability and responsibility for AI means that it is too easy to blame the AI victim when something goes wrong. The default assumption in semi-autonomous vehicle crashes, as in the Boeing 737 MAX tragedies, has been that drivers are solely at fault.17,<\/a>18,<\/a>19,<\/a>20,<\/a>21<\/a> Similarly, reports on the 737 crashes showed that \u201call the risk [was put] on the pilot, who would be expected to know what to do within seconds if a system he didn\u2019t know existed\u2026 forced the plane downward.\u201d22<\/a> The early days of automated flying demonstrated that educating pilots about the automation capabilities and how to act as a member of a human-machine team reduced the number of crashes significantly.23,<\/a>24,<\/a>25<\/a><\/p>\n As a separate concern, the individuals or communities subject to an AI can become unwilling or unknowing test subjects. Pedestrians can unknowingly be injured by still-learning, semi-autonomous vehicles;26<\/a> oncology patients can be diagnosed by an experimental IBM Watson (Watson is in a trial phase and not yet approved for clinical use);27<\/a> Pearson can offer different messaging to different students as an experiment in gauging student engagement.28<\/a> As the AI Now Institute at New York University (a research institute dedicated to understanding the social implications of AI technologies) puts it, \u201cthis is a repeated pattern when market dominance and profits are valued over safety, transparency, and assurance.\u201d29<\/a><\/a><\/p>\n The early days of automated flying demonstrated that educating pilots about the automation capabilities and how to act as a member of a human-machine team reduced the number of crashes significantly<\/p>\n<\/blockquote>\n <\/p>\n <\/p>\nWhy is this a fail?<\/h3>\n
What happens when things fail?<\/h3>\n
\n