{"id":4701,"date":"2020-03-16T10:14:45","date_gmt":"2020-03-16T10:14:45","guid":{"rendered":"https:\/\/sites.mitre.org\/aifails\/?page_id=4701"},"modified":"2020-07-13T08:38:40","modified_gmt":"2020-07-13T12:38:40","slug":"you-call-this-artificial-intelligence","status":"publish","type":"page","link":"https:\/\/sites.mitre.org\/aifails\/you-call-this-artificial-intelligence\/","title":{"rendered":"You Call This Artificial \u201cIntelligence\u201d? AI Meets the Real World"},"content":{"rendered":"\n

[et_pb_section fb_built=”1″ specialty=”on” _builder_version=”4.0.5″][et_pb_column type=”1_4″ _builder_version=”3.25″ custom_padding=”|||” custom_padding__hover=”|||”][et_pb_sidebar area=”et_pb_widget_area_41″ admin_label=”You Call This AI” _builder_version=”4.4.1″ min_height=”200px” custom_margin=”||0px||false|false” custom_padding=”||0px||false|false”][\/et_pb_sidebar][\/et_pb_column][et_pb_column type=”3_4″ specialty_columns=”3″ _builder_version=”3.25″ custom_padding=”|||” custom_padding__hover=”|||”][et_pb_row_inner _builder_version=”4.0.3″ custom_padding=”||0px||false|false”][et_pb_column_inner saved_specialty_column_type=”3_4″ _builder_version=”4.0.3″][et_pb_text admin_label=”You Call This Artificial \u201cIntelligence\u201d? ” _builder_version=”4.4.8″ custom_margin=”||0px||false|false” custom_padding=”0px||0px||false|false”]<\/p>\n

\"\"You Call This Artificial \u201cIntelligence\u201d? AI Meets the Real World<\/h1>\n

\u200bAI systems can perform specific, defined tasks so well that their capability can appear superhuman. For instance, AI can recognize common images and objects better than human beings, AI can sift through large amounts of data faster than human beings, and AI can master more languages than human beings.1<\/a> However, it is important to remember that an AI\u2019s success is task specific and AI\u2019s ability to complete a task \u2013 such as recognizing images \u2013 is contingent on the data it receives and the environment it operates in. Because of this, sometimes AI applications are fooled in ways that humans never would be, particularly if these systems encounter situations beyond their abilities. The examples below describe situations where environmental factors exceeded AI\u2019s \u201csuperhuman\u201d capabilities and invalidated any contingency planning that developers or deployers introduced.<\/p>\n

 [\/et_pb_text][et_pb_text admin_label=”Fails” _builder_version=”4.4.8″ custom_margin=”||0px||false|false” custom_padding=”5px||0px||false|false”]<\/p>\n

\n\u00a0\n<\/div>\n
Explore the Three Fails in This Category:<\/h5>\n

[\/et_pb_text][et_pb_tabs active_tab_background_color=”#a0ddf3″ inactive_tab_background_color=”#e5e7e8″ active_tab_text_color=”#000000″ admin_label=”Fail Why and Examples” module_class=”icon-tabs” _builder_version=”4.4.8″ tab_text_color=”#000000″ body_text_color=”#000000″ tab_font_size=”15px” tab_line_height=”1.3em” custom_margin=”|||0px|false|false” custom_padding=”|||0px|false|false”][et_pb_tab title=”Sensing is Believing” _builder_version=”4.4.8″ tab_line_height=”1em”]<\/p>\n

\n

Sensing is Believing<\/h3>\n

When sensors are faulty, or the code that interprets the data is faulty, the result can be extremely damaging.<\/p>\n

 <\/p>\n<\/div>\n

\n

\"\"<\/p>\n

Examples<\/h3>\n

When the battery died on early versions of \u201csmart\u201d thermostats, houses got really, really cold.2<\/a> Later versions had appropriate protections built into the code to ensure this wouldn\u2019t happen.<\/p>\n

A preliminary analysis of the Boeing 737 MAX airline crashes found that a faulty sensor \u201cerroneously reported that the airplane was stalling\u2026 which triggered an automated system\u2026 to point the aircraft\u2019s nose down,\u201d when the aircraft was not actually stalling.3<\/a> Boeing subsequently included the safety features that would have alerted pilots to the disagreement between working sensors and the failed sensor to all models.<\/p>\n

A woman discovered that any person\u2019s fingerprint could unlock her phone\u2019s \u201cvault-like security\u201d after she had fitted the phone with a $3 screen protector. Customers were told to avoid logging in through fingerprint until the vendor could fix the code.4<\/a><\/p>\n<\/div>\n

Why is this a fail?<\/h3>\n

Humans use sight, smell, hearing, taste, and touch to perceive and make sense of the world. These senses work in tandem and can serve as backups for one another; for instance, you might smell smoke before you see it. But human senses and processing aren\u2019t perfect; they can be influenced, be confused, or degrade.<\/p>\n

 <\/p>\n

What happens when things fail?<\/h3>\n

Similar to humans, some automated systems rely on sensors to get data about their operating environments and rely on code to process and act on that data. And like human senses, these sensors and the interpretation of their readings are imperfect and can be influenced<\/em> by the composition or labelling of the training dataset, can get confused<\/em> by erroneous or unexpected inputs, and can degrade<\/em> as parts get older. AI applications tend to break if we haven\u2019t included redundancy, guardrails to control behavior, or code to gracefully deal with programming errors.<\/p>\n

We can learn from a long history of research on sensor failure, for example in the automobile, power production, manufacturing, and aviation industries. In the latter case research findings have led to certification requirements like triple redundancy for any parts on an aircraft necessary for flight.5<\/a><\/p>\n

 <\/p>\n

 <\/p>\n\n\n \n \n \n \n \n \n \n
<\/th>\n <\/tr>\n
<\/th>\n <\/tr>\n
<\/th>\n <\/th>\n <\/th>\n <\/th>\n <\/tr>\n
Hold AI to a Higher Standard<\/a><\/td>\n Involve the Communities Affected by the AI <\/a><\/td>\n Make Our Assumptions Explicit<\/a><\/td>\n Monitor the AI’s Impact and Establish Layers of Accountability<\/a><\/td>\n <\/tr>\n
It’s OK to Say No to Automation<\/a><\/td>\n Plan to Fail<\/a><\/td>\n Try Human-AI Couples Counseling<\/a><\/td>\n Envision Safeguards for AI Advocates<\/a><\/td>\n <\/tr>\n
AI Challenges are Multidisciplinary, so They Require a Multidisciplinary Team<\/a><\/td>\n Ask for Help: Hire a Villain<\/a><\/td>\n Offer the User Choices<\/a><\/td>\n Require Objective, Third-party Verification and Validation\u00a0<\/a><\/td>\n <\/tr>\n
Incorporate Privacy, Civil Liberties, and Security from the Beginning<\/a><\/td>\n Use Math to Reduce Bad Outcomes Caused by Math<\/a><\/td>\n Promote Better Adoption through Gameplay<\/a><\/td>\n Entrust Sector-specific Agencies to Establish AI Standards for Their Domains\u00a0<\/a><\/td>\n <\/tr>\n<\/tbody>\n<\/table>\n

[\/et_pb_tab][et_pb_tab title=”Insecure AI ” _builder_version=”4.4.8″ tab_line_height=”1em”]<\/p>\n

\n

Insecure AI<\/h3>\n

When AI\u2019s software and information technology (IT) architecture are not hardened against cybersecurity threats, users and systems are vulnerable to accidental or malicious interference.<\/p>\n<\/div>\n

\n

\"\"<\/p>\n

Examples<\/h3>\n

Responding to what it thought were explicit commands but was actually background noise, an Amazon Echo recorded a family\u2019s private conversation and sent it to a random user.2<\/a> This is one way in which users can unknowingly cause data spills.<\/p>\n

HIS Group, a Japanese hotel chain, installed in-room cameras with facial recognition and speech recognition to cater to guest\u2019s needs. Hackers were able to remotely view video streams.3<\/a> This is one way purchasers can unintentionally create situations that attract malicious behavior.<\/p>\n<\/div>\n

Why is this a fail?<\/h3>\n

AI\u2019s software and IT architecture are as vulnerable to cybersecurity threats as other connected technologies \u2013 and potentially vulnerable in new ways as well (see more in the “AI Pwned” Fail). Just deploying an AI into the world introduces it as a new attack surface (i.e., something to attack).4<\/a> Even the most secure AI can face continuous attacks that aim to expose, alter, disable, destroy, or gain unauthorized access to it. Therefore, we must design all software systems in a way that makes cyber protections and privacy considerations inherent to the design from the beginning.5<\/a><\/p>\n

\n

AI systems often have access to potentially sensitive user information.<\/p>\n<\/blockquote>\n

<\/h3>\n

What happens when things fail?<\/h3>\n

Smart devices \u2013 like internet-connected speakers, wireless door locks, and wireless implants \u2013 have increasingly been introduced into people\u2019s homes and even into their bodies, which makes the consequences of their being hacked especially terrifying.6,<\/a>7,<\/a>8<\/a> Such systems are often networked because they rely on cloud resources to do some of the processing, or they communicate with other networked sensors. The market growth of these kind of products will make such devices more common.<\/p>\n

Another cybersecurity threat arises because AI systems often have access to potentially sensitive user information. For example, smart home devices have an unusual level of access, including contact lists, conversations, voice signatures, and times when someone is home. Any system that collects GPS data can recreate a detailed picture of someone\u2019s location and movement patterns.9<\/a> Physical AI systems that provide critical capabilities, such as autonomous vehicles, could become targets for attacks that would put a person\u2019s safety at risk.10<\/a> Finally, existing methods of de-identifying individuals from their personal data have been shown to be ineffective (although researchers are working on this challenge).11<\/a> As organizations seek to collect more data for their algorithms, the rewards for stealing this information grow as well.<\/p>\n

<\/a><\/p>\n

 <\/p>\n\n\n \n \n \n \n \n \n \n
<\/th>\n <\/tr>\n
<\/th>\n <\/tr>\n
<\/th>\n <\/th>\n <\/th>\n <\/th>\n <\/tr>\n
Hold AI to a Higher Standard<\/a><\/td>\n Involve the Communities Affected by the AI <\/a><\/td>\n Make Our Assumptions Explicit<\/a><\/td>\n Monitor the AI’s Impact and Establish Layers of Accountability<\/a><\/td>\n <\/tr>\n
It’s OK to Say No to Automation<\/a><\/td>\n Plan to Fail<\/a><\/td>\n Try Human-AI Couples Counseling<\/a><\/td>\n Envision Safeguards for AI Advocates<\/a><\/td>\n <\/tr>\n
AI Challenges are Multidisciplinary, so They Require a Multidisciplinary Team<\/a><\/td>\n Ask for Help: Hire a Villain<\/a><\/td>\n Offer the User Choices<\/a><\/td>\n Require Objective, Third-party Verification and Validation\u00a0<\/a><\/td>\n <\/tr>\n
Incorporate Privacy, Civil Liberties, and Security from the Beginning<\/a><\/td>\n Use Math to Reduce Bad Outcomes Caused by Math<\/a><\/td>\n Promote Better Adoption through Gameplay<\/a><\/td>\n Entrust Sector-specific Agencies to Establish AI Standards for Their Domains\u00a0<\/a><\/td>\n <\/tr>\n<\/tbody>\n<\/table>\n

[\/et_pb_tab][et_pb_tab title=”AI Pwned” _builder_version=”4.4.8″ tab_line_height=”1em”]<\/p>\n

\n

AI Pwned<\/h3>\n

Malicious actors can fool an AI or get it to reveal protected information.<\/p>\n

Note: <\/em>\u201cPwned\u201d is a computer-slang term that means \u201cto own\u201d or to completely get the better of an opponent or rival.2<\/a><\/p>\n

 <\/p>\n<\/div>\n

\n

\"\"<\/p>\n

Examples<\/h3>\n

Researchers created eyeglasses whose frames had a special pattern that defeats facial recognition algorithms by executing targeted (impersonation of another person) or untargeted (avoiding identification) attacks on the algorithms.3<\/a> A human being would easily be able to identify the person correctly.<\/p>\n

Researchers explored a commercial facial recognition system that used a picture of a face as input, searched its database, and outputted the name of the person with the closest matching face (and a confidence score in that match). Over time, the researchers discovered information about the individual faces the system had been trained on \u2013 information they should not have had access to. They then built their own AI system that, when supplied with a person\u2019s name, returned an imperfect image of the person, revealing data that had never been made public and should not have been.4<\/a> This kind of attack illustrates that the sensitive information used for training an AI may not be as well protected as desired.<\/p>\n<\/div>\n

Why is this a fail?<\/h3>\n

Cyber-attacks that target AI systems are called \u201cadversarial AI.\u201d AI may not have the defenses to prevent malicious actors from fooling the algorithm into doing what they want, or from interfering with the data on which the model trains, all without making any changes to the algorithm or gaining access to the code. At the most basic level, adversaries present lots of input to the AI and monitor what it does in response, so that they can track how the model makes very specific decisions. Adversaries can then very slightly alter the input so that a human cannot tell the difference, but the AI has great confidence in its wrong conclusion.5<\/a> Adversaries can also extract sensitive information about individual elements of the training sets6<\/a> or adversaries can make assumptions about which data sources are used and then insert data to bias the learning process.7<\/a><\/p>\n

<\/a><\/p>\n

What happens when things fail?<\/h3>\n

The results can have serious real-world consequences. Researchers have demonstrated examples of a self-driving car not \u201cseeing\u201d a stop sign8<\/a> and Google Home interpreting a greeting as a command to unlock the front door.9<\/a> Researchers have also documented a hacker\u2019s ability to identify and decipher an individual\u2019s healthcare records from a published database of de-identified names.10<\/a><\/p>\n

Pwning an AI is particularly powerful because 1) it is invisible to humans, so it is hard to detect; 2) it scales, so that a method to fool one AI can often trick other AIs; and 3) it works.11<\/a><\/p>\n

 <\/p>\n

 <\/p>\n\n\n \n \n \n \n \n \n \n
<\/th>\n <\/tr>\n
<\/th>\n <\/tr>\n
<\/th>\n <\/th>\n <\/th>\n <\/th>\n <\/tr>\n
Hold AI to a Higher Standard<\/a><\/td>\n Involve the Communities Affected by the AI <\/a><\/td>\n Make Our Assumptions Explicit<\/a><\/td>\n Monitor the AI’s Impact and Establish Layers of Accountability<\/a><\/td>\n <\/tr>\n
It’s OK to Say No to Automation<\/a><\/td>\n Plan to Fail<\/a><\/td>\n Try Human-AI Couples Counseling<\/a><\/td>\n Envision Safeguards for AI Advocates<\/a><\/td>\n <\/tr>\n
AI Challenges are Multidisciplinary, so They Require a Multidisciplinary Team<\/a><\/td>\n Ask for Help: Hire a Villain<\/a><\/td>\n Offer the User Choices<\/a><\/td>\n Require Objective, Third-party Verification and Validation\u00a0<\/a><\/td>\n <\/tr>\n
Incorporate Privacy, Civil Liberties, and Security from the Beginning<\/a><\/td>\n Use Math to Reduce Bad Outcomes Caused by Math<\/a><\/td>\n Promote Better Adoption through Gameplay<\/a><\/td>\n Entrust Sector-specific Agencies to Establish AI Standards for Their Domains\u00a0<\/a><\/td>\n <\/tr>\n<\/tbody>\n<\/table>\n

[\/et_pb_tab][\/et_pb_tabs][\/et_pb_column_inner][\/et_pb_row_inner][\/et_pb_column][\/et_pb_section][et_pb_section fb_built=”1″ fullwidth=”on” disabled_on=”off|off|off” _builder_version=”4.4.8″ module_alignment=”center” global_module=”3880″ saved_tabs=”all”][et_pb_fullwidth_code disabled_on=”off|off|off” admin_label=”Footer menu” _builder_version=”4.4.8″ background_color=”#d5dde0″ text_orientation=”center” module_alignment=”center” custom_padding=”10px||10px||false|false”]Add Your Experience! This site should be a community resource and would benefit from the addition of other examples and voices. You can write to us by clicking here<\/a>.[\/et_pb_fullwidth_code][\/et_pb_section]<\/p>\n","protected":false},"excerpt":{"rendered":"

You Call This Artificial \u201cIntelligence\u201d? AI Meets the Real World \u200bAI systems can perform specific, defined tasks so well that their capability can appear superhuman. For instance, AI can recognize common images and objects better than human beings, AI can sift through large amounts of data faster than human beings, and AI can master more […]<\/p>\n","protected":false},"author":142,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_acf_changed":false,"_et_pb_use_builder":"on","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"class_list":["post-4701","page","type-page","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/sites.mitre.org\/aifails\/wp-json\/wp\/v2\/pages\/4701","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sites.mitre.org\/aifails\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/sites.mitre.org\/aifails\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/sites.mitre.org\/aifails\/wp-json\/wp\/v2\/users\/142"}],"replies":[{"embeddable":true,"href":"https:\/\/sites.mitre.org\/aifails\/wp-json\/wp\/v2\/comments?post=4701"}],"version-history":[{"count":0,"href":"https:\/\/sites.mitre.org\/aifails\/wp-json\/wp\/v2\/pages\/4701\/revisions"}],"wp:attachment":[{"href":"https:\/\/sites.mitre.org\/aifails\/wp-json\/wp\/v2\/media?parent=4701"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}