![\"\"](\"https:\/\/sites.mitre.org\/aifails\/wp-content\/uploads\/sites\/15\/2020\/03\/icon_launch-003.jpg\"){kind=icon}
\u200bFailure to Launch: How People Can React to AI<\/h1>\n[et_pb_section fb_built=”1″ specialty=”on” _builder_version=”4.0.3″][et_pb_column type=”1_4″ _builder_version=”3.25″ custom_padding=”|||” custom_padding__hover=”|||”][et_pb_sidebar area=”et_pb_widget_area_20″ admin_label=”Failure to Launch menu widget” _builder_version=”4.0.9″ min_height=”200px” custom_margin=”||0px||false|false” custom_padding=”||0px||false|false”][\/et_pb_sidebar][\/et_pb_column][et_pb_column type=”3_4″ specialty_columns=”3″ _builder_version=”3.25″ custom_padding=”|||” custom_padding__hover=”|||”][et_pb_row_inner _builder_version=”4.0.3″ custom_padding=”||0px||false|false”][et_pb_column_inner saved_specialty_column_type=”3_4″ _builder_version=”4.0.3″][et_pb_text admin_label=”Failure to Launch content” _builder_version=”4.4.8″ custom_margin=”||0px||false|false” custom_padding=”5px||0px||false|false”]<\/p>\n
\u200bFailure to Launch: How People Can React to AI<\/h1>\nPeople often hold multiple, contradictory views at the same time. There are plenty of examples when it comes to human interaction with technology: people can be excited that Amazon or Netflix recommendations really reflect their tastes, yet worry about what that means for their privacy; they can use Siri and Google voice to help them remember things, yet lament about losing their short-term memory; they can rely on various newsfeeds to give them information, even if they know (or suspect) that the primary goal of the algorithms behind those newsfeeds is to keep their attention, not to deliver the broadest news coverage. These seeming dichotomies all revolve around trust, which involves belief and understanding, dependency and choice, perception and evidence, emotion and context. All of these elements of trust are critical to having someone accept and adopt an AI. When we as AI developers and deployers include technical, cultural, organizational, sociological, interpersonal, psychological, and neurological perspectives, we can more accurately align people\u2019s trust in the AI to the actual trustworthiness of the AI, and thereby facilitate how people adopt of the AI.<\/p>\n
<\/p>\n
[\/et_pb_text][et_pb_text admin_label=”Fail Title” _builder_version=”4.4.8″ custom_margin=”||0px||false|false” custom_padding=”5px||0px||false|false”]<\/p>\n
[\/et_pb_text][et_pb_tabs active_tab_background_color=”#a0ddf3″ inactive_tab_background_color=”#e5e7e8″ active_tab_text_color=”#000000″ admin_label=”Fails and Lessons Learned content” module_class=”icon-tabs” _builder_version=”4.4.8″ tab_text_color=”#000000″ body_text_color=”#000000″ tab_font_size=”15px” tab_line_height=”1.3em” custom_margin=”|||0px|false|false” custom_padding=”|||0px|false|false”][et_pb_tab title=”In AI We Overtrust ” _builder_version=”4.4.8″]<\/p>\n
When people aren\u2019t familiar with AI, cognitive biases and external factors can prompt them to trust the AI more than they should. Even professionals can overtrust AIs deployed in their own fields. Worse, people can change their perceptions and beliefs to be more in line with an algorithm\u2019s, rather than the other way around.<\/p>\n
<\/p>\n<\/div>\n
![\"\"](\"https:\/\/sites.mitre.org\/aifails\/wp-content\/uploads\/sites\/15\/2020\/03\/icon_example_robot-002.png\"){kind=icon}
<\/p>\n
A research team put 42 test participants into a fire emergency scenario featuring a robot responsible for escorting them to an emergency exit. Even though the robot passed obvious exits and got lost, 37 participants continued to follow it.1,<\/a>2<\/a><\/p>\n Consumers who received a digital ad said they were more interested in a product that was specifically targeted for them, and even adjusted their own preferences to align with what the ad suggested about them.3<\/a><\/p>\n In a research experiment, students were told that a robot would determine who had pushed a button and \u201cbuzzed in\u201d first, thus winning a game. In reality, the robot tried to maximize participant engagement by evenly distributing who won. Even as the robot made noticeably inaccurate choices, the participants did not attribute the discrepancy to the robot having ulterior motives.4<\/a><\/p>\n<\/div>\n When an AI is helping people do things better than they would on their own, it is easy to assume that the platform\u2019s goals mirror the user\u2019s goals. However, there is no such thing as a \u201cneutral\u201d AI.5<\/a> During the design process we make conscious and unconscious assumptions about what the AI\u2019s goals and priorities should be and what data streams the AI should learn from. Lots of times, our incentives and user incentives align, so this works out wonderfully: users drive to their destinations, or they enjoy the AI-recommended movie. But when goals don\u2019t align, most users don\u2019t realize that they\u2019re potentially acting against their interests. They are convinced that they\u2019re making rational and objective decisions, because they are listening to a rational and objective AI.6<\/a><\/p>\n Furthermore, how users actually act and how they think they\u2019ll act often differs. For example, a journalist documented eight drivers in 2013 who overrode their own intuition and blindly followed their GPS, including drivers who turned onto the stairs of the entrance to a park, a driver who drove into a body of water, and another driver who ran straight into a house, all because of their interpretation of the GPS instructions.7<\/a><\/p>\n Users are convinced that they\u2019re making rational and objective decisions, because they are listening to a rational and objective AI<\/p>\n<\/blockquote>\n Numerous biases can contribute to overtrusting technology. Research highlights three prevalent ones:<\/p>\n 1. Humans can have a bias to assume automation is perfect; therefore, they have high initial trust.<\/em>8<\/a> This \u201cautomation bias\u201d leads users to trust automated and decision support systems even when it is unwarranted.<\/p>\n 2. Similarly, people generally believe something is true if it comes from an authority or expert, even if no supporting evidence is supplied.<\/em>9<\/a> In this case, the AI is perceived as the expert.<\/p>\n 3. Lastly, humans use mental short-cuts to make sense of complex information, which can lead to overtrusting an AI if it behaves in a way that conforms to our expectations, or if we have an unclear understanding of how the AI works.<\/em> Cathy O\u2019Neil, mathematician and author, writes that our relationship to data is similar to an ultimate belief in God: \u201cI think it has a few hallmarks of worship \u2013 we turn off parts of our brain, we somehow feel like it\u2019s not our duty, not our right to question this.\u201d10<\/a><\/p>\n Therefore, the more an AI is associated with a supposedly flawless, data-driven authority, the more likely that humans will overtrust the AI. In these conditions, even professionals in a given field can cede their authority despite their specialized knowledge.11,<\/a>12<\/a><\/p>\n Another outcome of overtrust is that the AI reinforces a tendency to align with the model\u2019s solution rather than the individual\u2019s own, pushing AI predictions to become self-fulfilling.13<\/a> These outcomes also show that having a human supervise an AI will not necessarily work as a failsafe.14<\/a><\/p>\n <\/p>\n The phenomenon of overtrust in AI has contributed to two powerful and potentially frightening outcomes. First, since AIs often have a single objective and reinforce increasingly specialized ends (see more in the “Feeding the Feedback Loop” Fail <\/a>) users aren\u2019t presented with alternative perspectives and are directed toward more individualistic, non-inclusive ways of thinking.<\/p>\n Second, the pseudo-authority of AI has allowed pseudosciences to re-emerge with a veneer of validity. Demonstrably invalid examples of AI have been used to look at a person\u2019s face and assess that person\u2019s tendencies toward criminality or violence,15,<\/a>16<\/a> current feelings,17<\/a> sexual orientation,18<\/a> and IQ or personality traits.19<\/a> These phrenology and physiognomy products and claims are unethical, irresponsible, and dangerous.<\/p>\n Although these outcomes may seem extreme, overtrust has a wide range of consequences, from causing people to act against self-interest to promulgating discriminatory practices.<\/p>\nWhy is this a fail?<\/h3>\n
\n
What happens when things fail?<\/h3>\n