{"id":65,"date":"2019-09-12T14:01:53","date_gmt":"2019-09-12T14:01:53","guid":{"rendered":"https:\/\/sites.mitre.org\/aifails\/?page_id=65"},"modified":"2020-07-13T20:12:27","modified_gmt":"2020-07-14T00:12:27","slug":"references","status":"publish","type":"page","link":"https:\/\/sites.mitre.org\/aifails\/references\/","title":{"rendered":"References"},"content":{"rendered":"\n

[et_pb_section fb_built=”1″ _builder_version=”3.29.3″ background_color=”rgba(0,0,0,0)”][et_pb_row column_structure=”1_4,3_4″ admin_label=”row” module_class=”.footnote_plugin_tooltip_text” _builder_version=”4.4.1″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” border_color_right=”#000000″][et_pb_column type=”1_4″ _builder_version=”3.25″ custom_padding=”|||” custom_padding__hover=”|||”][et_pb_sidebar area=”et_pb_widget_area_10″ admin_label=”References” _builder_version=”4.4.8″][\/et_pb_sidebar][\/et_pb_column][et_pb_column type=”3_4″ _builder_version=”3.25″ custom_padding=”|||” custom_padding__hover=”|||”][et_pb_text disabled_on=”off|off|off” _builder_version=”4.5.0″ text_line_height=”1.2em” content__hover_enabled=”off|desktop”]<\/p>\n

References<\/h1>\n

\u00a0<\/h3>\n

Home Page<\/strong><\/em><\/h3>\n

<\/a>
\n1. \u201cBenjamin Franklin quotable quote,\u201d Goodreads. Accessed March 16, 2020. [Online]. Available: https:\/\/www.goodreads.com\/quotes\/460142-if-you-fail-to-plan-you-are-planning-to-fail<\/a><\/p>\n

<\/a>
\n2. Department of Defense, \u201cSummary of the 2018 Department of Defense Artificial Intelligence Strategy: Harnessing AI to Advance Our Security and Prosperity,\u201d defense.gov,<\/em> February 12, 2019. [Online]. Available: https:\/\/media.defense.gov\/2019\/Feb\/12\/2002088963\/-1\/-1\/1\/SUMMARY-OF-DOD-AI-STRATEGY.PDF<\/a><\/p>\n

<\/a>
\n3. ichristianization, \u201cMicrosoft build 2017 translator demo,\u201d YouTube, June 13, 2017. [Online]. Available: https:\/\/www.youtube.com\/watch?v=u4cJoX-DoiY<\/a><\/p>\n

<\/a>
\n4. N. Martin, \u201cArtificial intelligence is being used to diagnose disease and design new drugs,\u201d Forbes<\/em>, Sept. 30, 2019. [Online]. Available: https:\/\/www.forbes.com\/sites\/nicolemartin1\/2019\/09\/30\/artificial-intelligence-is-being-used-to-diagnose-disease-and-design-new-drugs\/#8874c44db51f<\/a><\/p>\n

<\/a>
\n5. \u201cMeet the AI robots helping take care of elderly patients,\u201d Time Magazine<\/em>, Aug. 23, 2019. [Online]. Available: https:\/\/time.com\/5660046\/robots-elderly-care\/<\/a><\/p>\n

<\/a>
\n6. A. Chang, \u201cThe Facebook and Cambridge Analytica scandal, explained with a simple diagram,\u201d Vox<\/em>, May 2, 2018. [Online]. Available: https:\/\/www.vox.com\/policy-and-politics\/2018\/3\/23\/17151916\/facebook-cambridge-analytica-trump-diagram<\/a><\/p>\n

<\/a>
\n7. P. Taddonio, \u201cHow China\u2019s government is using AI on its Uighur Muslim population,\u201d Frontline<\/em>, Nov. 21, 2019. [Online]. Available: https:\/\/www.pbs.org\/wgbh\/frontline\/article\/how-chinas-government-is-using-ai-on-its-uighur-muslim-population\/<\/a><\/p>\n

<\/a>
\n8. D. Z. Morris, \u201cChina will block travel for those with bad \u2018social credit,\u2019\u201d Fortune<\/em>, March 18, 2018. [Online]. Available: https:\/\/fortune.com\/2018\/03\/18\/china-travel-ban-social-credit\/<\/a><\/p>\n

<\/a>
\n9. R. Adams, \u201cHong Kong protesters are worried about facial recognition technology. But there are many other ways they’re being watched,\u201d BuzzFeed News<\/em>, Aug. 17, 2019. [Online]. Available: https:\/\/www.buzzfeednews.com\/article\/rosalindadams\/hong-kong-protests-paranoia-facial-recognition-lasers<\/a><\/p>\n

<\/a>
\n10. S. Gibbs, \u201cTesla Model S cleared by auto safety regulator after fatal Autopilot crash,\u201d Guardian<\/em>, Jan. 20, 2017. [Online]. Available: https:\/\/www.theguardian.com\/technology\/2017\/jan\/20\/tesla-model-s-cleared-auto-safety-regulator-after-fatal-autopilot-crash<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

Fails<\/em><\/strong><\/h3>\n

No Human Needed: the AI\u2019s Got This<\/strong><\/p>\n

<\/a>
\n1. E. Hunt, \u201cTay, Microsoft’s AI chatbot, gets a crash course in racism from Twitter,\u201d Guardian<\/em>, March 24, 2016.\u00a0[Online]. Available: https:\/\/www.theguardian.com\/technology\/2016\/mar\/24\/tay-microsofts-ai-chatbot-gets-a-crash-course-in-racism-from-twitter<\/a><\/p>\n

<\/a>
\n2. C. Lecher, \u201cHow Amazon automatically tracks and fires warehouse workers for \u2018productivity,\u2019\u201d The Verge<\/em>, Apr. 25, 2019. [Online]. Available: https:\/\/www.theverge.com\/2019\/4\/25\/18516004\/amazon-warehouse-fulfillment-centers-productivity-firing-terminations<\/a><\/p>\n

<\/a>
\n3. C. F. de Winter & D. Dodou, \u201cWhy the Fitts list has persisted throughout the history of function allocation,\u201d SpringerLink<\/em>, August 25, 2011. [Online]. Available: https:\/\/link.springer.com\/article\/10.1007\/s10111-011-0188-1<\/a><\/p>\n

<\/a>
\n4. E. Brynjolfsson and A. McAfee, The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies. New York, NY, USA: W. W. Norton, 2016.<\/p>\n

<\/a>
\n5. M. Johnson, J. M. Bradshaw, R. R. Hoffman, P. J. Feltovich, and D. D. Woods, \u201cSeven cardinal virtues of human-machine teamwork: Examples from the DARPA robotic challenge,\u201d IEEE Intelligent Systems<\/em>, Nov.\/Dec. 2014. [Online]. Available: http:\/\/www.jeffreymbradshaw.net\/publications\/56.%20Human-Robot%20Teamwork_IEEE%20IS-2014.pdf<\/a><\/p>\n

<\/a>
\n6. N. D. Sarter, D. D. Woods, and C. E. Billings, \u201cAutomation surprises,\u201d in G.\u00a0 Salvendy\u00a0 (Ed.),\u00a0 Handbook of Human Factors & Ergonomics (2nd ed., pp. 1926-1943). New York, NY, USA: John Wiley, 1997.<\/p>\n

<\/a>
\n7. S. M. Casner and E. L. Hutchins, \u201cWhat do we tell the drivers? Toward minimum driver training standards for partially automated cars,\u201d Journal of Cognitive Engineering and Decision Making<\/em>, March 8, 2019. [Online]. Available: https:\/\/journals.sagepub.com\/doi\/full\/10.1177\/1555343419830901<\/a><\/p>\n

<\/a>
\n8. D. Sarter, D. D. Woods, and C. E. Billings, \u201cAutomation surprises,\u201d in G. Salvendy (Ed.), Handbook of Human Factors & Ergonomics<\/em> (2nd ed., pp. 1926-1943). New York, NY, USA: John Wiley, 1997.<\/p>\n

<\/a>
\n9. S. M. Casner and E. L. Hutchins, \u201cWhat do we tell the drivers? Toward minimum driver training standards for partially automated cars,\u201d Journal of Cognitive Engineering and Decision Making<\/em>, March 8, 2019. [Online]. Available: https:\/\/journals.sagepub.com\/doi\/full\/10.1177\/1555343419830901<\/a><\/p>\n

<\/a>
\n10. T. Lewis, \u201cA brief history of artificial intelligence,\u201d Live Science<\/em>, Dec. 4, 2014. [Online]. Available: https:\/\/www.livescience.com\/49007-history-of-artificial-intelligence.html<\/a><\/p>\n

<\/a>
\n11. Data & Society, \u201cAlgorithmic accountability: A primer,\u201d Tech Algorithm Briefing: How Algorithms Perpetuate Racial Bias and Inequality, prepared for the Congressional Progressive Caucus, April 18, 2018. [Online]. Available: https:\/\/datasociety.net\/wp-content\/uploads\/2018\/04\/Data_Society_Algorithmic_Accountability_Primer_FINAL-4.pdf<\/a><\/p>\n

<\/a>
\n12. T. D. Jajal, \u201cDistinguishing between narrow AI, general AI and super AI,\u201d Medium, May 21, 2018. [Online]. Available: https:\/\/medium.com\/@tjajal\/distinguishing-between-narrow-ai-general-ai-and-super-ai-a4bc44172e22<\/a><\/p>\n

\u00a0<\/strong><\/p>\n

\u00a0<\/h3>\n

AI Perfectionists and AI \u201cPixie Dusters\u201d<\/strong><\/p>\n

<\/a>
\n1. J. Dastin, \u201cAmazon scraps secret AI recruiting tool that showed bias against women,\u201d Reuters<\/em>, Oct. 9, 2018. [Online]. Available: https:\/\/www.reuters.com\/article\/us-amazon-com-jobs-automation-insight\/amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK08G<\/a><\/p>\n

<\/a>
\n2. Defense Science Board, Task Force Report: The Role of Autonomy in DoD Systems<\/em>, Washington, D.C., June 2016. [Online]. Available: https:\/\/www.hsdl.org\/?abstract&did=722318<\/a><\/p>\n

<\/a>
\n3. M. McDonough, \u201cBusiness-focus on artificial intelligence rising,\u201d Twitter, Feb. 28, 2017. [Online]. Available: https:\/\/twitter.com\/M_McDonough\/status\/836580294484451328<\/a><\/p>\n

<\/a>
\n4. C. O\u2019Neil, \u201cThe era of blind faith in big data must end,\u201d TED, April 2017. [Online]. Available: https:\/\/www.ted.com\/talks\/cathy_o_neil_the_era_of_blind_faith_in_big_data_must_end<\/a><\/p>\n

<\/a>
\n5. \u201cHere to help,\u201d xkcd<\/em>. Accessed March 18, 2020. [Online]. Available: https:\/\/www.xkcd.com\/1831\/<\/a><\/p>\n

<\/a>
\n6. J. Brownlee, \u201cA gentle introduction to transfer learning for deep learning,\u201d Machine Learning Mastery, Sept. 16, 2019. [Online]. Available: https:\/\/machinelearningmastery.com\/transfer-learning-for-deep-learning\/<\/a><\/p>\n

<\/a>
\n7. S. Schuchmann, \u201cHistory of the second AI winter,\u201d towards data science, May 12, 2019. [Online]. Available: https:\/\/towardsdatascience.com\/history-of-the-second-ai-winter-406f18789d45<\/a><\/p>\n

<\/a>
\n8. Defense Science Board, Task Force Report: The Role of Autonomy in DoD Systems<\/em>, Washington, D.C., June 2016. [Online]. Available: https:\/\/www.hsdl.org\/?abstract&did=722318<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

Developers Are Wizards and Operators Are Muggles<\/strong><\/p>\n

<\/a>
\n1. A. Gregg, J. O’Connell, A. Ba Tran, and F. Siddiqui. \u201cAt tense meeting with Boeing executives, pilots fumed about being left in dark on plane software,\u201d Washington Post<\/em>, March 13, 2019. [Online]. Available: https:\/\/www.washingtonpost.com\/business\/economy\/new-software-in-boeing-737-max-planes-under-scrutinty-after-second-crash\/2019\/03\/13\/06716fda-45c7-11e9-90f0-0ccfeec87a61_story.html<\/a><\/p>\n

<\/a>
\n2. A. MacGillis, \u201cThe case against Boeing,\u201d New Yorker<\/em>, Nov. 11, 2019. [Online]. Available: https:\/\/www.newyorker.com\/magazine\/2019\/11\/18\/the-case-against-boeing<\/a><\/p>\n

<\/a>
\n3. P. McCausland, \u201cSelf-driving Uber car that hit and killed woman did not recognize that pedestrians jaywalk,\u201c NBC News<\/em>, Nov. 9, 2019. [Online]. Available: https:\/\/www.nbcnews.com\/tech\/tech-news\/self-driving-uber-car-hit-killed-woman-did-not-recognize-n1079281<\/a><\/p>\n

<\/a>
\n4. M. McFarland, \u201cMy seat keeps vibrating. Will it make me a better driver before driving me insane?\u201d Washington Post<\/em>, Jan. 12, 2015. [Online]. Available: https:\/\/www.washingtonpost.com\/news\/innovations\/wp\/2015\/01\/12\/my-seat-keeps-vibrating-will-it-make-me-a-better-driver-before-driving-me-insane\/?noredirect=on&utm_term=.31792eb87c03<\/a><\/p>\n

<\/a>
\n5. M. Cyril, \u201cWatching the Black body,\u201d Electronic Frontier Foundation, Feb. 28, 2019. [Online]. Available: https:\/\/www.eff.org\/deeplinks\/2019\/02\/watching-black-body<\/a><\/p>\n

<\/a>
\n6. Barry Friedman: Is technology making police better\u2014or\u2026\u201d Recode Decode podcast<\/em>, Nov. 24, 2019. [Online]. Available: https:\/\/www.stitcher.com\/podcast\/vox\/recode-decode\/e\/65519494?curator=MediaREDEF<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

You Call This Artificial \u201cIntelligence\u201d? AI Meets the Real World<\/strong><\/p>\n

<\/a>
\n1. R. Steinberg, \u201c6 areas where artificial neural networks outperform humans,\u201d Venture Beat<\/em>, Dec. 8, 2017. [Online]. Available: https:\/\/venturebeat.com\/2017\/12\/08\/6-areas-where-artificial-neural-networks-outperform-humans\/<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

Sensing is Believing<\/strong><\/p>\n

1. Reference 1 in this section comes from the category header: “You Call This Artificial \u201cIntelligence\u201d? AI Meets the Real World”<\/p>\n

<\/a>
\n2. Bilton, \u201cNest thermostat glitch leaves users in the cold,\u201d New York Times<\/em>, Jan. 13, 2016. [Online]. Available: https:\/\/www.nytimes.com\/2016\/01\/14\/fashion\/nest-thermostat-glitch-battery-dies-software-freeze.html<\/a><\/p>\n

<\/a>
\n3. A. J. Hawkins, \u201cEverything you need to know about the Boeing 737 Max airplane crashes,\u201d The Verge<\/em>, March 22, 2019. [Online]. Available: https:\/\/www.theverge.com\/2019\/3\/22\/18275736\/boeing-737-max-plane-crashes-grounded-problems-info-details-explained-reasons<\/a><\/p>\n

<\/a>
\n4. E. Ongweso, \u201cSamsung Galaxy S10 \u2018vault-like security\u2019 beaten by a $3 screen protector,\u201d Vice<\/em>, Oct. 17, 2019. [Online]. Available: https:\/\/www.vice.com\/en_us\/article\/59nqdb\/samsung-galaxy-s10-vault-like-security-beaten-by-a-dollar3-screen-protector<\/a><\/p>\n

<\/a>
\n5. \u201cAirplane redundancy systems\u201d Poente Technical. <\/em>Accessed April 3, 2020. [Online]. Available: https:\/\/www.poentetechnical.com\/aircraft-engineer\/airplane-redundancy-systems\/<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

Insecure AI<\/strong><\/p>\n

1. Reference 1 in this section comes from the category header: “You Call This Artificial \u201cIntelligence\u201d? AI Meets the Real World”<\/p>\n

<\/a>
\n2. AJ Vicens, \u201cAn Amazon Echo recorded a family\u2019s private conversation and sent it to some random person,\u201d Mother Jones<\/em>, May 24, 2018. [Online]. Available: https:\/\/www.motherjones.com\/politics\/2018\/05\/an-amazon-echo-recorded-a-familys-private-conversation-and-sent-it-to-some-random-person\/<\/a><\/p>\n

<\/a>
\n3. J. Oates, \u201cJapanese hotel chain sorry that hackers may have watched guests through bedside robots,\u201d Register<\/em>, Oct. 22, 2019. [Online]. Available: https:\/\/www.theregister.co.uk\/2019\/10\/22\/japanese_hotel_chain_sorry_that_bedside_robots_may_have_watched_guests<\/a><\/p>\n

<\/a>
\n4. T. G. Dietterich and E. J. Horvitz, \u201cRise of Concerns about AI: Reflections and Directions,\u201d Communications of the ACM<\/em>, vol. 58, no. 10, pp. 38-40, October 2015. [Online]. Available: http:\/\/erichorvitz.com\/CACM_Oct_2015-VP.pdf<\/a><\/p>\n

<\/a>
\n5. J. S. McEwen and S. S. Shapiro, \u201cMITRE\u2019S Privacy Engineering Tools and Their Use in a Privacy Assessment Framework,\u201d The MITRE Corporation, McLean, VA, Nov. 2019. [Online]. Available: https:\/\/www.mitre.org\/publications\/technical-papers\/mitre%E2%80%99s-privacy-engineering-tools-and-their-use-in-a-privacy<\/a><\/p>\n

<\/a>
\n6. University of Michigan Engineering, \u201cWatch engineers hack a \u2018smart home\u2019 door lock,\u201d YouTube, May 2, 2016. [Online]. Available: https:\/\/www.youtube.com\/watch?v=Iwm6nvC9Xhc<\/a><\/p>\n

<\/a>
\n7. M. Hanrahan, \u201cRing security camera hacks see homeowners subjected to racial abuse, ransom demands,\u201d ABC News<\/em>, Dec. 12, 2019. [Online]. Available: https:\/\/abcnews.go.com\/US\/ring-security-camera-hacks-homeowners-subjected-racial-abuse\/story?id=67679790<\/a><\/p>\n

<\/a>
\n8. “Cybersecurity Vulnerabilities Affecting Medtronic Implantable Cardiac Devices, Programmers, and Home Monitors: FDA Safety Communication.\u201d US Food & Drug Administration, March 2019. [Online]. Available: https:\/\/www.fda.gov\/medical-devices\/safety-communications\/cybersecurity-vulnerabilities-affecting-medtronic-implantable-cardiac-devices-programmers-and-home<\/a><\/p>\n

<\/a>
\n9. J. Herrman, \u201cGoogle knows where you\u2019ve been but does it know who you are,\u201d New York Times Magazine<\/em>, Sept. 12, 2018. [Online]. Available: https:\/\/www.nytimes.com\/2018\/09\/12\/magazine\/google-maps-location-data-privacy.html<\/a><\/p>\n

<\/a>
\n10. A. Greenberg, \u201cHackers remotely kill a Jeep on the highway\u2014with me in it,\u201d Wired<\/em>, July 21, 2015. [Online]. Available: https:\/\/www.wired.com\/2015\/07\/hackers-remotely-kill-jeep-highway\/<\/a><\/p>\n

<\/a>
\n11. L. Rocher, J. M. Hendrickx, and Y.-A. de Montjoye, \u201cEstimating the success of re-identifications in incomplete datasets using generative models,\u201d Nature Communications<\/em>, July 23, 2019. [Online]. Available: https:\/\/www.nature.com\/articles\/s41467-019-10933-3<\/a><\/p>\n

\u00a0<\/strong><\/p>\n

 <\/p>\n

AI Pwned<\/strong><\/p>\n

1. Reference 1 in this section comes from the category header: “You Call This Artificial \u201cIntelligence\u201d? AI Meets the Real World”<\/p>\n

<\/a>
\n2.\u00a0\u201cpwned,\u201d Urban Dictionary<\/em>. Accessed on: March 11, 2020. [Online]. Available: https:\/\/www.urbandictionary.com\/define.php?term=pwned<\/a><\/p>\n

<\/a>
\n3. M. Sharif, S. Bhagavatula, L. Bauer, and M. K. Reiter, \u201cA general framework for adversarial examples with objectives,\u201d arXiv.org<\/em>, April 4, 2019. [Online]. Available: https:\/\/arxiv.org\/abs\/1801.00349<\/a><\/p>\n

<\/a>
\n4. M. Fredrikson, S. Jha, T. Ristenpart, \u201cModel Inversion Attacks that Exploit Confidence Information and Basic Countermeasures,\u201d CCS ’15: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security,<\/em> October 2015, pp. 1322\u20131333. [Online]. Available: https:\/\/www.cs.cmu.edu\/~mfredrik\/papers\/fjr2015ccs.pdf<\/a><\/p>\n

<\/a>
\n5. M. James, \u201cAdversarial attacks on voice input,\u201d I Programmer<\/em>, Jan. 31, 2018. [Online]. Available: https:\/\/www.i-programmer.info\/news\/105-artificial-intelligence\/11515-adversarial-attacks-on-voice-input.html<\/a><\/p>\n

<\/a>
\n6. G. Ateniese et al., \u201cHacking Smart Machines with Smarter Ones: How to Extract Meaningful Data from Machine Learning Classifiers,\u201d arXiv.org<\/em>, June 19, 2013. [Online]. Available: https:\/\/arxiv.org\/abs\/1306.4447<\/a><\/p>\n

<\/a>
\n7. A. Polyakov, \u201cHow to attack Machine Learning (Evasion, Poisoning, Inference, Trojans, Backdoors),\u201d towards data science, August 6, 2019. [Online]. Available: https:\/\/towardsdatascience.com\/how-to-attack-machine-learning-evasion-poisoning-inference-trojans-backdoors-a7cb5832595c<\/a><\/p>\n

<\/a>
\n8. K. Eykholt et al., \u201cRobust physical-world attacks on deep learning models,\u201d arXiv.org<\/em>, April 10, 2018. [Online]. Available: https:\/\/arxiv.org\/abs\/1707.08945<\/a><\/p>\n

<\/a>
\n9. M. James, \u201cAdversarial attacks on voice input,\u201d I Programmer<\/em>, Jan. 31, 2018. [Online]. Available: https:\/\/www.i-programmer.info\/news\/105-artificial-intelligence\/11515-adversarial-attacks-on-voice-input.html<\/a><\/p>\n

<\/a>
\n10. A. Dorschel, \u201cRethinking data privacy: The impact of machine learning,\u201d Medium<\/em>, April 24, 2019. [Online]. Available: https:\/\/medium.com\/luminovo\/data-privacy-in-machine-learning-a-technical-deep-dive-f7f0365b1d60<\/a><\/p>\n

<\/a>
\n11. M. Sharif, S. Bhagavatula, L. Bauer, and M. K. Reiter, \u201cA general framework for adversarial examples with objectives,\u201d arXiv.org<\/em>, April 4, 2019. [Online]. Available: https:\/\/arxiv.org\/abs\/1801.00349<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

Irrelevant Data, Irresponsible Outcomes<\/strong><\/p>\n

<\/a>
\n1. M. Simon, \u201cHP looking into claim webcams can\u2019t see black people,\u201d CNN.com<\/em>, Dec. 23, 2009. [Online]. Available: http:\/\/www.cnn.com\/2009\/TECH\/12\/22\/hp.webcams\/index.html<\/a><\/p>\n

<\/a>
\n2. B. Barrett, \u201cLawmakers can\u2019t ignore facial recognition\u2019s bias anymore,\u201d Wired<\/em>, July 26, 2018. [Online]. Available: https:\/\/www.wired.com\/story\/amazon-facial-recognition-congress-bias-law-enforcement\/<\/a><\/p>\n

<\/a>
\n3. P. Egan, \u201cData glitch was apparent factor in false fraud charges against jobless claimants,\u201d Detroit Free Press<\/em>, July 30, 2017. [Online]. Available: https:\/\/www.freep.com\/story\/news\/local\/michigan\/2017\/07\/30\/fraud-charges-unemployment-jobless-claimants\/516332001\/<\/a><\/p>\n

<\/a>
\n4. S. Mullainathan, \u201cBiased algorithms are easier to fix than biased people,\u201d New York Times<\/em>, Dec. 6, 2019. [Online]. Available: https:\/\/www.nytimes.com\/2019\/12\/06\/business\/algorithm-bias-fix.html?searchResultPosition=1<\/a><\/p>\n

<\/a>
\n5. Z. Obermeyer, B. Powers, C. Vogeli, and S. Mullainathan, \u201cDissecting racial bias in an algorithm used to manage the health of populations,\u201d Science,<\/em> vol. 366, no. 6464, pp. 447-453, Oct. 25, 2019. [Online]. Available: https:\/\/science.sciencemag.org\/content\/366\/6464\/447<\/a><\/p>\n

<\/a>
\n6. K. Hao, \u201cThis is how AI bias really happens\u2014and why it\u2019s so hard to fix,\u201d MIT Technology Review<\/em>, Feb. 4, 2020. [Online]. Available: https:\/\/www.technologyreview.com\/s\/612876\/this-is-how-ai-bias-really-happensand-why-its-so-hard-to-fix\/<\/a><\/p>\n

<\/a>
\n7. Data & Society, \u201cAlgorithmic accountability: A primer,\u201d Tech Algorithm Briefing: How Algorithms Perpetuate Racial Bias and Inequality, Prepared for the Congressional Progressive Caucus, April 18, 2018. [Online]. Available: https:\/\/datasociety.net\/wp-content\/uploads\/2018\/04\/Data_Society_Algorithmic_Accountability_Primer_FINAL-4.pdf<\/a><\/p>\n

<\/a>
\n8. N. Barrowman, \u201cWhy data is never raw,\u201d New Atlantis<\/em>, Summer\/Fall 2018. [Online]. Available: https:\/\/www.thenewatlantis.com\/publications\/why-data-is-never-raw<\/a><\/p>\n

<\/a>
\n9. K. Hao, \u201cThis is how AI bias really happens\u2014and why it\u2019s so hard to fix,\u201d MIT Technology Review<\/em>, Feb. 4, 2020. [Online]. Available: https:\/\/www.technologyreview.com\/s\/612876\/this-is-how-ai-bias-really-happensand-why-its-so-hard-to-fix\/<\/a><\/p>\n

<\/a>
\n10. K. Crawford and R. Calo, \u201cThere is a blind spot in AI research,\u201d Nature<\/em>, Oct. 13, 2016. [Online]. Available: https:\/\/www.nature.com\/articles\/538311a<\/a><\/p>\n

\u00a0<\/strong><\/p>\n

 <\/p>\n

You Told Me to Do This<\/strong><\/p>\n

<\/a>
\n1. N. V. Patel, \u201cWhy doctors aren\u2019t afraid of better, more efficient AI diagnosing cancer,\u201d Daily Beast<\/em>, Dec. 22, 2017. [Online]. Available: https:\/\/www.thedailybeast.com\/why-doctors-arent-afraid-of-better-more-efficient-ai-diagnosing-cancer<\/a><\/p>\n

<\/a>
\n2. T. Murphy VII, \u201cThe first level of Super Mario Bros. is easy with lexicographic orderings and time travel… after that it gets a little tricky,\u201d April 1, 2013. [Online]. Available: http:\/\/www.cs.cmu.edu\/~tom7\/mario\/mario.pdf<\/a><\/p>\n

<\/a>
\n3. J. Vincent, \u201cOpenAI has published the text-generating AI it said was too dangerous to share,\u201d The Verge,<\/em> November 7, 2019. [Online]. Available: https:\/\/www.theverge.com\/2019\/11\/7\/20953040\/openai-text-generation-ai-gpt-2-full-model-release-1-5b-parameters<\/a><\/p>\n

<\/a>
\n4. \u201cGPT-2: 1.5B Release,\u201d OpenAI<\/em>, November 5, 2019. [Online]. Available: https:\/\/openai.com\/blog\/gpt-2-1-5b-release\/<\/a><\/p>\n

<\/a>
\n5. Amodei, \u201cConcrete problems in AI safety,\u201d arXiv.org<\/em>, July 25, 2016. [Online]. Available: https:\/\/arxiv.org\/pdf\/1606.06565.pdf<\/a><\/p>\n

<\/a>
\n6. Data & Society, \u201cAlgorithmic accountability: A primer,\u201d Tech Algorithm Briefing: How Algorithms Perpetuate Racial Bias and Inequality, Prepared for the Congressional Progressive Caucus, April 18, 2018. [Online]. Available: https:\/\/datasociety.net\/wp-content\/uploads\/2018\/04\/Data_Society_Algorithmic_Accountability_Primer_FINAL-4.pdf<\/a><\/p>\n

<\/a>
\n7. Narayanan, \u201c21 fairness definitions and their politics,\u201d presented at Conference on Fairness, Accountability, and Transparency, Feb. 23, 2018. [Online]. Available: https:\/\/fairmlbook.org\/tutorial2.html<\/a><\/p>\n

<\/a>
\n8. \u201cCollege Board Announces Improved Admissions Resource,\u201d College Board, <\/em>August 27, 2019. [Online]. Available: https:\/\/www.collegeboard.org\/releases\/2019\/college-board-announces-improved-admissions-resource<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

Feeding the Feedback Loop<\/strong><\/p>\n

<\/a>
\n1. A. Jenkins, \u201cThis town is fining drivers to fight ‘horrific’ traffic from Google Maps and Waze,\u201d Travel + Leisure<\/em>, Dec. 26, 2017. [Online]. Available: https:\/\/www.travelandleisure.com\/travel-news\/leonia-waze-google-maps-fines<\/a><\/p>\n

<\/a>
\n2. A. Feng and S. Wu, \u201cThe myth of the impartial machine,\u201d Parametric Press<\/em>, no. 01 (Science + Society), May 1, 2019. [Online]. Available: https:\/\/parametric.press\/issue-01\/the-myth-of-the-impartial-machine\/<\/a><\/p>\n

<\/a>
\n3. E. Lacey, \u201cThe toxic potential of YouTube\u2019s feedback loop,\u201d Wired<\/em>, July 13, 2019. [Online]. Available: https:\/\/www.wired.com\/story\/the-toxic-potential-of-youtubes-feedback-loop\/<\/a><\/p>\n

<\/a>
\n4. \u201cAlgorithms and artificial intelligence: CNIL\u2019s report on the ethical issues,\u201d CNIL [Commission Nationale de l’Informatique et des Libert\u00e9s], May 25, 2018. [Online]. Available: https:\/\/www.cnil.fr\/en\/algorithms-and-artificial-intelligence-cnils-report-ethical-issues<\/a><\/p>\n

<\/a>
\n5. M. Heid, \u201cThe unsettling ways tech is changing your personal reality,\u201d Elemental<\/em>, Oct. 3, 2019. [Online]. Available: https:\/\/elemental.medium.com\/technology-is-fundamentally-changing-the-ways-you-think-and-feel-b4bbfdefc2ee<\/a><\/p>\n

<\/a>
\n6. M. Whittaker et al., AI Now Report 2018<\/em>. New York, NY, USA: AI Now Institute, 2018. [Online]. Available: https:\/\/ainowinstitute.org\/AI_Now_2018_Report.pdf<\/a><\/p>\n

<\/a>
\n7. W. Oremus, \u201cWho controls your Facebook feed,\u201d Slate<\/em>, Jan. 3, 2016. [Online]. Available: http:\/\/www.slate.com\/articles\/technology\/cover_story\/2016\/01\/how_facebook_s_news_feed_algorithm_works.html<\/a><\/p>\n

<\/a>
\n8. \u201cTech experts: What you post online could be directly impacting your insurance coverage,\u201d CBS New York<\/em>, March 21, 2019. [Online]. Available: https:\/\/newyork.cbslocal.com\/2019\/03\/21\/online-posting-dangerous-selfies-insurance-coverage\/<\/a><\/p>\n

<\/a>
\n9. R. Deller, \u201cBook review: Automating inequality: How high-tech tools profile, police and punish the poor by Virginia Eubanks,\u201d LSE Review of Books blog<\/em>, July 2, 2018. [Online]. Available: https:\/\/blogs.lse.ac.uk\/lsereviewofbooks\/2018\/07\/02\/book-review-automating-inequality-how-high-tech-tools-profile-police-and-punish-the-poor-by-virginia-eubanks\/<\/a><\/p>\n

<\/a>
\n10. \u201cAlgorithms and artificial intelligence: CNIL\u2019s report on the ethical issues,\u201d CNIL [Commission Nationale de l’Informatique et des Libert\u00e9s], May 25, 2018. [Online]. Available: https:\/\/www.cnil.fr\/en\/algorithms-and-artificial-intelligence-cnils-report-ethical-issues<\/a><\/p>\n

\u00a0<\/strong><\/p>\n

 <\/p>\n

A Special Case: AI Arms Race<\/strong><\/p>\n

<\/a>
\n1. \u201cHow artificial intelligence could increase the risk of nuclear war,\u201d The RAND blog<\/em>, April 23, 2018. [Online]. Available: https:\/\/www.rand.org\/blog\/articles\/2018\/04\/how-artificial-intelligence-could-increase-the-risk.html<\/a><\/p>\n

<\/a>
\n2. \u201cHow artificial intelligence could increase the risk of nuclear war,\u201d The RAND blog<\/em>, April 23, 2018. [Online]. Available: https:\/\/www.rand.org\/blog\/articles\/2018\/04\/how-artificial-intelligence-could-increase-the-risk.html<\/a><\/p>\n

<\/a>
\n3. P. Scharre, \u201cKiller apps: The real dangers of an AI arms race,\u201d Foreign Affairs<\/em>, March\/April 2019. [Online]. Available: https:\/\/www.foreignaffairs.com\/articles\/2019-04-16\/killer-apps<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

Testing in the Wild<\/strong><\/p>\n

<\/a>
\n1. A. MacGillis, \u201cThe case against Boeing,\u201d New Yorker<\/em>, Nov. 11, 2019. [Online]. Available: https:\/\/www.newyorker.com\/magazine\/2019\/11\/18\/the-case-against-boeing<\/a><\/p>\n

<\/a>
\n2. N. Sonnad, \u201cA flawed algorithm led the UK to deport thousands of students,\u201d Quartz<\/em>, May 3, 2018. [Online]. Available: https:\/\/qz.com\/1268231\/a-toeic-test-led-the-uk-to-deport-thousands-of-students\/<\/a><\/p>\n

<\/a>
\n3. \u201cAhsan v The Secretary of State for the Home Department (Rev 1) [2017] EWCA Civ 2009 (05 December 2017).\u201d British and Irish Legal Information Institute, December 5, 2017. [Online]. Available: http:\/\/www.bailii.org\/ew\/cases\/EWCA\/Civ\/2017\/2009.html<\/a><\/p>\n

<\/a>
\n4. P. Wu, \u201cTest your machine learning algorithm with metamorphic testing,\u201d Medium<\/em>, Nov. 13, 2017. [Online]. Available: https:\/\/medium.com\/trustableai\/testing-ai-with-metamorphic-testing-61d690001f5c<\/a><\/p>\n

<\/a>
\n5. I. Goodfellow and N. Papernot, \u201cThe challenge of verification and testing of machine learning,\u201d Cleverhans blog<\/em>, June 14, 2017. [Online]. Available: http:\/\/www.cleverhans.io\/security\/privacy\/ml\/2017\/06\/14\/verification.html<\/a><\/p>\n

<\/a>
\n6. Raphael, \u201cIntroducing tf-explain, interpretability for TensorFlow 2.0,\u201d Sicara blog<\/em>, July 30, 2019. [Online]. Available: https:\/\/blog.sicara.com\/tf-explain-interpretability-tensorflow-2-9438b5846e35<\/a><\/p>\n

<\/a>
\n7. \u201cFit interpretable machine learning models. Explain blackbox machine learning,\u201d GitHub. Accessed March 13, 2020. [Online]. Available: https:\/\/github.com\/Microsoft\/interpret<\/a><\/p>\n

<\/a>
\n8. Y. Sun et al., \u201d Structural test coverage criteria for deep neural networks,\u201d in 2019 IEEE\/ACM 41st International Conference on Software Engineering: Companion Proceedings<\/em>, 2019. [Online]. Available: https:\/\/www.kroening.com\/papers\/emsoft2019.pdf<\/a><\/p>\n

<\/a>
\n9. L. M. Strickhart and H.N.J. Lee, \u201cShow your work: Machine learning explainer tools and their use in artificial intelligence assurance,\u201d The MITRE Corporation, McLean, VA, June 2019, unpublished.<\/p>\n

<\/a>
\n10. D. Sculley et al., \u201cMachine learning: The high interest credit card of technical debt,\u201d in SE4ML: Software Engineering for Machine Learning (NIPS 2014 Workshop)<\/em>. Accessed March 16, 2020. [Online]. Available: https:\/\/ai.google\/research\/pubs\/pub43146<\/a><\/p>\n

<\/a>
\n11. A. Madan, \u201d3 practical ways to future-proof your IoT devices,\u201d IoT Times<\/em>, July 2, 2019. [Online]. Available:\u00a0 https:\/\/iot.eetimes.com\/3-practical-ways-to-future-proof-your-iot-devices\/<\/a><\/p>\n

<\/a>
\n12. A. Gonfalonieri, \u201cWhy machine learning models degrade in production,\u201d towards data science, July 25, 2019. [Online]. Available: https:\/\/towardsdatascience.com\/why-machine-learning-models-degrade-in-production-d0f2108e9214<\/a><\/p>\n

<\/a>
\n13. D. Sculley et al., \u201cMachine learning: The high interest credit card of technical debt,\u201d in SE4ML: Software Engineering for Machine Learning (NIPS 2014 Workshop)<\/em>. Accessed March 16, 2020. [Online]. Available: https:\/\/ai.google\/research\/pubs\/pub43146<\/a><\/p>\n

<\/a>
\n14. D. Sculley et al., \u201cMachine learning: The high interest credit card of technical debt,\u201d in SE4ML: Software Engineering for Machine Learning (NIPS 2014 Workshop)<\/em>. Accessed March 16, 2020. [Online]. Available: https:\/\/ai.google\/research\/pubs\/pub43146<\/a><\/p>\n

<\/a>
\n15. R. Potember, \u201cPerspectives on Research in Artificial Intelligence and Artificial General Intelligence Relevant to DoD,\u201d Defense Technical Information Center, Jan. 1, 2017. [Online]. Available: https:\/\/apps.dtic.mil\/docs\/citations\/AD1024432<\/a><\/p>\n

<\/a>
\n16. J. Zittrain, \u201cThe hidden costs of automated thinking,\u201d The New Yorker<\/em>, July 23, 2019. [Online]. Available:\u00a0 https:\/\/www.newyorker.com\/tech\/annals-of-technology\/the-hidden-costs-of-automated-thinking<\/a><\/p>\n

<\/a>
\n17. N. Carne, \u201cBlaming the driver in a \u2018driverless\u2019 car,\u201d Cosmos<\/em>, Oct. 29. 2019. [Online]. Available: https:\/\/cosmosmagazine.com\/technology\/blaming-the-driver-in-a-driverless-car<\/a><\/p>\n

<\/a>
\n18. S. Captain, \u201cHumans were to blame in Google self-driving car crash, police say,\u201d Fast Company, May 4, 2018. [Online]. Available: https:\/\/www.fastcompany.com\/40568609\/humans-were-to-blame-in-google-self-driving-car-crash-police-say<\/a><\/p>\n

<\/a>
\n19. J. Stewart, \u201cTesla’s autopilot was involved in another deadly car crash,\u201d Wired<\/em>, March 30, 2018. [Online]. Available: https:\/\/www.wired.com\/story\/tesla-autopilot-self-driving-crash-california\/<\/a><\/p>\n

<\/a>
\n20. J. Stewart, \u201cWhy Tesla\u2019s autopilot can\u2019t see a stopped firetruck,\u201d Wired<\/em>, Aug. 27, 2018. [Online]. Available: https:\/\/www.wired.com\/story\/tesla-autopilot-why-crash-radar\/<\/a><\/p>\n

<\/a>
\n21. M. McFarland, \u201cUber self-driving car kills pedestrian in first fatal autonomous crash,\u201d CNN Business<\/em>, March 19, 2018. [Online]. Available: https:\/\/money.cnn.com\/2018\/03\/19\/technology\/uber-autonomous-car-fatal-crash\/index.html<\/a><\/p>\n

<\/a>
\n22. A. MacGillis, \u201cThe case against Boeing,\u201d New Yorker<\/em>, Nov. 11, 2019. [Online]. Available: https:\/\/www.newyorker.com\/magazine\/2019\/11\/18\/the-case-against-boeing<\/a><\/p>\n

<\/a>
\n23. S. M. Casner and E. L. Hutchins, \u201cWhat do we tell the drivers? Toward minimum driver training standards for partially automated cars,\u201d Journal of Cognitive Engineering and Decision Making<\/em>, March 8, 2019. [Online]. Available: https:\/\/journals.sagepub.com\/doi\/full\/10.1177\/1555343419830901<\/a><\/p>\n

<\/a>
\n24. W. Langewiesche, \u201cThe human factor,\u201d Vanity Fair<\/em>, Oct. 2014. [Online]. Available: https:\/\/www.vanityfair.com\/news\/business\/2014\/10\/air-france-flight-447-crash<\/a><\/p>\n

<\/a>
\n25. \u201cA320, vicinity Tel Aviv Israel, 2012,\u201d SKYbrary<\/em>. Accessed on: March 11, 2020. [Online]. Available: https:\/\/www.skybrary.aero\/index.php\/A320,_vicinity_Tel_Aviv_Israel,_2012<\/a><\/p>\n

<\/a>
\n26. S. Gibbs, \u201cTesla Model S cleared by auto safety regulator after fatal Autopilot crash,\u201d Guardian<\/em>, Jan. 20, 2017. [Online]. Available: https:\/\/www.theguardian.com\/technology\/2017\/jan\/20\/tesla-model-s-cleared-auto-safety-regulator-after-fatal-autopilot-crash<\/a><\/p>\n

<\/a>
\n27. C. Ross and I. Swetlitz, \u201cIBM\u2019s Watson supercomputer recommended \u2018unsafe and incorrect\u2019 cancer treatments, internal documents show,\u201d STATnews, <\/em>July 25, 2018. [Online]. Available: https:\/\/www.statnews.com\/wp-content\/uploads\/2018\/09\/IBMs-Watson-recommended-unsafe-and-incorrect-cancer-treatments-STAT.pdf<\/a><\/p>\n

<\/a>
\n28. S. Fussell, \u201cPearson Embedded a ‘Social-Psychological’ Experiment in Students’ Educational Software [Updated],\u201d Gizmodo<\/em>, April 18, 2018. [Online]. Available: https:\/\/gizmodo.com\/pearson-embedded-a-social-psychological-experiment-in-s-1825367784<\/a><\/p>\n

<\/a>
\n29. M. Whittaker et al., AI Now Report 2018<\/em>. New York, NY, USA: AI Now Institute, 2018. [Online]. Available: https:\/\/ainowinstitute.org\/AI_Now_2018_Report.pdf<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

Government Dependence on Black Box Vendors <\/strong><\/p>\n

<\/a>
\n1. S. Corbett-Davies, E. Pierson, A. Feller, and S. Goel, \u201cA computer program used for bail and sentencing decisions was labeled biased against blacks. It’s actually not that clear,\u201d Washington Post<\/em>, Oct. 17, 2016. [Online]. Available: https:\/\/www.washingtonpost.com\/news\/monkey-cage\/wp\/2016\/10\/17\/can-an-algorithm-be-racist-our-analysis-is-more-cautious-than-propublicas\/?noredirect=on&utm_term=.a9cfb19a549d<\/a><\/p>\n

<\/a>
\n2. R. Wexler, \u201cWhen a computer program keeps you in jail,\u201d New York Times<\/em>, June 13, 2017. [Online]. Available: https:\/\/www.nytimes.com\/2017\/06\/13\/opinion\/how-computers-are-harming-criminal-justice.html<\/a><\/p>\n

<\/a>
\n3. C. Langford, \u201cHouston Schools Must Face Teacher Evaluation Lawsuit,\u201d Courthouse News Service, <\/em>May 8, 2017. [Online]. Available: https:\/\/www.courthousenews.com\/houston-schools-must-face-teacher-evaluation-lawsuit\/<\/a><\/p>\n

\u00a0<\/strong><\/p>\n

 <\/p>\n

Clear as Mud<\/strong>
\n<\/a>
\n1. A. Yoo, \u201cUPS: Driving performance by optimizing driver behavior,\u201d Harvard Business School Digital Initiative, April 5, 2017. [Online]. Available: https:\/\/digital.hbs.edu\/platform-digit\/submission\/ups-driving-performance-by-optimizing-driver-behavior\/<\/a><\/p>\n

<\/a>
\n2. K. Hill, \u201cFacebook recommended that this psychiatrist’s patients friend each other,\u201d Splinternews<\/em>, Aug. 29, 2016. [Online]. Available: https:\/\/splinternews.com\/facebook-recommended-that-this-psychiatrists-patients-f-1793861472<\/a><\/p>\n

<\/a>
\n3. B. Khaleghi, \u201cThe what of explainable AI,\u201d Element AI, Sept. 3, 2019. [Online]. Available: https:\/\/www.elementai.com\/news\/2019\/the-what-of-explainable-ai<\/a><\/p>\n

<\/a>
\n4. C. Rudin, \u201cStop explaining black box machine learning models for high stakes decisions and use interpretable models instead,\u201d arXiv.org<\/em>, Sep. 22, 2019. [Online]. Available: https:\/\/arxiv.org\/abs\/1811.10154<\/a><\/p>\n

<\/a>
\n5. P. L. McDermott, \u201cHuman-machine teaming systems engineering guide,\u201d The MITRE Corporation, Dec. 2018. [Online]. Available: https:\/\/www.mitre.org\/publications\/technical-papers\/human-machine-teaming-systems-engineering-guide<\/a><\/p>\n

<\/a>
\n6. D. Gunning, \u201cExplainable artificial intelligence (XAI),\u201d Defense Advanced Research Projects Agency, Nov. 2017. [Online]. Available: https:\/\/www.darpa.mil\/attachments\/XAIProgramUpdate.pdf?source=post_page—————————<\/a><\/p>\n

<\/a>
\n7. Z. C. Lipton, \u201cThe mythos of model interpretability,\u201d arXiv.org<\/em>, March 6, 2017. [Online]. Available: https:\/\/arxiv.org\/abs\/1606.03490<\/a><\/p>\n

<\/a>
\n8. C. Rudin, \u201cStop explaining black box machine learning models for high stakes decisions and use interpretable models instead,\u201d arXiv.org<\/em>, Sep. 22, 2019. [Online]. Available: https:\/\/arxiv.org\/abs\/1811.10154<\/a><\/p>\n

<\/a>
\n9. Z. C. Lipton, \u201cThe mythos of model interpretability,\u201d arXiv.org<\/em>, March 6, 2017. [Online]. Available: https:\/\/arxiv.org\/abs\/1606.03490<\/a><\/p>\n

<\/a>
\n10. C. Rudin, \u201cStop explaining black box machine learning models for high stakes decisions and use interpretable models instead,\u201d arXiv.org<\/em>, Sep. 22, 2019. [Online]. Available: https:\/\/arxiv.org\/abs\/1811.10154<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

In AI We Overtrust<\/strong><\/p>\n

<\/a>
\n1. P. Robinette, W. Li, R. Allen, A. M. Howard, and A. R. Wagner, \u201cOvertrust of robots in emergency evacuation scenarios,\u201d presented at 2016 11th ACM\/IEEE International Conference on Human-Robot Interaction (HRI), Christchurch, 2016, pp. 101-108. [Online]. Available: https:\/\/www.cc.gatech.edu\/~alanwags\/pubs\/Robinette-HRI-2016.pdf<\/a><\/p>\n

<\/a>
\n2. Georgia Tech, \u201cIn emergencies, should you trust a robot?\u201d YouTube. Accessed March 13, 2020. [Online]. Available: https:\/\/www.youtube.com\/watch?v=frr6cVBQPXQ<\/a><\/p>\n

<\/a>
\n3. M. Heid, \u201cThe unsettling ways tech is changing your personal reality,\u201d Elemental<\/em>, Oct. 3, 2019. [Online]. Available: https:\/\/elemental.medium.com\/technology-is-fundamentally-changing-the-ways-you-think-and-feel-b4bbfdefc2ee<\/a><\/p>\n

<\/a>
\n4. M. Vazquez, A. May, A. Steinfeld, and W.-H. Chen, \u201cA deceptive robot referee in a multiplayer gaming environment,\u201d Conference Paper, Proceedings of 2011 International Conference on Collaboration Technologies and Systems (CTS)<\/em>, pp. 204-211, May 2011. [Online]. Available: https:\/\/www.ri.cmu.edu\/publications\/a-deceptive-robot-referee-in-a-multiplayer-gaming-environment\/<\/a><\/p>\n

<\/a>
\n5. \u201cAlgorithms and artificial intelligence: CNIL\u2019s report on the ethical issues,\u201d CNIL [Commission Nationale de l’Informatique et des Libert\u00e9s], May 25, 2018. [Online]. Available: https:\/\/www.cnil.fr\/en\/algorithms-and-artificial-intelligence-cnils-report-ethical-issues<\/a><\/p>\n

<\/a>
\n6. \u201cAlgorithms and artificial intelligence: CNIL\u2019s report on the ethical issues,\u201d CNIL [Commission Nationale de l’Informatique et des Libert\u00e9s], May 25, 2018. [Online]. Available: https:\/\/www.cnil.fr\/en\/algorithms-and-artificial-intelligence-cnils-report-ethical-issues<\/a><\/p>\n

<\/a>
\n7. L. Hansen, \u201d8 drivers who blindly followed their GPS into disaster,\u201d The Week<\/em>, May 7, 2013. [Online]. Available: https:\/\/theweek.com\/articles\/464674\/8-drivers-who-blindly-followed-gps-into-disaster<\/a><\/p>\n

<\/a>
\n8. P. Madhavan and D. A. Wiegmann, \u201cSimilarities and differences between human-human and human-automation trust: An integrative review,\u201d Theoretical Issues in Ergonomics Science<\/em>, vol. 8, no. 4, pp. 277-301, 2007).<\/p>\n

<\/a>
\n9. \u201cAppeal to authority,\u201d Legally Fallacious. Accessed March 25, 2020. [Online]. Available:\u00a0 https:\/\/www.logicallyfallacious.com\/logicalfallacies\/Appeal-to-Authority<\/a><\/p>\n

<\/a>
\n10. M. Chalabi, \u201cWeapons of math destruction: Cathy O\u2019Neil adds up the damage of algorithms,\u201d Guardian<\/em>, Oct. 27, 2016. [Online]. Available: https:\/\/www.theguardian.com\/books\/2016\/oct\/27\/cathy-oneil-weapons-of-math-destruction-algorithms-big-data<\/a><\/p>\n

<\/a>
\n11. S. M. Casner and E. L. Hutchins, \u201cWhat do we tell the drivers? Toward minimum driver training standards for partially automated cars,\u201d Journal of Cognitive Engineering and Decision Making<\/em>, March 8, 2019. [Online]. Available: https:\/\/journals.sagepub.com\/doi\/full\/10.1177\/1555343419830901<\/a><\/p>\n

<\/a>
\n12. Data & Society, \u201cAlgorithmic accountability: A primer,\u201d Tech Algorithm Briefing: How Algorithms Perpetuate Racial Bias and Inequality, Prepared for the Congressional Progressive Caucus, April 18, 2018. [Online]. Available: https:\/\/datasociety.net\/wp-content\/uploads\/2018\/04\/Data_Society_Algorithmic_Accountability_Primer_FINAL-4.pdf<\/a><\/p>\n

<\/a>
\n13. \u201cAlgorithms and artificial intelligence: CNIL\u2019s report on the ethical issues,\u201d CNIL [Commission Nationale de l’Informatique et des Libert\u00e9s], May 25, 2018. [Online]. Available: https:\/\/www.cnil.fr\/en\/algorithms-and-artificial-intelligence-cnils-report-ethical-issues<\/a><\/p>\n

<\/a>
\n14. B. Aguera y Arcas, \u201cPhysiognomy\u2019s new clothes,\u201d Medium<\/em>, May 6, 2017. [Online]. Available: https:\/\/medium.com\/@blaisea\/physiognomys-new-clothes-f2d4b59fdd6a<\/a><\/p>\n

<\/a>
\n15. Synced, \u201c2018 in review: 10 AI failures,\u201d Medium<\/em>, Dec. 10, 2018. [Online]. Available: https:\/\/medium.com\/syncedreview\/2018-in-review-10-ai-failures-c18faadf5983<\/a><\/p>\n

<\/a>
\n16. M. Whittaker et al., AI Now Report 2018<\/em>. New York, NY, USA: AI Now Institute, 2018. [Online]. Available: https:\/\/ainowinstitute.org\/AI_Now_2018_Report.pdf<\/a><\/p>\n

<\/a>
\n17. S. Levin, \u201cNew AI can guess whether you’re gay or straight from a photograph,\u201d Guardian<\/em>, Sept. 7, 2017. [Online]. Available: https:\/\/www.theguardian.com\/technology\/2017\/sep\/07\/new-artificial-intelligence-can-tell-whether-youre-gay-or-straight-from-a-photograph<\/a><\/p>\n

<\/a>
\n18. Synced, \u201c2018 in review: 10 AI failures,\u201d Medium<\/em>, Dec. 10, 2018. [Online]. Available: https:\/\/medium.com\/syncedreview\/2018-in-review-10-ai-failures-c18faadf5983<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

Lost in Translation: Automation Surprise<\/strong><\/p>\n

<\/a>
\n1. S. M. Casner and E. L. Hutchins, \u201cWhat do we tell the drivers? Toward minimum driver training standards for partially automated cars,\u201d Journal of Cognitive Engineering and Decision Making<\/em>, March 8, 2019. [Online]. Available: https:\/\/journals.sagepub.com\/doi\/full\/10.1177\/1555343419830901<\/a><\/p>\n

<\/a>
\n2. \u201cA320, vicinity Tel Aviv Israel, 2012,\u201d SKYbrary<\/em>. Accessed on: March 11, 2020. [Online]. Available: https:\/\/www.skybrary.aero\/index.php\/A320,_vicinity_Tel_Aviv_Israel,_2012<\/a><\/p>\n

<\/a>
\n3. R. Nieva, \u201cFacebook put cork in chatbots that created a secret language,\u201d CNET<\/em>, July 31, 2017. [Online]. Available: https:\/\/www.cnet.com\/news\/what-happens-when-ai-bots-invent-their-own-language\/<\/a><\/p>\n

<\/a>
\n4. N. D. Sarter, D. D. Woods, and C. E. Billings, \u201cAutomation surprises,\u201d in G. Salvendy (Ed.), Handbook of Human Factors & Ergonomics<\/em> (2nd ed., pp. 1926-1943). New York, NY, USA: John Wiley, 1997.<\/p>\n

<\/a>
\n5. S. M. Casner and E. L. Hutchins, \u201cWhat do we tell the drivers? Toward minimum driver training standards for partially automated cars,\u201d Journal of Cognitive Engineering and Decision Making<\/em>, March 8, 2019. [Online]. Available: https:\/\/journals.sagepub.com\/doi\/full\/10.1177\/1555343419830901<\/a><\/p>\n

<\/a>
\n6. G. Klien et al., \u201cTen challenges for making automation a \u2018team player\u2019 in joint human-agent activity,\u201d IEEE: Intelligent Systems<\/em>, vol. 19, no. 6, pp. 91-95, Nov.\/Dec. 2004. [Online]. Available: http:\/\/jeffreymbradshaw.net\/publications\/17._Team_Players.pdf_1.pdf<\/a><\/p>\n

<\/a>
\n7. J. B. Lyons, \u201cBeing transparent about transparency: A model for human-robot interaction,\u201d in 2013 AAAI Spring Symposium Series,<\/em> 2013. [Online]. Available: https:\/\/www.semanticscholar.org\/paper\/Being-Transparent-about-Transparency%3A-A-Model-for-Lyons\/840080df8a02de6aab098e7eabef84831ac95428<\/a><\/p>\n

<\/a>
\n8. D. Woods, \u201cGeneric support requirements for cognitive work: laws that govern cognitive work in action,\u201d Proceedings of the Human Factors and Ergonomics Society Annual Meeting,<\/em> vol. 49, pp. 317-321, Sept. 1, 2005. [Online]. Available: https:\/\/journals.sagepub.com\/doi\/10.1177\/154193120504900322<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

The AI Resistance\u00a0 <\/strong><\/p>\n

<\/a>
\n1. \u201cLuddite,\u201d Merriam-Webster<\/em>. Accessed April 11, 2020. [Online]. Available: https:\/\/www.merriam-webster.com\/dictionary\/Luddite<\/a><\/p>\n

<\/a>
\n2. D. Wray, \u201cThe companies cleaning the deepest, darkest parts of social media,\u201d Vice<\/em>, June 26, 2018. [Online]. Available: https:\/\/www.vice.com\/en_us\/article\/ywe7gb\/the-companies-cleaning-the-deepest-darkest-parts-of-social-media<\/a><\/p>\n

<\/a>
\n3. \u201cWhy a #Google walkout organizer left Google,\u201d Medium<\/em>, June 7, 2019. [Online]. Available: https:\/\/medium.com\/@GoogleWalkout\/why-a-googlewalkout-organizer-left-google-26d1e3fbe317<\/a><\/p>\n

<\/a>
\n4. S. Romero, \u201cWielding rocks and knives, Arizonans attack self-driving cars,\u201d New York Times<\/em>, Dec. 31, 2018. [Online]. Available: https:\/\/www.nytimes.com\/2018\/12\/31\/us\/waymo-self-driving-cars-arizona-attacks.html<\/a><\/p>\n

<\/a>
\n5. D. Simberkoff, \u201cHow Facebook’s Cambridge Analytica scandal impacted the intersection of privacy and regulation,\u201d CMS Wire<\/em>, Aug. 30, 2018. [Online]. Available: https:\/\/www.cmswire.com\/information-management\/how-facebooks-cambridge-analytica-scandal-impacted-the-intersection-of-privacy-and-regulation\/<\/a><\/p>\n

<\/a>
\n6. \u201cTechnology adoption life cycle,\u201d Wikipedia. Accessed March 17, 2020. [Online]. Available: https:\/\/en.wikipedia.org\/wiki\/Technology_adoption_life_cycle<\/a><\/p>\n

<\/a>
\n7. M. Anderson, \u201cUseful or creepy? Machines suggest Gmail replies,\u201d AP News<\/em>, Aug. 30, 2018. [Online]. Available: https:\/\/apnews.com\/bcc384298fe944e89367e42e20d43f05<\/a><\/p>\n

<\/a>
\n8. \u201cHouse Intelligence Committee hearing on \u2018Deepfake\u2019 videos,\u201d C-SPAN<\/em>, June 13, 2019. [Online]. Available: https:\/\/www.c-span.org\/video\/?461679-1\/house-intelligence-committee-hearing-deepfake-videos<\/a><\/p>\n

<\/a>
\n9. C. F. Kerry, \u201cProtecting privacy in an AI-driven world,\u201d Brookings, Feb. 10, 2020. [Online]. Available:\u00a0 https:\/\/www.brookings.edu\/research\/protecting-privacy-in-an-ai-driven-world\/<\/a> \u00a0<\/u><\/p>\n

<\/a>
\n10. C. Forrest, \u201cFear of losing job to AI is the no. 1 cause of stress at work,\u201d TechRepublic<\/em>, June 6, 2017.\u00a0 [Online]. Available: https:\/\/www.techrepublic.com\/article\/report-fear-of-losing-job-to-ai-is-the-no-1-cause-of-stress-at-work\/<\/a><\/p>\n

<\/a>
\n11. S. Browne, Dark Matters: On the Surveillance of Blackness<\/em>, Durham, NC, USA: Duke University Press Books, 2015. [Online]. Available: https:\/\/www.dukeupress.edu\/dark-matters<\/a><\/p>\n

<\/a>
\n12. A. M. Bedoya, \u201cThe color of surveillance: What an infamous abuse of power teaches us about the modern spy era,\u201d Slate<\/em>, Jan. 18, 2016. [Online]. Available: https:\/\/slate.com\/technology\/2016\/01\/what-the-fbis-surveillance-of-martin-luther-king-says-about-modern-spying.html<\/a><\/p>\n

<\/a>
\n13. M. Cyril, \u201cWatching the Black body,\u201d Electronic Frontier Foundation, Feb. 28, 2019. [Online]. Available: https:\/\/www.eff.org\/deeplinks\/2019\/02\/watching-black-body<\/a><\/p>\n

<\/a>
\n14. P. McCausland, \u201cSelf-driving Uber car that hit and killed woman did not recognize that pedestrians jaywalk,\u201d NBC News<\/em>, Nov. 9, 2019. [Online]. Available: https:\/\/www.nbcnews.com\/tech\/tech-news\/self-driving-uber-car-hit-killed-woman-did-not-recognize-n1079281<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

Good (Grief!) Governance<\/strong><\/p>\n

<\/a>
\n1. A. M. Barry-Jester, B. Casselman, and D. Goldstein, \u201cShould prison sentences be based on crimes that haven\u2019t been committed yet?\u2019 FiveThirtyEight<\/em>, Aug. 4, 2015. [Online]. Available: https:\/\/fivethirtyeight.com\/features\/prison-reform-risk-assessment\/<\/a><\/p>\n

<\/a>
\n2. E. Ongweso, \u201cGoogle is investigating why it trained facial recognition on \u2018dark skinned\u2019 homeless people,\u201d Vice<\/em>, Oct. 4, 2019. [Online]. Available: https:\/\/www.vice.com\/en_us\/article\/43k7yd\/google-is-investigating-why-it-trained-facial-recognition-on-dark-skinned-homeless-people<\/a><\/p>\n

<\/a>
\n3. J. Stanley, \u201cSecret Service announces test of face recognition system around White House,\u201d ACLU bog<\/em>, Dec. 4, 2018. [Online]. Available: https:\/\/www.aclu.org\/blog\/privacy-technology\/surveillance-technologies\/secret-service-announces-test-face-recognition<\/a><\/p>\n

<\/a>
\n4. R. Courtland, \u201cBias detectives: The researchers striving to make algorithms fair,\u201d Nature<\/em>, June 20, 2018. [Online]. Available: https:\/\/www.nature.com\/articles\/d41586-018-05469-3<\/a><\/p>\n

<\/a>
\n5. D. Robinson and L. Koepke, \u201cStuck in a pattern: Early evidence on \u2018predictive policing\u2019 and civil rights,\u201d Upturn, Aug. 2016. [Online]. Available: https:\/\/www.upturn.org\/reports\/2016\/stuck-in-a-pattern\/<\/a><\/p>\n

<\/a>
\n6. \u201cAn ethics guidelines global inventory,\u201d Algorithm Watch. Accessed on: Jan. 17, 2020. [Online]. Available: https:\/\/algorithmwatch.org\/en\/project\/ai-ethics-guidelines-global-inventory\/<\/a><\/p>\n

<\/a>
\n7. \u201cAn ethics guidelines global inventory,\u201d Algorithm Watch. Accessed on: Jan. 17, 2020. [Online]. Available: https:\/\/algorithmwatch.org\/en\/project\/ai-ethics-guidelines-global-inventory\/<\/a><\/p>\n

<\/a>
\n8. T. Hagendorff, \u201cThe ethics of AI ethics: An evaluation of guidelines,\u201d arXiv.org<\/em>, Oct. 11, 2019. [Online]. Available: https:\/\/arxiv.org\/abs\/1903.03425<\/a><\/p>\n

<\/a>
\n9. R. Vought, \u201cGuidance for regulation of artificial intelligence applications,\u201d Draft memorandum, WhiteHouse.gov<\/em>. Accessed on: Jan. 21, 2020. [Online]. Available: https:\/\/www.whitehouse.gov\/wp-content\/uploads\/2020\/01\/Draft-OMB-Memo-on-Regulation-of-AI-1-7-19.pdf<\/a><\/p>\n

<\/a>
\n10. \u201cWrestling with AI governance around the world,\u201d Forbes<\/em>, March 27, 2019. [Online]. Available: https:\/\/www.forbes.com\/sites\/insights-intelai\/2019\/03\/27\/wrestling-with-ai-governance-around-the-world\/#7d3f84ed1766<\/a><\/p>\n

<\/a>
\n11. G. Vyse, \u201cThree American cities have now banned the use of facial recognition technology in local government amid concerns it’s inaccurate and biased,\u201d Governing<\/em>, July 24, 2019. [Online]. Available: https:\/\/www.governing.com\/topics\/public-justice-safety\/gov-cities-ban-government-use-facial-recognition.html<\/a><\/p>\n

<\/a>
\n12. P. Martineau, \u201cCities examine proper\u2014and improper\u2014uses of facial recognition,\u201d Wired<\/em>, Nov. 10, 2019. [Online]. Available: https:\/\/www.wired.com\/story\/cities-examine-proper-improper-facial-recognition\/<\/a><\/p>\n

<\/a>
\n13. \u201cBan facial recognition.\u201d Accessed March 17, 2020. [Online]. Available: https:\/\/www.banfacialrecognition.com\/map\/<\/a><\/p>\n

<\/a>
\n14. \u201cAlgorithms and artificial intelligence: CNIL\u2019s report on the ethical issues,\u201d CNIL [Commission Nationale de l’Informatique et des Libert\u00e9s], May 25, 2018. [Online]. Available: https:\/\/www.cnil.fr\/en\/algorithms-and-artificial-intelligence-cnils-report-ethical-issues<\/a><\/p>\n

<\/a>
\n15. R. Courtland, \u201cBias detectives: The researchers striving to make algorithms fair,\u201d Nature<\/em>, June 20, 2018. [Online]. Available: https:\/\/www.nature.com\/articles\/d41586-018-05469-3<\/a><\/p>\n

<\/a>
\n16. D. Robinson and L. Koepke, \u201cStuck in a pattern: Early evidence on \u2018predictive policing\u2019 and civil rights,\u201d Upturn, Aug. 2016. [Online]. Available: https:\/\/www.upturn.org\/reports\/2016\/stuck-in-a-pattern\/<\/a><\/p>\n

<\/a>
\n17. D. Robinson and L. Koepke, \u201cStuck in a pattern: Early evidence on \u2018predictive policing\u2019 and civil rights,\u201d Upturn, Aug. 2016. [Online]. Available: https:\/\/www.upturn.org\/reports\/2016\/stuck-in-a-pattern\/<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

Just Add (Technical) People<\/strong><\/p>\n

<\/a>
\n1. J. Spitzer, \u201cIBM’s Watson recommended ‘unsafe and incorrect’ cancer treatments, STAT report finds,\u201d Becker\u2019s Health IT<\/em>, July 25, 2018. [Online]. Available: https:\/\/www.beckershospitalreview.com\/artificial-intelligence\/ibm-s-watson-recommended-unsafe-and-incorrect-cancer-treatments-stat-report-finds.html<\/a><\/p>\n

<\/a>
\n2. A. Liptak, \u201cThe US Navy will replace its touchscreen controls with mechanical ones on its destroyers,\u201d The Verge<\/em>, Aug. 11, 2019. [Online]. Available: https:\/\/www.theverge.com\/2019\/8\/11\/20800111\/us-navy-uss-john-s-mccain-crash-ntsb-report-touchscreen-mechanical-controls<\/a><\/p>\n

<\/a>
\n3. T. Simonite, \u201cWhen It Comes to Gorillas, Google Photos Remains Blind,\u201d Wired<\/em>, January 11, 2018. [Online]. Available: https:\/\/www.wired.com\/story\/when-it-comes-to-gorillas-google-photos-remains-blind\/<\/a><\/p>\n

<\/a>
\n4. B. Marr, \u201cThe AI skills crisis and how to close the gap,\u201d Forbes<\/em>, June 25, 2018. [Online]. Available: https:\/\/www.forbes.com\/sites\/bernardmarr\/2018\/06\/25\/the-ai-skills-crisis-and-how-to-close-the-gap\/#6525b57b31f3<\/a><\/p>\n

<\/a>
\n5. NICE cybersecurity workforce framework resource center,\u201d National Institute of Standards and Technology. Accessed March 17, 2020. [Online]. Available: https:\/\/www.nist.gov\/itl\/applied-cybersecurity\/nice\/nice-cybersecurity-workforce-framework-resource-center<\/a><\/p>\n

<\/a>
\n6. S. Anand and T. B\u00e4rnighausen, \u201cHealth workers at the core of the health system: Framework and research issues,\u201d Global Health Workforce Alliance, 2011. [Online]. Available: https:\/\/www.who.int\/workforcealliance\/knowledge\/resources\/frameworkandresearch_dec2011\/en\/<\/a><\/p>\n

<\/a>
\n7. Lippincott Solutions, \u201cInterdisciplinary care plans: Teamwork makes the dream work,\u201d Calling the Shots blog<\/em>, Sept. 6, 2018. [Online]. Available:\u00a0 http:\/\/lippincottsolutions.lww.com\/blog.entry.html\/2018\/09\/06\/interdisciplinaryca-z601.html<\/a><\/p>\n

<\/a>
\n8. M. Mahdizadeh, A. Heydari, and H. K. Moonaghi, \u201cClinical interdisciplinary collaboration models and frameworks from similarities to differences: A systematic review,\u201d Global Journal of Health Science<\/em>, vol. 7, no. 6, pp. 170-180, Nov. 2015. [Online]. Available: https:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC4803863\/<\/a><\/p>\n

<\/a>
\n9. C. Hagel, \u201cReagan national defense forum keynote,\u201d Secretary of Defense Speech, Ronald Reagan Presidential Library, Simi Valley, CA, Nov. 15, 2014. [Online]. Available: https:\/\/www.defense.gov\/Newsroom\/Speeches\/Speech\/Article\/606635\/<\/a><\/p>\n

<\/a>
\n10. \u201cReports,\u201d National Security Commission on Artificial Intelligence. Accessed March 18, 2020. [Online]. Available: https:\/\/www.nscai.gov\/reports<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

Square Data, Round Problem<\/strong><\/p>\n

<\/a>
\n1. \u201cBad data costs United Airlines $1B annually,\u201d Travel Data Daily. Accessed March 16, 2020. [Online]. Available: https:\/\/www.traveldatadaily.com\/bad-data-costs-united-airlines-1b-annually\/<\/a><\/p>\n

<\/a>
\n2. B. Vergakis, \u201cThe Navy, Air Force and Army collect different data on aircraft crashes. That’s a big problem,\u201d Task & Purpose<\/em>, Aug. 16, 2018. [Online]. Available: https:\/\/taskandpurpose.com\/aviation-mishaps-data-collection<\/a><\/p>\n

<\/a>
\n3. B. Marr, \u201cHow much data do we create every day? The mind-blowing stats everyone should read,\u201d Forbes<\/em>, May 21, 2018. [Online]. Available: https:\/\/www.forbes.com\/sites\/bernardmarr\/2018\/05\/21\/how-much-data-do-we-create-every-day-the-mind-blowing-stats-everyone-should-read\/<\/a><\/p>\n

<\/a>
\n4. \u201cNo AI until the data is fixed,\u201d Wired<\/em>, Feb. 22, 2019. [Online]. Available: https:\/\/www.wired.co.uk\/article\/no-ai-until-the-data-is-fixed<\/a><\/p>\n

<\/a>
\n5. D. Robinson and L. Koepke, \u201cStuck in a pattern: Early evidence on \u2018predictive policing\u2019 and civil rights,\u201d Upturn, Aug. 2016. [Online]. Available: https:\/\/www.upturn.org\/reports\/2016\/stuck-in-a-pattern\/<\/a><\/p>\n

<\/a>
\n6. \u201cAlgorithms and artificial intelligence: CNIL\u2019s report on the ethical issues,\u201d CNIL [Commission Nationale de l’Informatique et des Libert\u00e9s], May 25, 2018. [Online]. Available: https:\/\/www.cnil.fr\/en\/algorithms-and-artificial-intelligence-cnils-report-ethical-issues<\/a><\/p>\n

 <\/p>\n

\u00a0<\/strong><\/p>\n

My 8-Track Still Works So What\u2019s the Issue? <\/strong><\/p>\n

<\/a>
\n1. U.S. Government Accountability Office, \u201cInformation technology: Federal agencies need to address aging legacy systems,\u201d GAO-16-696T, May 25, 2016. [Online]. Available: https:\/\/www.gao.gov\/products\/GAO-16-696T<\/a><\/p>\n

<\/a>
\n2. D. Cassel, \u201cCOBOL is everywhere. Who will maintain it?\u201d The New Stack<\/em>, May 6, 2017. [Online]. Available: https:\/\/thenewstack.io\/cobol-everywhere-will-maintain\/<\/a><\/p>\n

<\/a>
\n3. J. Uchill, \u201cHow did the government\u2019s technology get so bad?\u201d The Hill<\/em>, Dec. 13, 2016. [Online]. Available: https:\/\/thehill.com\/policy\/technology\/310271-how-did-the-governments-technology-get-so-bad<\/a><\/p>\n

<\/a>
\n4. B. Balter, \u201c19 reasons why technologists don\u2019t want to work at your government agency,\u201d April 21, 2015. [Online]. Available: https:\/\/ben.balter.com\/2015\/04\/21\/why-technologists-dont-want-to-work-at-your-agency\/<\/a><\/p>\n

<\/a>
\n5. U.S. Government Accountability Office, \u201cInformation technology: Federal agencies need to address aging legacy systems,\u201d GAO-16-696T, May 25, 2016. [Online]. Available: https:\/\/www.gao.gov\/products\/GAO-16-696T<\/a><\/p>\n

<\/a>
\n6. D. Cassel, \u201cCOBOL is everywhere. Who will maintain it?\u201d The New Stack<\/em>, May 6, 2017. [Online]. Available: https:\/\/thenewstack.io\/cobol-everywhere-will-maintain\/<\/a><\/p>\n

 <\/p>\n

\u00a0<\/strong><\/p>\n

Lessons Learned<\/em><\/strong><\/h3>\n

Hold AI to a Higher Standard<\/strong>
\n<\/a>
\n1. \u201cAlgorithms and artificial intelligence: CNIL\u2019s report on the ethical issues,\u201d CNIL [Commission Nationale de l’Informatique et des Libert\u00e9s], May 25, 2018. [Online]. Available: https:\/\/www.cnil.fr\/en\/algorithms-and-artificial-intelligence-cnils-report-ethical-issues<\/a><\/p>\n

<\/a>
\n2. M. Whittaker et al., AI Now Report 2018<\/em>. New York, NY, USA: AI Now Institute, 2018. [Online]. Available: https:\/\/ainowinstitute.org\/AI_Now_2018_Report.pdf<\/a><\/p>\n

<\/a>
\n3. S. Gibbs, \u201cTesla Model S cleared by auto safety regulator after fatal Autopilot crash,\u201d Guardian<\/em>, Jan. 20, 2017. [Online]. Available: https:\/\/www.theguardian.com\/technology\/2017\/jan\/20\/tesla-model-s-cleared-auto-safety-regulator-after-fatal-autopilot-crash<\/a><\/p>\n

<\/a>
\n4. D. Tomchek and S. Krawlzik, \u201cLooking beyond the technical to fill America’s cyber workforce gap,\u201d Nextgov<\/em>, Sept. 27, 2019. [Online]. Available: https:\/\/www.nextgov.com\/ideas\/2019\/09\/looking-beyond-technical-fill-americas-cyber-workforce-gap\/160222\/<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

It\u2019s OK to Say No to Automation<\/strong>
\n<\/a>
\n1. M. Johnson, J. M. Bradshaw, R. R. Hoffman, P. J. Feltovich, and D. D. Woods, \u201cSeven cardinal virtues of human-machine teamwork: Examples from the DARPA robotic challenge,\u201d IEEE Intelligent Systems<\/em>, Nov.\/Dec. 2014. [Online]. Available: http:\/\/www.jeffreymbradshaw.net\/publications\/56.%20Human-Robot%20Teamwork_IEEE%20IS-2014.pdf<\/a><\/p>\n

<\/a>
\n2. \u201cEthics & algorithms toolkit.\u201d Accessed March 13, 2020. [Online]. Available: http:\/\/ethicstoolkit.ai\/<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

AI Challenges Are Multidisciplinary, so They Require a Multidisciplinary Team<\/strong><\/p>\n

<\/a>
\n1. S. Ferro, \u201cHere\u2019s why facial recognition tech can\u2019t figure out black people,\u201d HuffPost<\/em>, March 2, 2016. [Online]. Available: https:\/\/www.huffpost.com\/entry\/heres-why-facial-recognition-tech-cant-figure-out-black-people_n_56d5c2b1e4b0bf0dab3371eb<\/a><\/p>\n

<\/a>
\n2. S. J. Freedberg, \u201c\u2019Guess what, there\u2019s a cost for that\u2019: Getting cloud & AI right,\u201d Breaking Defense, Nov. 26, 2019. [Online]. Available: https:\/\/breakingdefense.com\/2019\/11\/guess-what-theres-a-cost-for-that-getting-cloud-ai-right\/<\/a><\/p>\n

<\/a>
\n3. A. Campolo et al., AI Now Report 2017<\/em>. New York, NY, USA: AI Now Institute, 2017. [Online]. Available:\u00a0 https:\/\/ainowinstitute.org\/AI_Now_2017_Report.pdf<\/a><\/p>\n

\u00a0<\/strong><\/p>\n

 <\/p>\n

Incorporate Privacy, Civil Liberties, and Security from the Beginning <\/strong><\/p>\n

<\/a>
\n1. R. V. Yampolskiy and M. S. Spellchecker, \u201cArtificial intelligence safety and cybersecurity: A timeline of AI failures,\u201d arXiv.org<\/em>. Accessed March 25, 2020. [Online]. Available: https:\/\/arxiv.org\/ftp\/arxiv\/papers\/1610\/1610.07997.pdf<\/a><\/p>\n

<\/a>
\n2. J. Rotner, \u201cThe person at the other end of the data,\u201d Knowledge-Driven Enterprise blog<\/em>, The MITRE Corporation, Oct. 1, 2019. [Online]. Available: https:\/\/kde.mitre.org\/blog\/2019\/10\/01\/the-person-at-the-other-end-of-the-data\/<\/a><\/p>\n

<\/a>
\n3. J. Whittlestone, A. Alexandrova, R. Nyrup, and S. Cave, \u201cThe role and limits of principles in AI ethics: Towards a focus on tensions,\u201d presented at AIES \u201919, Jan. 27\u201328, 2019, Honolulu, HI, USA. [Online]. Available: https:\/\/www.researchgate.net\/publication\/334378492_The_Role_and_Limits_of_Principles_in_AI_Ethics_Towards_a_Focus_on_Tensions\/link\/5d269de0a6fdcc2462d41592\/download<\/a><\/p>\n

<\/a>
\n4. I. Goodfellow, P. McDaniel, and N. Papernot, \u201cMaking machine learning robust against adversarial inputs,\u201d Communications of the ACM<\/em>, vol. 61, no. 7, pp. 56-66, July 2018. [Online]. Available: https:\/\/cacm.acm.org\/magazines\/2018\/7\/229030-making-machine-learning-robust-against-adversarial-inputs\/fulltext<\/a><\/p>\n

<\/a>
\n5. R. V. Yampolskiy and M. S. Spellchecker, \u201cArtificial intelligence safety and cybersecurity: A timeline of AI failures,\u201d arXiv.org<\/em>. Accessed March 25, 2020. [Online]. Available: https:\/\/arxiv.org\/ftp\/arxiv\/papers\/1610\/1610.07997.pdf<\/a><\/p>\n

<\/a>
\n6. \u201cGeneral data protection regulation,\u201d European Union. Accessed March 25, 2020. [Online]. Available: https:\/\/eugdpr.com\/<\/a><\/p>\n

<\/a>
\n7. D. Miralis and P. Gibson, \u201cAustralia: Data protection 2019,\u201d ICLG.com<\/em>, March 7, 2019. [Online]. Available: https:\/\/iclg.com\/practice-areas\/data-protection-laws-and-regulations\/australia<\/a><\/p>\n

<\/a>
\n8. \u201cData protection laws of the world: New Zealand,\u201d DLA Piper. Accessed March 16, 2020. [Online]. Available: https:\/\/www.dlapiperdataprotection.com\/index.html?t=law&c=NZ<\/a><\/p>\n

<\/a>
\n9. G. Vyse, \u201cThree American cities have now banned the use of facial recognition technology in local government amid concerns it’s inaccurate and biased,\u201d Governing.com<\/em>, July 24, 2019. [Online]. Available: https:\/\/www.governing.com\/topics\/public-justice-safety\/gov-cities-ban-government-use-facial-recognition.html<\/a><\/p>\n

<\/a>
\n10. L. Hautala, \u201cCalifornia\u2019s new data privacy law the toughest in the US,\u201d CNET.com<\/em>, June 29, 2018. [Online]. Available: https:\/\/www.cnet.com\/news\/californias-new-data-privacy-law-the-toughest-in-the-us\/<\/a><\/p>\n

 <\/p>\n

\u00a0<\/strong><\/p>\n

Involve the Communities Affected by the AI<\/strong><\/p>\n

<\/a>
\n1. M. Whittaker et al., AI Now Report 2018<\/em>. New York, NY, USA: AI Now Institute, 2018. [Online]. Available: https:\/\/ainowinstitute.org\/AI_Now_2018_Report.pdf<\/a><\/p>\n

<\/a>
\n2. \u201cDiverse Voices: A How-To Guide for Facilitating Inclusiveness in Tech Policy.\u201d Accessed April 8, 2020. [Online]. Available: https:\/\/techpolicylab.uw.edu\/project\/diverse-voices\/<\/a><\/p>\n

<\/a>
\n3. A. Campolo et al., AI Now Report 2017<\/em>. New York, NY, USA: AI Now Institute, 2017. [Online]. Available:\u00a0 https:\/\/ainowinstitute.org\/AI_Now_2017_Report.pdf<\/a><\/p>\n

<\/a>
\n4. M. Whittaker et al., AI Now Report 2018<\/em>. New York, NY, USA: AI Now Institute, 2018. [Online]. Available: https:\/\/ainowinstitute.org\/AI_Now_2018_Report.pdf<\/a><\/p>\n

<\/a>
\n5. A. Campolo et al., AI Now Report 2017<\/em>. New York, NY, USA: AI Now Institute, 2017. [Online]. Available:\u00a0 https:\/\/ainowinstitute.org\/AI_Now_2017_Report.pdf<\/a><\/p>\n

<\/a>
\n6. M. Whittaker et al., AI Now Report 2018<\/em>. New York, NY, USA: AI Now Institute, 2018. [Online]. Available: https:\/\/ainowinstitute.org\/AI_Now_2018_Report.pdf<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

Plan to Fail<\/strong><\/p>\n

<\/a>
\n1. \u201cBenjamin Franklin quotable quote,\u201d Goodreads. Accessed March 16, 2020. [Online]. Available: https:\/\/www.goodreads.com\/quotes\/460142-if-you-fail-to-plan-you-are-planning-to-fail<\/a><\/p>\n

<\/a>
\n2. M. Johnson, J. M. Bradshaw, R. R. Hoffman, P. J. Feltovich, and D. D. Woods, \u201cSeven cardinal virtues of human-machine teamwork: Examples from the DARPA robotic challenge,\u201d IEEE Intelligent Systems<\/em>, Nov.\/Dec. 2014. [Online]. Available: http:\/\/www.jeffreymbradshaw.net\/publications\/56.%20Human-Robot%20Teamwork_IEEE%20IS-2014.pdf<\/a><\/p>\n

<\/a>
\n3. M. Baker and D. Gates, \u201cLack of redundancies on Boeing 737 MAX system baffles some involved in developing the jet,\u201d Seattle Times<\/em>, March 27, 2019. [Online]. Available: https:\/\/www.seattletimes.com\/business\/boeing-aerospace\/a-lack-of-redundancies-on-737-max-system-has-baffled-even-those-who-worked-on-the-jet\/<\/a><\/p>\n

<\/a>
\n4. E. Lacey, \u201cThe toxic potential of YouTube\u2019s feedback loop,\u201d Wired<\/em>, July 13, 2019. [Online]. Available: https:\/\/www.wired.com\/story\/the-toxic-potential-of-youtubes-feedback-loop\/<\/a><\/p>\n

<\/a>
\n5. D. Amodei, \u201cConcrete problems in AI safety,\u201d arXiv.org<\/em>, July 25, 2016. [Online]. Available: https:\/\/arxiv.org\/pdf\/1606.06565.pdf<\/a><\/p>\n

\u00a0<\/strong><\/p>\n

 <\/p>\n

Ask for Help: Hire a Villain<\/strong><\/p>\n

<\/a>
\n1. \u201cThe Netflix Simian Army,\u201d The Netflix Tech Blog,<\/em> July 19, 2011. [Online]. Available: https:\/\/netflixtechblog.com\/the-netflix-simian-army-16e57fbab116<\/a><\/p>\n

<\/a>
\n2. C. A. Cois, \u201cDevOps case study: Netflix and the chaos monkey,\u201d DevOps blog<\/em>, Software Engineering Institute, April 30, 2015. [Online]. Available: https:\/\/insights.sei.cmu.edu\/devops\/2015\/04\/devops-case-study-netflix-and-the-chaos-monkey.html<\/a><\/p>\n

<\/a>
\n3. \u201cWhite-hat,\u201d Your Dictionary<\/em>. Accessed March 13, 2020. [Online]. Available: https:\/\/www.yourdictionary.com\/white-hat<\/a><\/p>\n

<\/a>
\n4. E. Tittel and E. Follis, \u201cHow to become a white hat hacker,\u201d Business News Daily<\/em>, June 17, 2019. [Online]. Available: https:\/\/www.businessnewsdaily.com\/10713-white-hat-hacker-career.html<\/a><\/p>\n

<\/a>
\n5. K. Lerwing, \u201cApple hired the hackers who created the first Mac firmware virus,\u201d Business Insider<\/em>, Feb. 3, 2016. [Online]. Available: https:\/\/www.businessinsider.com\/apple-hired-the-hackers-who-created-the-first-mac-firmware-virus-2016-2<\/a><\/p>\n

<\/a>
\n6. HackerOne, \u201cWhat was it like to hack the Pentagon?\u201d h1 blog<\/em>, June 17, 2016. [Online]. Available: https:\/\/www.hackerone.com\/blog\/hack-the-pentagon-results<\/a><\/p>\n

<\/a>
\n7. J. Talamantes, “What is red teaming and why do I need it?\u201d RedTeam blog<\/em>. Accessed March 16, 2020. [Online]. Available: https:\/\/www.redteamsecure.com\/what-is-red-teaming-and-why-do-i-need-it-2\/<\/a><\/p>\n

\u00a0<\/strong><\/p>\n

 <\/p>\n

Use Math to Reduce Bad Outcomes Caused by Math<\/strong><\/p>\n

<\/a>
\n1. K. Hao, \u201cThis is how AI bias really happens\u2014and why it\u2019s so hard to fix,\u201d MIT Technology Review<\/em>, Feb. 4, 2020. [Online]. Available: https:\/\/www.technologyreview.com\/s\/612876\/this-is-how-ai-bias-really-happensand-why-its-so-hard-to-fix\/<\/a><\/p>\n

<\/a>
\n2. A. Feng and S. Wu, \u201cThe myth of the impartial machine,\u201d Parametric Press<\/em>, no. 01 (Science + Society), May 1, 2019. [Online]. Available: https:\/\/parametric.press\/issue-01\/the-myth-of-the-impartial-machine\/<\/a><\/p>\n

<\/a>
\n3. \u201cAI fairness 360 open source toolkit,\u201d IBM Research Trusted AI. Accessed March 13, 2020. [Online]. Available: http:\/\/aif360.mybluemix.net\/<\/a><\/p>\n

<\/a>
\n4. “Bias and fairness audit toolkit,\u201d GitHub. Accessed March 13, 2020. [Online]. Available: https:\/\/github.com\/dssg\/aequitas<\/a><\/p>\n

<\/a>
\n5. \u201cA Python package that implements a variety of algorithms that mitigate unfairness in supervised machine learning,\u201d GitHub. Accessed March 13, 2020. [Online]. Available: https:\/\/github.com\/Microsoft\/fairlearn<\/a><\/p>\n

<\/a>
\n6. \u201cWhat-if tool,\u201d GitHub. Accessed March 13, 2020. [Online]. Available: https:\/\/pair-code.github.io\/what-if-tool\/<\/a><\/p>\n

<\/a>
\n7. \u201cFacets,\u201d GitHub. Accessed March 13, 2020. [Online]. Available: https:\/\/pair-code.github.io\/facets\/<\/a><\/p>\n

<\/a>
\n8. T. Bolukbasi, K. Chang, J. Zou, V. Saligrama, and A. Kalai, \u201cMan is to computer programmer as woman is to homemaker? Debiasing word embeddings,\u201d arXiv.org<\/em>, July 21, 2016. [Online]. Available: https:\/\/arxiv.org\/abs\/1607.06520<\/a><\/p>\n

<\/a>
\n9. J. Zhao, T. Wang, M. Yatskar, V. Ordonez, and K. Chang, \u201cMen also like shopping: Reducing gender bias amplification using Corpus-level constraints,\u201d arXiv.org<\/em>, July 29, 2017.[Online]. Available: https:\/\/arxiv.org\/pdf\/1707.09457.pdf<\/a><\/p>\n

<\/a>
\n10. A. Feng and S. Wu, \u201cThe myth of the impartial machine,\u201d Parametric Press<\/em>, no. 01 (Science + Society), May 1, 2019. [Online]. Available: https:\/\/parametric.press\/issue-01\/the-myth-of-the-impartial-machine\/<\/a><\/p>\n

<\/a>
\n11. D. Sculley et al., \u201dHidden technical debt in machine learning systems,\u201d in Advances in Neural Information Processing Systems 28 (NIPS 2015)<\/em>. Accessed March 16, 2020. [Online]. Available: https:\/\/papers.nips.cc\/paper\/5656-hidden-technical-debt-in-machine-learning-systems.pdf<\/a><\/p>\n

<\/a>
\n12. A. Feng and S. Wu, \u201cThe myth of the impartial machine,\u201d Parametric Press<\/em>, no. 01 (Science + Society), May 1, 2019. [Online]. Available: https:\/\/parametric.press\/issue-01\/the-myth-of-the-impartial-machine\/<\/a><\/p>\n

<\/a>
\n13. D. Sculley et al., \u201cHidden technical debt in machine learning systems,\u201d in Advances in Neural Information Processing Systems 28 (NIPS 2015)<\/em>. Accessed March 16, 2020. [Online]. Available: https:\/\/papers.nips.cc\/paper\/5656-hidden-technical-debt-in-machine-learning-systems.pdf<\/a><\/p>\n

<\/a>
\n14. Z. Rogers, \u201cHave strategists drunk the \u2018AI race\u2019 Kool-Aid,\u201d War on the Rocks<\/em>, June 4, 2019. [Online]. Available: https:\/\/warontherocks.com\/2019\/06\/have-strategists-drunk-the-ai-race-kool-aid\/<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

Make Our Assumptions Explicit<\/strong><\/p>\n

<\/a>
\n1. J. Stoyanovich and B. Howe, \u201cFollow the data! Algorithmic transparency starts with data transparency,\u201d Shorenstein Center on Media, Politics and Public Policy, Harvard Kennedy School, Nov. 27, 2018. [Online]. Available: https:\/\/ai.shorensteincenter.org\/ideas\/2018\/11\/26\/follow-the-data-algorithmic-transparency-starts-with-data-transparency<\/a><\/p>\n

<\/a>
\n2. T. Gebru et al., \u201cDatasheets for datasets,\u201d arXiv.org<\/em>, Jan. 14, 2020. [Online]. Available: [Online]. Available: https:\/\/arxiv.org\/abs\/1803.09010<\/a><\/p>\n

<\/a>
\n3. \u201cAlgorithms and artificial intelligence: CNIL\u2019s report on the ethical issues,\u201d CNIL [Commission Nationale de l’Informatique et des Libert\u00e9s], May 25, 2018. [Online]. Available: https:\/\/www.cnil.fr\/en\/algorithms-and-artificial-intelligence-cnils-report-ethical-issues<\/a><\/p>\n

<\/a>
\n4. M. Mitchell et al., \u201cModel cards for model reporting,\u201d arXiv.org<\/em>, Jan. 14, 2019. [Online]. Available: [Online]. Available: https:\/\/arxiv.org\/abs\/1810.03993<\/a><\/p>\n

<\/a>
\n5. \u201cAbout Us,\u201d Partnership On AI. <\/em>Accessed May 27, 2020. [Online]. Available: https:\/\/www.partnershiponai.org\/about\/<\/a><\/p>\n

<\/a>
\n6. \u201cDeployed Examples,\u201d Partnership On AI. <\/em>Accessed May 27, 2020. [Online]. Available: https:\/\/www.partnershiponai.org\/about-ml\/#examples<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

Try Human-AI Couples Counseling<\/strong><\/p>\n

<\/a>
\n1. J. M. Bradshaw, R. Hoffman, M. Johnson, and D. D. Woods, \u201cThe seven deadly myths of \u2018autonomous systems,\u2019\u201d IEEE: Intelligent Systems<\/em>, vol. 28, no. 3, pp. 54-61, May 2013. [Online]. Available: https:\/\/ieeexplore.ieee.org\/document\/6588858<\/a><\/p>\n

<\/a>
\n2. J. M. Bradshaw, R. Hoffman, M. Johnson, and D. D. Woods, \u201cThe seven deadly myths of \u2018autonomous systems,\u2019\u201d IEEE: Intelligent Systems<\/em>, vol. 28, no. 3, pp. 54-61, May 2013. [Online]. Available: https:\/\/ieeexplore.ieee.org\/document\/6588858<\/a><\/p>\n

<\/a>
\n3. S. M. Casner and E. L. Hutchins, \u201cWhat do we tell the drivers? Toward minimum driver training standards for partially automated cars,\u201d Journal of Cognitive Engineering and Decision Making<\/em>, March 8, 2019. [Online]. Available: https:\/\/journals.sagepub.com\/doi\/full\/10.1177\/1555343419830901<\/a><\/p>\n

<\/a>
\n4. M. Johnson, J. M. Bradshaw, R. R. Hoffman, P. J. Feltovich, and D. D. Woods, \u201cSeven cardinal virtues of human-machine teamwork: Examples from the DARPA robotic challenge,\u201d IEEE Intelligent Systems<\/em>, Nov.\/Dec. 2014. [Online]. Available: http:\/\/www.jeffreymbradshaw.net\/publications\/56.%20Human-Robot%20Teamwork_IEEE%20IS-2014.pdf<\/a><\/p>\n

<\/a>
\n5. J. M. Bradshaw, R. Hoffman, M. Johnson, and D. D. Woods, \u201cThe seven deadly myths of \u2018autonomous systems,\u2019\u201d IEEE: Intelligent Systems<\/em>, vol. 28, no. 3, pp. 54-61, May 2013. [Online]. Available: https:\/\/ieeexplore.ieee.org\/document\/6588858<\/a><\/p>\n

<\/a>
\n6. M. Johnson, J. M. Bradshaw, R. R. Hoffman, P. J. Feltovich, and D. D. Woods, \u201cSeven cardinal virtues of human-machine teamwork: Examples from the DARPA robotic challenge,\u201d IEEE Intelligent Systems<\/em>, Nov.\/Dec. 2014. [Online]. Available: http:\/\/www.jeffreymbradshaw.net\/publications\/56.%20Human-Robot%20Teamwork_IEEE%20IS-2014.pdf<\/a><\/p>\n

<\/a>
\n7. G. Klein et al., \u201cTen challenges for making automation a \u2019team player\u2019 in joint human-agent activity,\u201d IEEE: Intelligent Systems<\/em>, vol. 19, no. 6, pp. 91-95, Nov.\/Dec. 2004. [Online]. Available: http:\/\/jeffreymbradshaw.net\/publications\/17._Team_Players.pdf_1.pdf<\/a><\/p>\n

<\/a>
\n8. W. Lawless, R. Mittu, D. Sofge, and L. Hiatt, \u201cArtificial intelligence, autonomy, and human-machine teams\u2014Interdependence, context, and explainable AI,\u201d AI Magazine<\/em>, vol. 40, no. 3, pp. 5-13, 2019.<\/p>\n

<\/a>
\n9. \u201cA framework for discussing trust in increasingly autonomous systems,\u201d The MITRE Corporation, June 2017. [Online]. Available: https:\/\/www.mitre.org\/sites\/default\/files\/publications\/17-2432-framework-discussing-trust-increasingly-autonomous-systems.pdf<\/a><\/p>\n

\u00a0<\/strong><\/p>\n

\u00a0<\/strong><\/p>\n

Offer the User Choices<\/strong><\/p>\n

<\/a>
\n1. M. Kearns, \u201cThe ethical algorithm,\u201d Carnegie Council for Ethics in International Affairs, Nov. 6, 2019. [Online]. Available: https:\/\/www.carnegiecouncil.org\/studio\/multimedia\/20191106-the-ethical-algorithm-michael-kearns<\/a><\/p>\n

<\/a>
\n2. J. Stoyanovich and B. Howe, \u201cFollow the data! Algorithmic transparency starts with data transparency,\u201d Shorenstein Center on Media, Politics and Public Policy, Harvard Kennedy School, Nov. 27, 2018. [Online]. Available: https:\/\/ai.shorensteincenter.org\/ideas\/2018\/11\/26\/follow-the-data-algorithmic-transparency-starts-with-data-transparency<\/a><\/p>\n

<\/a>
\n3. L. M. Strickhart and H.N.J. Lee, \u201cShow your work: Machine learning explainer tools and their use in artificial intelligence assurance,\u201d The MITRE Corporation, McLean, VA, June 2019, unpublished.<\/p>\n

<\/a>
\n4. \u201cAlgorithms and artificial intelligence: CNIL\u2019s report on the ethical issues,\u201d CNIL [Commission Nationale de l’Informatique et des Libert\u00e9s], May 25, 2018. [Online]. Available: https:\/\/www.cnil.fr\/en\/algorithms-and-artificial-intelligence-cnils-report-ethical-issues<\/a><\/p>\n

\u00a0<\/strong><\/p>\n

 <\/p>\n

Promote Better Adoption through Gameplay<\/strong>
\n<\/a>
\n1. N. D. Sarter and D. D. Woods, \u201cHow in the world did I ever get into that mode? Mode error and awareness in supervisory control,\u201d Human Factors<\/em>, vol. 37, pp. 5-19, 1995.<\/p>\n

<\/a>
\n2. \u201cVirtuous cycle of AI: Build good product, get more users, collect more data, build better product, get more users, collect more data, etc.,\u201d in A. Ng, AI Transformation Playbook: How to Lead Your Company into the AI Era<\/em>, Landing AI, Dec. 13, 2018. [Online]. Available: https:\/\/landing.ai\/ai-transformation-playbook\/<\/a>.<\/p>\n

<\/a>
\n3. J. Whittlestone, A. Alexandrova, R. Nyrup, and S. Cave, \u201cThe role and limits of principles in AI ethics: Towards a focus on tensions,\u201d presented at AIES \u201919, Jan. 27\u201328, 2019, Honolulu, HI, USA. [Online]. Available: https:\/\/www.researchgate.net\/publication\/334378492_The_Role_and_Limits_of_Principles_in_AI_Ethics_Towards_a_Focus_on_Tensions\/link\/5d269de0a6fdcc2462d41592\/download<\/a><\/p>\n

<\/a>
\n4. \u201cProject ExplAIn interim report,\u201d U.K. Information Commissioner\u2019s Office, 2019. [Online]. Available: https:\/\/ico.org.uk\/about-the-ico\/research-and-reports\/project-explain-interim-report\/<\/a><\/p>\n

<\/a>
\n5. \u201cSquad X improves situational awareness, coordination for dismounted units,\u201d Defense Advanced Research Projects Agency, Nov. 30, 2018. [Online]. Available: https:\/\/www.darpa.mil\/news-events\/2018-11-30a<\/a><\/p>\n

<\/a>
\n6. DARPAtv, \u201cSquad X experimentation exercise,\u201d YouTube, July 12, 2019. [Online]. Available; https:\/\/www.youtube.com\/watch?v=DgM7hbCNMmU<\/a><\/p>\n

<\/a>
\n7. S. J. Freedberg, \u201cSimulating a super brain: Artificial intelligence in wargames,\u201d Breaking Defense, April 26, 2019. [Online]. Available: https:\/\/breakingdefense.com\/2019\/04\/simulating-a-super-brain-artificial-intelligence-in-wargames\/<\/a>.<\/p>\n

<\/a>
\n8. B. Jensen, S. Cuomo, and C. Whyte, \u201cWargaming with Athena: How to make militaries smarter, faster, and more efficient with artificial intelligence,\u201d War on the Rocks<\/em>, June 5, 2018. [Online]. Available: https:\/\/warontherocks.com\/2018\/06\/wargaming-with-athena-how-to-make-militaries-smarter-faster-and-more-efficient-with-artificial-intelligence\/<\/a><\/p>\n

<\/a>
\n9. J. Whittlestone, A. Alexandrova, R. Nyrup, and S. Cave, \u201cThe role and limits of principles in AI ethics: Towards a focus on tensions,\u201d presented at AIES \u201919, Jan. 27\u201328, 2019, Honolulu, HI, USA. [Online]. Available: https:\/\/www.researchgate.net\/publication\/334378492_The_Role_and_Limits_of_Principles_in_AI_Ethics_Towards_a_Focus_on_Tensions\/link\/5d269de0a6fdcc2462d41592\/download<\/a><\/p>\n

\u00a0<\/strong><\/p>\n

 <\/p>\n

Monitor the AI\u2019s Impact and Establish Chains of Accountability<\/strong><\/p>\n

<\/a>
\n1. A. Gonfalonieri, \u201cWhy machine learning models degrade in production,\u201d towards data science, July 25, 2019. [Online]. Available: https:\/\/towardsdatascience.com\/why-machine-learning-models-degrade-in-production-d0f2108e9214<\/a><\/p>\n

<\/a>
\n2. A. Gonfalonieri, \u201cWhy machine learning models degrade in production,\u201d towards data science, July 25, 2019. [Online]. Available: https:\/\/towardsdatascience.com\/why-machine-learning-models-degrade-in-production-d0f2108e9214<\/a><\/p>\n

<\/a>
\n3. A. Gonfalonieri, \u201cWhy machine learning models degrade in production,\u201d towards data science, July 25, 2019. [Online]. Available: https:\/\/towardsdatascience.com\/why-machine-learning-models-degrade-in-production-d0f2108e9214<\/a><\/p>\n

<\/a>
\n4. A. Campolo et al., AI Now Report 2017<\/em>. New York, NY, USA: AI Now Institute, 2017. [Online]. Available:\u00a0 https:\/\/ainowinstitute.org\/AI_Now_2017_Report.pdf<\/a><\/p>\n

<\/a>
\n5. \u201cAlgorithms and artificial intelligence: CNIL\u2019s report on the ethical issues,\u201d CNIL [Commission Nationale de l’Informatique et des Libert\u00e9s], May 25, 2018. [Online]. Available: https:\/\/www.cnil.fr\/en\/algorithms-and-artificial-intelligence-cnils-report-ethical-issues<\/a><\/p>\n

<\/a>
\n6. J. C. Newman, \u201cDecision Points in AI Governance,\u201d UC Berkeley Center for Long-Term Cybersecurity, <\/em>May 5, 2020. [Online]. Available: https:\/\/cltc.berkeley.edu\/2020\/05\/05\/decision-points-in-ai-governance\/<\/a><\/p>\n

<\/a>
\n7. T. Hagendorff, \u201cThe ethics of AI ethics: An evaluation of guidelines,\u201d arXiv.org<\/em>, Oct. 11, 2019. [Online]. Available: https:\/\/arxiv.org\/abs\/1903.03425<\/a><\/p>\n

<\/a>
\n8. J. C. Newman, \u201cDecision Points in AI Governance,\u201d UC Berkeley Center for Long-Term Cybersecurity, <\/em>May 5, 2020. [Online]. Available: https:\/\/cltc.berkeley.edu\/2020\/05\/05\/decision-points-in-ai-governance\/<\/a><\/p>\n

\u00a0<\/strong><\/p>\n

 <\/p>\n

Envision Safeguards for AI Advocates<\/strong><\/p>\n

<\/a>
\n1. R. Sandler, \u201cAmazon, Microsoft, Wayfair: Employees stage internal protests against working with ICE,\u201d Forbes<\/em>, July 19, 2019. [Online]. Available: https:\/\/www.forbes.com\/sites\/rachelsandler\/2019\/07\/19\/amazon-salesforce-wayfair-employees-stage-internal-protests-for-working-with-ice\/<\/a><\/p>\n

<\/a>
\n2. J. Bhuiyan, \u201cHow the Google walkout transformed tech workers into activists,\u201d Los Angeles Times<\/em>, Nov. 6, 2019. [Online]. Available: https:\/\/www.latimes.com\/business\/technology\/story\/2019-11-06\/google-employee-walkout-tech-industry-activism<\/a><\/p>\n

<\/a>
\n3. J. McLaughlin, Z. Dorfman, and S. D. Naylor, \u201cPentagon intelligence employees raise concerns about supporting domestic surveillance amid protests,\u201d Yahoo News,<\/em> June 4, 2020. [Online]. Available: https:\/\/news.yahoo.com\/pentagon-intelligence-employees-raise-concerns-about-supporting-domestic-surveillance-amid-protests-194906537.html<\/a><\/p>\n

<\/a>
\n4. J. Menn, \u201cGoogle fires fifth activist employee in three weeks; complaint filed,\u201d Reuters, <\/em>Dec. 17, 2019. [Online]. Available: https:\/\/www.reuters.com\/article\/google-unions\/google-fires-fifth-activist-employee-in-three-weeks-complaint-filed-idUSL1N28R02L<\/a><\/p>\n

<\/a>
\n5. A. Palmer, \u201cAmazon employees plan \u2018online walkout\u2019 to protest firings and treatment of warehouse workers,\u201d CNBC, <\/em>April 16, 2020. [Online]. Available: https:\/\/www.cnbc.com\/2020\/04\/16\/amazon-employees-plan-online-walkout-over-firings-work-conditions.html<\/a><\/p>\n

<\/a>
\n6. J. Eidelson and H. Kanu, \u201cSoftware Startup Accused of Union-Busting Will Pay Ex-Employees,\u201d Bloomberg,<\/em> Nov. 10, 2018. [Online]. Available: https:\/\/www.bloomberg.com\/news\/articles\/2018-11-10\/software-startup-accused-of-union-busting-will-pay-ex-employees<\/a><\/p>\n

<\/a>
\n7. M. Whittaker et al., AI Now Report 2018<\/em>. New York, NY, USA: AI Now Institute, 2018. [Online]. Available: https:\/\/ainowinstitute.org\/AI_Now_2018_Report.pdf<\/a><\/p>\n

<\/a>
\n8. \u201cAlgorithms and artificial intelligence: CNIL\u2019s report on the ethical issues,\u201d CNIL [Commission Nationale de l’Informatique et des Libert\u00e9s], May 25, 2018. [Online]. Available: https:\/\/www.cnil.fr\/en\/algorithms-and-artificial-intelligence-cnils-report-ethical-issues<\/a><\/p>\n

 <\/p>\n

 <\/p>\n

Require Objective, Third-party Verification and Validation<\/strong>
\n<\/a>
\n1. \u201cAlgorithms and artificial intelligence: CNIL\u2019s report on the ethical issues,\u201d CNIL [Commission Nationale de l’Informatique et des Libert\u00e9s], May 25, 2018. [Online]. Available: https:\/\/www.cnil.fr\/en\/algorithms-and-artificial-intelligence-cnils-report-ethical-issues<\/a><\/p>\n

<\/a>
\n2. M. Whittaker et al., AI Now Report 2018<\/em>. New York, NY, USA: AI Now Institute, 2018. [Online]. Available: https:\/\/ainowinstitute.org\/AI_Now_2018_Report.pdf<\/a><\/p>\n

<\/a>
\n3. ENERGY STAR homepage. Accessed on: Jan. 21, 2020. [Online]. Available: https:\/\/www.energystar.gov\/<\/a><\/p>\n

<\/a>
\n4. C. Martin and M. Dent, \u201cHow Nestle, Google and other businesses make money by going green,\u201d Los Angeles Times<\/em>, Sep. 20, 2019. [Online]. Available: https:\/\/www.latimes.com\/business\/story\/2019-09-20\/how-businesses-profit-from-environmentalism<\/a><\/p>\n

<\/a>
\n5. \u201cSafeAI.\u201d Accessed April 2, 2020. [Online]. Available:\u00a0\u00a0 https:\/\/www.forhumanity.center\/safeai\/<\/a><\/p>\n

<\/a>
\n6. A. Campolo et al., AI Now Report 2017<\/em>. New York, NY, USA: AI Now Institute, 2017. [Online]. Available:\u00a0 https:\/\/ainowinstitute.org\/AI_Now_2017_Report.pdf<\/a><\/p>\n

<\/a>
\n7. J. Stoyanovich and B. Howe, \u201cFollow the data! Algorithmic transparency starts with data transparency,\u201d Shorenstein Center on Media, Politics and Public Policy, Harvard Kennedy School, Nov. 27, 2018. [Online]. Available: https:\/\/ai.shorensteincenter.org\/ideas\/2018\/11\/26\/follow-the-data-algorithmic-transparency-starts-with-data-transparency<\/a><\/p>\n

<\/a>
\n8. M. Whittaker et al., AI Now Report 2018<\/em>. New York, NY, USA: AI Now Institute, 2018. [Online]. Available: https:\/\/ainowinstitute.org\/AI_Now_2018_Report.pdf<\/a><\/p>\n

<\/a>
\n9. Z. C. Lipton, \u201cThe doctor just won\u2019t accept that,\u201d arXiv.org<\/em>, Nov. 24, 2017. [Online]. Available: https:\/\/arxiv.org\/abs\/1711.08037<\/a><\/p>\n

<\/a>
\n10. \u201cAlgorithms and artificial intelligence: CNIL\u2019s report on the ethical issues,\u201d CNIL [Commission Nationale de l’Informatique et des Libert\u00e9s], May 25, 2018. [Online]. Available: https:\/\/www.cnil.fr\/en\/algorithms-and-artificial-intelligence-cnils-report-ethical-issues<\/a><\/p>\n

<\/a>
\n11. M. Whittaker et al., AI Now Report 2018<\/em>. New York, NY, USA: AI Now Institute, 2018. [Online]. Available: https:\/\/ainowinstitute.org\/AI_Now_2018_Report.pdf<\/a><\/p>\n

<\/a>
\n12. Occupational Safety and Health Administration, \u201cOSHA\u2019s Nationally Recognized Testing Laboratory (NRTL) program,\u201d OSHA.gov<\/em>. Accessed on: Jan. 30, 2020. [Online]. Available: https:\/\/www.osha.gov\/dts\/otpca\/nrtl\/<\/a><\/p>\n

\u00a0<\/strong><\/p>\n

 <\/p>\n

Entrust Sector-specific Agencies to Establish AI Standards for Their Domains<\/strong>
\n<\/a>
\n1. M. Whittaker et al., AI Now Report 2018<\/em>. New York, NY, USA: AI Now Institute, 2018. [Online]. Available: https:\/\/ainowinstitute.org\/AI_Now_2018_Report.pdf<\/a><\/p>\n

<\/a>
\n2. F. Balamuth et al., \u201cImproving recognition of pediatric severe sepsis in the emergency department: Contributions of a vital sign\u2013based electronic alert and bedside clinician identification,\u201d Annals of Emergency Medicine<\/em>, vol. 79, no. 6, pp. 759-768.e2, Dec. 2017. [Online]. Available: https:\/\/www.sciencedirect.com\/science\/article\/abs\/pii\/S0196064417303153<\/a><\/p>\n

<\/a>
\n3. G. Siddiqui, \u201cWhy doctors reject tools that make their jobs easier,\u201d Scientific American<\/em>, Oct. 15, 2018. [Online]. Available: https:\/\/blogs.scientificamerican.com\/observations\/why-doctors-reject-tools-that-make-their-jobs-easier\/<\/a><\/p>\n

<\/a>
\n4. A. M. Barry-Jester, B. Casselman, and D. Goldstein, \u201cShould prison sentences be based on crimes that haven\u2019t been committed yet?\u2019 FiveThirtyEight<\/em>, Aug. 4, 2015. [Online]. Available: https:\/\/fivethirtyeight.com\/features\/prison-reform-risk-assessment\/<\/a><\/p>\n

<\/a>
\n5. J. Angwin, J. Larson, S. Mattu, and L. Kirchner, \u201cMachine bias,\u201d ProPublica<\/em>, May 23, 2016. [Online]. Available: https:\/\/www.propublica.org\/article\/machine-bias-risk-assessments-in-criminal-sentencing<\/a><\/p>\n

<\/a>
\n6. S. Corbett-Davies, E. Pierson, A. Feller, and S. Goel, \u201cA computer program used for bail and sentencing decisions was labeled biased against blacks. It\u2019s actually not that clear,\u201d Washington Post<\/em>, Oct. 17, 2016. [Online]. Available: https:\/\/www.washingtonpost.com\/news\/monkey-cage\/wp\/2016\/10\/17\/can-an-algorithm-be-racist-our-analysis-is-more-cautious-than-propublicas\/?noredirect=on&utm_term=.a9cfb19a549d<\/a><\/p>\n

<\/a>
\n7. \u201cCase of first impression,\u201d Legal Dictionary<\/em>, March 21, 2017. [Online]. Available: https:\/\/legaldictionary.net\/case-first-impression\/<\/a><\/p>\n

<\/a>
\n8. \u201cFair cross section requirement,\u201d Stephen G. Rodriquez & Partners. Accessed on: Jan. 21, 2020. [Online]. Available: https:\/\/www.lacriminaldefenseattorney.com\/legal-dictionary\/f\/fair-cross-section-requirement\/<\/a><\/p>\n

<\/a>
\n9. I. Masic, M. Miokovic, and B. Muhamedagic, \u201cEvidence based medicine\u2014new approaches and challenges,\u201d Acta Informatica Medica<\/em>, vol. 16, no. 4, pp. 219-225, 2018. [Online]. Available: https:\/\/www.ncbi.nlm.nih.gov\/pmc\/articles\/PMC3789163\/<\/a><\/p>\n

<\/a>
\n10. \u201cHippocratic Oath,\u201d Encyclopaedia Britannica<\/em>, Dec. 4, 2019. [Online]. Available: https:\/\/www.britannica.com\/topic\/Hippocratic-oath<\/a><\/p>\n

<\/a>
\n11. R. Vought, \u201cGuidance for regulation of artificial intelligence applications,\u201d Draft memorandum, WhiteHouse.gov<\/em>. Accessed on: Jan. 21, 2020. [Online]. Available: https:\/\/www.whitehouse.gov\/wp-content\/uploads\/2020\/01\/Draft-OMB-Memo-on-Regulation-of-AI-1-7-19.pdf<\/a><\/p>\n

<\/a>
\n12. G. Vyse, \u201cThree American cities have now banned the use of facial recognition technology in local government amid concerns it’s inaccurate and biased,\u201d Governing<\/em>, July 24, 2019. [Online]. Available: https:\/\/www.governing.com\/topics\/public-justice-safety\/gov-cities-ban-government-use-facial-recognition.html<\/a><\/p>\n

<\/a>
\n13. \u201cAlgorithms and artificial intelligence: CNIL\u2019s report on the ethical issues,\u201d CNIL [Commission Nationale de l’Informatique et des Libert\u00e9s], May 25, 2018. [Online]. Available: https:\/\/www.cnil.fr\/en\/algorithms-and-artificial-intelligence-cnils-report-ethical-issues<\/a><\/p>\n

<\/a>
\n14. A. Dafoe, \u201cAI governance: A research agenda,\u201d Future of Humanity Institute, University of Oxford, Oxford, UK, Aug. 27, 2018.\u00a0 [Online]. Available: https:\/\/www.fhi.ox.ac.uk\/wp-content\/uploads\/GovAIAgenda.pdf<\/a><\/p>\n

[\/et_pb_text][et_pb_sidebar area=”sidebar-1″ _builder_version=”4.4.8″][\/et_pb_sidebar][\/et_pb_column][\/et_pb_row][\/et_pb_section][et_pb_section fb_built=”1″ fullwidth=”on” disabled_on=”off|off|off” _builder_version=”4.4.8″ module_alignment=”center” global_module=”3880″ saved_tabs=”all”][et_pb_fullwidth_code disabled_on=”off|off|off” admin_label=”Footer menu” _builder_version=”4.5.0″ background_color=”#d5dde0″ text_orientation=”center” module_alignment=”center” custom_padding=”10px||10px||false|false”]Add Your Experience! This site should be a community resource and would benefit from your examples and voices. You can write to us by clicking here<\/a>.[\/et_pb_fullwidth_code][\/et_pb_section]<\/p>\n","protected":false},"excerpt":{"rendered":"

References \u00a0 Home Page 1. \u201cBenjamin Franklin quotable quote,\u201d Goodreads. Accessed March 16, 2020. [Online]. Available: https:\/\/www.goodreads.com\/quotes\/460142-if-you-fail-to-plan-you-are-planning-to-fail 2. Department of Defense, \u201cSummary of the 2018 Department of Defense Artificial Intelligence Strategy: Harnessing AI to Advance Our Security and Prosperity,\u201d defense.gov, February 12, 2019. [Online]. Available: https:\/\/media.defense.gov\/2019\/Feb\/12\/2002088963\/-1\/-1\/1\/SUMMARY-OF-DOD-AI-STRATEGY.PDF 3. ichristianization, \u201cMicrosoft build 2017 translator demo,\u201d YouTube, June […]<\/p>\n","protected":false},"author":142,"featured_media":0,"parent":0,"menu_order":2,"comment_status":"closed","ping_status":"closed","template":"","meta":{"_acf_changed":false,"_et_pb_use_builder":"on","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"class_list":["post-65","page","type-page","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/sites.mitre.org\/aifails\/wp-json\/wp\/v2\/pages\/65","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/sites.mitre.org\/aifails\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/sites.mitre.org\/aifails\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/sites.mitre.org\/aifails\/wp-json\/wp\/v2\/users\/142"}],"replies":[{"embeddable":true,"href":"https:\/\/sites.mitre.org\/aifails\/wp-json\/wp\/v2\/comments?post=65"}],"version-history":[{"count":0,"href":"https:\/\/sites.mitre.org\/aifails\/wp-json\/wp\/v2\/pages\/65\/revisions"}],"wp:attachment":[{"href":"https:\/\/sites.mitre.org\/aifails\/wp-json\/wp\/v2\/media?parent=65"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}