Lessons Learned

I. Expand Early Project Considerations

Hold AI to a Higher Standard

​New technology applications, and the companies that develop those technologies, are increasingly using AI and more advanced forms of automation. The problems present in previous generations of automated technology are now exacerbated by the scope and scale of AI. How?

1. An AI system replicates the social values of its developers and also embeds them into systems

As developers and deployers, our choices, assumptions, simplifications, and trade-offs all shape the behavior of the system, and we can (intentionally or not) encode those values as the new standard. All too often those values represent how young, white, technically oriented, Western men interact with the world. We need to improve our outreach to and understanding of a far broader set of stakeholder communities.

2. An AI system’s reach can centralize power in the hands of a few

If one person makes a decision or influences one other person’s behavior, the effects are limited. But an AI allows us to aggregate and amplify our influence over many people’s behaviors. Even an entirely automated decision is never neutral – outcomes always affect people differently. Therefore, we should explore how AI changes human behavior at scale, and apply what we learn to the AI we create.1,2

3. People can be influenced to trust AI more than they should

In certain conditions, people place more trust in an AI than is warranted, because they assume it is more impartial and infallible than they are. Individuals also have cognitive biases that lead them to treat connections and correlations as conclusions and inferences. Because AI can connect exponentially more information than a small group can on its own, it can magnify the effects of false or misleading conclusions. We should do our best to ensure that the trust people place in the AI is matched by a higher degree of trustworthiness.

4. There is a tension between global pressures to develop and deploy AI quickly, and understanding and mitigating an AI’s impacts

When AI systems scale, or act so fast that humans cannot respond in time, then humans must rely on the guardrails and risk mitigation practices incorporated in the system. If these protections and practices are limited because developers focused on deploying AI as rapidly as possible, the chances for unwanted outcomes increase. Therefore, we need to ensure we integrate risk assessment and mitigation protections early in the AI’s development and throughout the system’s lifecycle.

5. It is unclear who is accountable for an AI system’s decisions

As of today, legal responsibility for the consequences of AI system use has not been established, and this results in a lack of accountability or in holding the wrong person accountable.3 When no one is considered legally accountable if something goes wrong, and no one is made responsible for fixing it, the consequences of mistakes and misuse can easily lead to abuse of privacy and civil rights. We have to exercise particular care to reach out to those who best understand the domain and risks, and be more inclusive in our design teams as a way to prevent bad outcomes to the extent possible.

We are in the best position to recognize the potential impacts of this technology. If we hold AI to a higher standard, our example has the potential to raise the standard across the board. If we establish rigorous practices for quality control and assurance within our organizations, then other AI vendors will feel pressure to match the evolved market expectations. When companies and the government set standards for workforce training, AI team composition, and governance practices, those standards become a baseline for a common lexicon, curricula in universities, and expectations across the public, private, and academic sectors.4

The rest of the lessons learned provide more detail on specific aspects of ensuring proper use of AI and offer actionable implementation guidance.

It’s OK to Say No to Automation

The first things we should ask when starting an AI project is simply, “Is this actually a problem that we need AI to address? Can AI even be effective for this purpose?” Our end goal is really to meet stakeholder needs, independent of the particular technology or approach we choose.1

Sometimes, automation is simply not the right choice

As a general rule, the more the outcome should depend on human judgment, the more “artificial” an AI solution is. Some more guidelines follow:

• Our AI systems should incorporate more human judgment and teaming as applications and environments become more complex or dynamic.
• We should enlist human scrutiny to ensure that the data we use is relevant and representative of our purposes, and that there is no historical pattern of bias and discrimination in the data and application domain.
• If the risk of using the data or the purpose of the AI could cause financial, psychological, physical, or other types of harm, then we must ask whether we should create or deploy the AI at all.2

As a general rule, the more the outcome should depend on human judgment, the more “artificial” an AI solution is

Applying AI more selectively will help stakeholders accept that those AI solutions are appropriate. Distinguishing which challenges would benefit from AI and which challenges do not lend themselves to AI, gives customers and the public more confidence that AI is deployed responsibly, justifiably, and in consideration of existing norms and public safety.

AI Challenges Are Multidisciplinary, so They Require a Multidisciplinary Team

If we do not actively work to incorporate other valid perspectives into the development process, we risk having the AI reflect our assumptions about how the product will be used and by whom, instead of being based on research evidence and empirical data.

The challenges to overcome when developing or implementing AI are diverse and can be both technical and social in nature. As a result, no one person or discipline can singlehandedly “fix” AI. Those of us on the front lines of building the AI share many attributes (i.e., similar education and degrees, life experiences, and cultural backgrounds).1 If we do not actively work to incorporate other valid perspectives into the development process, we risk having the AI reflect our assumptions about how the product will be used and by whom, instead of being based on research evidence and empirical data.

Therefore, our development teams need members with diverse demographic and professional backgrounds. Examples of members of a well-rounded team include:

• Data engineers to ensure that data is usable and relevant
• Model developers to help the AI achieve the project’s objectives
• Strategic decision makers who understand the technical aspects of AI as well as broader strategic issues or business needs
• Domain specialists to supply context about how people in their field actually behave, existing business practices, and any historical biases. Domain experts can be scientific or non-scientific; they may be military personnel, teachers, doctors and patients, artists … any people who are actual experts in the area for which the AI is being designed.2
• Qualitative experts or social scientists to help technologists and decision makers clarify ideas, create metrics, and objectively examine factors that would affect adoption of the AI
• Human factors or cognitive engineers to help ensure that AI is not just integrated into a technology or process, but is adopted willingly and with appropriately calibrated trust
• Accident analysis experts who can draw on a long history of post-accident insights and frameworks to improve system design and anticipate areas of concern
• Legal and policy experts to oversee that data use and governance are covered by relevant authorities, to identify legal implications of the deployed AI, and to ensure that the process is following established mechanisms of oversight.
• Privacy, civil liberties, and cybersecurity experts to help evaluate and if necessary mitigate how design choices could affect concerns in their respective areas
• The users of the AI and the communities that will be affected by the AI to reinforce the importance of meeting the desired outcomes of all stakeholders
•  

The most successful teams are ones in which all perspectives are voiced and considered. To that end, we must remember to not only include multidisciplinary experts on the team, but also make sure that all teammates have equal decision-making power.3

Incorporate Privacy, Civil Liberties, and Security from the Beginning

Let’s borrow and extend the “Fundamental Theorem of Security” stated by Roman Yampolskiy, a professor at the University of Louisville, to say, “Every security system will eventually fail; {every piece of data collected will be used in unanticipated ways}. If your system has not failed, just wait longer.”1 (text in curly braces represents additions to the quotation).

Many AI-enabled systems rely on growing amounts of data in order to enable more accurate and more tailored pattern recognition. As that data becomes increasingly personal and sensitive, the costs that result from those datasets being misused, stolen, or more intricately connected become much greater and more alarming. Privacy, civil liberties, and security experts are now more essential to AI development than ever, because they specialize in recognizing and mitigating against the ways in which data can be used in unforeseen ways.2

We must consider privacy-, civil liberties-, security-, and mission-related objectives at the beginning of the development project, when we can evaluate tradeoffs among the four.

To aid us in understanding the risks involved and being proactive in preventing those risks, experts in these fields can help us navigate and resolve some of the following tensions:

• Collecting and using more data to achieve better quality outcomes vs. respecting individuals’ privacy and ownership over their data3
• Making models or datasets openly available to the public for broader use and scrutiny vs. revealing more information that lets adversaries find new ways to hack the information4
• Meeting consumer demand for products that are becoming more integrated into their homes (and bodies) vs. mitigating the increasing consequences to their safety when those devices fail or are hacked5
• Balancing data and privacy protection in legislation, such as in Europe’s General Data Protection Regulation (GDPR). Current policy differs across countries6,7,8 and states.9,10

Every security system will eventually fail; {every piece of data collected will be used in unanticipated ways}. If your system has not failed, just wait longer.

These considerations cannot be afterthoughts. Too often, the seductive values of cost savings and efficiencies blind commercial and government organizations to the need for addressing privacy, civil liberties, and security concerns adequately. Incorporating this expertise on our teams early offers a means for developing AI systems that can meet mission needs and simultaneously address these considerations.

II. Build Resiliency into the AI and the Organization

Involve the Communities Affected by the AI

​When we design an application with only the end-user in mind, the application can have very different objectives and success criteria than if we design for the communities that the AI will affect.

Treating these communities as customers, and even giving them a vote in choosing success criteria for the algorithm, is another step that would lead toward more human-centric outcomes.

Two particularly powerful examples of one-sided implementation – AIs that recommend which patients receive healthcare, and facial recognition AIs used for policing (see more in the "AI Developers Are Wizards and Operators Are Muggles" Fail) illustrate that both end-users and affected communities may be able to find common ground on desired outcomes if given the opportunity. But since the affected communities were not invited to discussions with AI developers, the developers did not design the system to reflect the communities’ perspectives.

Therefore, we should be sure to include representatives from the communities that will be affected by the algorithm, in addition to the end-users. Treating these communities as customers, and even giving them a vote in choosing success criteria for the algorithm, is another step that would lead toward more human-centric outcomes.1

These conversations should start early and continue past algorithm deployment. The University of Washington’s Tech Policy Lab offers a step-by-step guide for facilitating inclusivity in technology policy.2 It includes actions that can help organizations identify appropriate stakeholder groups, run group sessions, and close the loop between developers and the invited communities.

Why are these types of approaches so necessary?

Education and exposure are powerful tools. They help us fill gaps in our knowledge: they help us to learn about communities’ previous experiences with automation, and they give us insight regarding the level of explainability and transparency required for successful outcomes. In turn, those communities and potential users of the AI can learn how the AI works, align their expectations to the actual capabilities of the AI, and understand the risks involved in relying on the AI. Involving these communities will clarify the kinds of AI education, training, and advocacy needed to improve AI adoption and outcomes.3,4 Then, we and the consumers of our AI products will be better able to anticipate adoption challenges, appreciate whether the risks and rewards of the systems apply evenly across individual users and communities, recognize how previous solutions (automated or not) have become successful, and protect under-represented populations.5,6

Plan to Fail

Benjamin Franklin once said, “If you fail to plan, you are planning to fail.”1 The uncertain and the unexpected are part of reality, but resiliency comes from having many ways to prevent, moderate, or recover from mistakes or failure.2 Not all resilient methods have to be technical; they can rely on human participation and partnership. The overall amount of resiliency needed in an application increases as the AI’s success becomes more critical for the overall outcome.

Planning through prevention

If it’s possible to reduce the criticality of the AI to the mission, we should do it. When it’s not, we should follow the aircraft industry’s example and eliminate single points of failure. Boeing, for example, has “three flight computers that function independently, with each computer containing three different processors manufactured by different companies.”3 Analog backups, such as old-fashioned paper and pen, can’t be hacked or lose power.

Planning through moderation

We should try to include some checks and balances. One idea might be to simply “cap” how extreme an outcome might be; as an analogy, a video-sharing platform could limit showing videos that are categorized as “too extreme.”4 Alternatively, AI projects should make use of human judgment by adding “alerts” for both us and for users; as an example, a video-sharing platform could alert viewers that a suggested video is linked to an account that has previously uploaded more extreme content.5 These caps and alerts should correspond to the objectives and risk criteria set early in the AI development process.

Planning through recovery

We should anticipate that the AI will fail and try to envision the consequences. This means that we should consider identifying all systems that might be impacted, whether back-ups or analogs exist, if technical staff are trained to address those failures, how users are likely to respond to an AI failure, and hiring bad guys to find vulnerabilities before the technology is deployed .

We can apply other lessons

We can usually improve resiliency by treating the intended users as partners. Communicating why we made particular decisions can go a long way toward reducing misunderstandings and misaligned assumptions. Also, offering a choice to the users or individuals affected by the AI allows people to decide what’s best for their needs at the moment.

Ask for Help: Hire a Villain

While we can leave it to bad actors or luck to identify vulnerabilities in a deployed AI, or we can proactively hire a team that’s on our side to do it. Such “red teams” take the perspective of an adversary.

Red-teaming the technology

From the technology perspective, these surrogate villains can deploy automated software testing tools to find bugs and vulnerabilities. One interesting approach to meeting this shortfall is Netflix’s “Simian Army,” which intentionally introduces different types of automation failures in order to build resiliency into their architecture.1 One such tool is the “chaos monkey”,2 which randomly shuts down services or elements of code to reveal where more strengthening can be beneficial.

We can also turn to professional “white-hat hackers.” White-hat hackers are experts (often formally certified) who hack for a good cause or to aid a company, organization, or government agency without causing harm.3,4 Organizations such as Apple5 and the Department of Defense6 have hired white-hats or posted rewards for identifying and sharing vulnerabilities.

Red-teaming the people and processes

Red teams and white hats look for vulnerabilities that come from people and processes as well as the tech.7 For example, is that entry to a building unguarded? Can a person be convinced to insert a USB stick with a virus on it into a system? Can that system be tricked into giving more access than intended? Red teams and white hats will try all that and more.

Hiring a villain reduces vulnerabilities and helps us build in more technical and procedural resiliency.

Use Math to Reduce Bad Outcomes Caused by Math

First, we must accept that no data-driven solution will be perfect, and our goal shouldn’t be to achieve perfection. Instead we should try to understand and contextualize our errors.1

Looking at the data

We can apply existing statistical sampling mitigations to combat mathematical forms of bias that arise from sampling errors (which are distinct from bias caused by human influence). These mitigations include collecting larger samples and intentionally sampling from categorized populations (e.g., stratified random sampling).2 In the last few years, statistical bias toolkits3,4,5,6,7 have emerged that incorporate visualizations to help us understand our data. Specific toolkits8 have also been developed to help us understand datasets that contain associations that are human-influenced (for example, the term “female” is more closely associated with “homemaker” than with “computer programmer” in a search of Google News articles9).

Looking at the algorithms

We can also offset an AI’s tendency to amplify patterns at the model level. One set of intervention methods imposes model constraints that push predictions toward a target statistical distribution10 or uses guardrails that enforce limits to outcomes or trigger alerts for human investigation.11 Another method helps reduce runaway feedback loops (which push behavior toward increasingly specialized and extreme ends) by restricting how outputs generated from model predictions should be fed back into the algorithm12 (see more in the "Feeding the Feedback Loop" Fail). One simple diagnostic is to compare the distributions of predicted to observed outputs.13

Math alone isn't enough

Mathematical approaches can reduce the occurrence of undesired, mathematically-based outcomes. We must remember, though, that removing all mathematical error may not answer the social concerns about the AI’s impact. We must also remember that the allure of a purely technical, seemingly objective solution takes resources and attention away from the educational and sociopolitical approaches that are necessary to address the more fundamental challenges behind complex issues.14

III. Calibrate Our Trust In the AI and the Data

Make Our Assumptions Explicit

Let’s start with an example: say we collect images of irises that grow in North America, and we train an AI to classify three different types of irises. The algorithm is pretty successful, and we want to share it with the world. If some potential users live in Europe and wants to use the algorithm, it’s important for them to know that the accuracy would diminish for them because European irises look different, or that we only collected images in the daytime, or that we could only find a small sample for one type of iris. These users need to know the assumptions and tradeoffs behind the chosen training data, model parameters, and environment for that algorithm. Otherwise, they could be using the AI incorrectly or for purposes it was not intended to fulfill, but would trust in the outcomes nonetheless.

Generalizing from this example, many groups of people benefit from understanding the original developers’ assumptions:

• Those who acquire or want to repurpose the AI systems need to know where the data comes from and what its characteristics are in order to make sure it aligns with their purposes.
• End users and consumers need to know how to appropriately interact with the AI so that they encounter fewer surprises and can more accurately weigh the risks of integrating the technology into their processes.
• AI policymakers and legislators need to know the original intended and unintended uses for the AI in order to apply, update, monitor, and govern it in an informed way.
• Objective third parties need to assess if the data and the algorithm’s outcomes are mathematically and socially representative of the historical norms established in the domain where the AI is being deployed.1

Once they recognize the value of conveying these assumptions, organizations can take two steps to promote this practice

1. Have the developers fill out standardized templates that capture assumptions and decisions.

The documentation process should... prompt us to bring in end users and affected communities to ensure they have the information they need, and... have the opportunity to offer suggestions. At the same time, the process should prompt analysts or decision makers... to capture how the input from an algorithm affected their overall assessment of a problem.

No one knows better about the intended and unintended uses for their data and tools than the original developers. Two sets of researchers from industry and academia have created templates that help draw out the developers’ intents, assumptions, and discussions. The first, datasheets for datasets, documents the dataset’s “[purpose], composition, collection process, recommended uses,” decisions, and justifications.2 Data choice and relevance are particularly critical to reduce bias and avoid placing miscalibrated trust in AIs.3

Serving as a complementary process, model cards for model reporting “clarify the intended use cases of machine learning models… provide benchmarked evaluation in a variety of conditions… and disclose the context in which models are intended to be used.”4 Understanding the intended context and use for the models is crucial to avoiding unwelcome surprises once the AI is deployed (in this case for machine learning, one type of AI). Importantly, these two documents highlight both what was intended and specifically not intended.

Adopting the two templates as standard practice will go a long way toward helping us achieve transparency, explainability, and accountability in the AI we develop. The Partnership on AI – an organization of AI experts and stakeholders looking to formulate best practices on AI5 – posted several examples as part of their ABOUT ML (Annotation and Benchmarking on Understanding and Transparency of Machine learning Lifecycles) initiative, aimed at experimenting with and scaling machine learning documentation efforts.6

2. Structure documentation processes in a way that facilitates proactive and ongoing outreach.

The documentation process should require recurring conversations with a diverse team.

We as developers are best positioned to articulate the strengths and weaknesses of our systems, but other perspectives are needed to highlight the risks and design tradeoffs that we may not have considered. For example, end-users, lawyers, and policymakers (among others) may all have different questions that help us make informed decisions about the AI’s appropriate uses, and they offer different considerations for mitigating potential risks. Even then, there are limitations to that group’s collective knowledge. They might not catch all the biases or shortcomings in a first go-around, but the next user group would benefit greatly from the lessons learned in previous versions. Knowing what’s been considered earlier helps new development teams integrate the different perspectives that were already offered and avoid repeating the same mistakes. Therefore, the documentation process should require recurring conversations with a diverse team.

Another key aspect of the documentation process is that we should be proactive in communicating bias and other limitations of systems to potential users, and not wait for a periodic review. Conveying design choices can be fundamentally transformative to the user’s assessments of model appropriateness and trustworthiness. The documentation process should involve asking questions that prompt us to bring in end users and affected communities to ensure they have the information they need, and have the opportunity to offer suggestions early enough that we can incorporate their input in the product. At the same time, the process should prompt analysts or decision makers (if internal to the organization) to capture how the input from an algorithm affected their overall assessment of a problem. Making informed decisions is a joint responsibility.

Documentation is just part of the development process

How each organization will implement these processes may differ. It might be easiest to expand on existing steps like user interviews, requirements generation, project management checks, quality control reviews, and other steps in a product’s lifecycle. The organization may need to develop new processes specifically for documentation and explanations. Either way, thinking about these goals in advance means that we can make transparency part of the development process from the beginning of a project, and are therefore more likely to ensure it is done well.

Try Human-AI Couples Counseling

No AI, robot, or person works completely alone. If you’ve ever become frustrated with automation, you aren’t alone – senior researchers from the Florida Institute for Human and Machine Cognition describe the feeling: “there’s nothing worse than a so-called smart machine that can’t tell you what it’s doing, why it’s doing something, or when it will finish. Even more frustrating—or dangerous— is a machine that’s incapable of responding to human direction when something (inevitably) goes wrong.”1 And although an AI may not get frustrated, it can require the same things its human partners do: explanations from and an ability to influence its partners.

Partnership is not simply a game of tag – passing a task off and saying “Good luck.” Human-AI partnership means two things: communicating what each party needs or expects from all its partners (whether human or AI), and designing a system that reinforces collaboration.

The solution is not for us to build systems that people trust completely, or for users only to accept systems that never err. Instead, lessons point to the importance of forming good partnerships based on evidence and perception.

The first step is talking it out

Better AI-to-human (AI → H) communication gives humans a chance to calibrate their confidence and trust in the AI. This allows humans to trust that the AI can complete a task independently, and to understand why the system made its decisions and what the outcomes were. On the other hand, better H → AI communication gives the AI a better understanding of the users’ priorities and needs, so it can adjust to those preferences. Overall, improved H ↔ AI communication makes it clearer when tradeoffs will occur, who (or what) is responsible for which part of the task, how humans and AI can best contribute on interdependent tasks, and how behaviors and preferences change over time.2,3

The second step is thinking “combine and succeed” rather than “divide and conquer”4

Each teammate, whether a human or an AI, must be able to observe, predict, and direct the state and actions of others on the team.5,6

In other words, both the human and the algorithmic partners have to maintain common ground, act in expected ways, and change behavior based on the partner’s input. This result can manifest itself in the forms of explanations, signals, requests for attention, and declarations of current action.7,8

Building trust is about forming good partnerships

AI adopters often ask about ways to increase trust in the AI. The solution is not for us to build systems that people trust completely, or for users only to accept systems that never err. Instead, lessons point to the importance of forming good partnerships based on evidence and perception. Good partnerships help humans understand the AI’s abilities and intents, believe that the AI will work as anticipated, and rely on the AI to the appropriate degree. Then stakeholders can calibrate their trust and weigh the potential consequences of the AI’s decisions before granting appropriate authorities to the AI.9

Offer the User Choices

During the design process, we make dozens of choices, assumptions, simplifications, and trade-offs (CAST) that affect the outcome of the AI system. In order to better understand the application domain, we invite stakeholders to share their preferences, desired outcomes, and how they would use the system. But at the end of the day, the CASTs remain with us since we’re the technical experts. One way to reduce this knowledge gap is for us to document our decisions. But are there situations where user experience, or evolving user goals or behavior, make it more appropriate for the user to make decisions? What might it look like if, after deployment, we “extended” users’ involvement by empowering them to weigh in on some of the choices, when its appropriate to do so?

Sometimes, the user knows best

One idea for giving the user more appropriate agency is to present the user with options that juxtapose how specific developer decisions influence the AI’s objectives. For example, when debating between different instantiations of fairness (see more in the "You Told Me to Do This" Fail), instead of leaving that decision to the developer, we could add a “dial” that would let the user switch between definitions. In this way they could select the approach that better aligns to their principles, or they can view a range of outcomes and form a more complete picture of the solutions space. When the dial is accompanied by explanations that include context around the developer’s CASTs (perhaps an overview of what the algorithm is optimizing, properties of the data, and how the algorithm defines success), this implementation could improve outcomes by appropriately shifting decisions to the stakeholder that knows the situation or environment best.1

Provide different degrees of explanations, depending on the user need

Explanations can contain different levels of detail: users may accept an AI’s decision at face value, want confidence scores of those decisions, want confidence scores and descriptions of how those scores are generated,2 or may even want examples of how the algorithm reached a decision. Certain algorithms can provide text and visual examples of what training data was most helpful and most misleading for arriving at the correct solution (for example, “this tumor is classified as malignant because to the model it looks most like these other tumors, and it looks least like these benign conditions”).3 With this approach the users can select how much they need to know about the AI in order to make an informed decision about applying or not applying its outcomes.

More research is needed

More research is needed into how empowering users with choice would affect the accuracy and desirability of outcomes, and more research is needed into how to best capture and present the developer’s CASTs in such a way that is meaningful for the user. On the one hand, the AI developers comprehend the complexities of AI design and the ramification of design decisions. Giving users seeming control over aspects they don't understand has the potential to give the illusions of clarity and informed control, cause additional automation bias, or simply allow the user to select an option that gives them the answer they want.

Yet, the decisions of the developers should not substitute for the range of outcomes and intents that the user might want. More research could suggest ways to give users agency relative to their technical understanding of an AI, and appropriate to how the AI is applied in their domain. At best, this approach can reemphasize the value of algorithms offering competing perspectives, or evidence and counterevidence, which can elicit more diverse ideas and open dialogue – thus reinforcing principles that are foundational to the health of democracies.4

Promote Better Adoption through Gameplay

There’s a big difference between imagining how an AI works and interacting with it in the real world. As a way to bridge that gap, we could invite different users to play with the AI in a more controlled environment. Gameplay lets different stakeholders explore how a technology may affect their lives, their work, or their attention. It allows everyone to move from “knowing” to “feeling” and forming mental models of how the AI works.1 Gameplay is especially important for stakeholders to better understand AI technologies, which learn and adapt the more they interact.2

Discovering misalignment early is better than waiting until after deployment, when the AI may have had an adverse impact

Gameplay is vital for bringing to light some of the differences between our assumptions and the behavior of stakeholders. These differences can manifest themselves in several ways:3

• Various groups may interpret outcomes, definitions, and behaviors differently. For example, some cultures view increased personalization as a global good, while other cultures focus on communal outcomes.
• Various groups value and endorse different outcomes. For example, more data leads to better quality outcomes, but often comes at the cost of individual privacy and autonomy.
• Individuals change the relative value of particular outcomes depending on the context. In some contexts (e.g., AI medical diagnoses) user groups prefer accuracy over explanations, but prefer the reverse for AI-enabled job recruiting.4

Gameplay approaches depend on the system's maturity

If the technology is mature enough for us to create a working prototype, gameplay can take the form of user evaluations, table-top exercises (TTXs), or experiments. One example is the Defense Advanced Research Projects Agency’s (DARPA) engagement with Marines while developing the Squad X program. DARPA paired AI-enabled ground and air vehicles with a squad of Marines, then gave the teams realistic operational tasks. Through gameplay, the AI-enabled vehicles progressed from providing reconnaissance – a traditional role for unmanned vehicles – to becoming valued members of the squad, protected by and enabling the Marines to achieve their objectives more efficiently.5,6

If the technology is still in a conceptual phase – perhaps just a “what if” – we can try simulation techniques or traditional wargaming. Simulation helps to demonstrate and develop how individuals will use the technology and informs what design changes will make the product better. Alternatively, traditional wargaming plays out how conceptual technologies can be integrated into tactics, decision making, and future training.7,8

Exploring the discrepancies between expectations and actual AI behavior as well as the differences in how stakeholders interact with the AI, is a powerful way to reach technical, social, and policy resolutions in specific situations. Discovering misalignment early is better than waiting until after deployment, when the AI may have had an adverse impact.9

IV. Broaden the Ways to Assess AI’s Impacts

Monitor the AI’s Impact and Establish Layers of Accountability

It is the impact of the AI on people’s lives that matters most

Modern-day engineers who design AI systems have the best of intentions. While we want our systems to benefit users, communities, and society in general, the reality is that after we deploy an AI, something – the data, the environment, how users interact with the AI – will change, and the algorithm will work in unexpected ways. When weighing all these potential outcomes, it is the impact of the AI on people’s lives that matters most. Therefore, we need a strategy for monitoring the AI and assigning parties to implement changes to the AI based on that impact. When individual and organizational accountability is tied to that strategy, we get more responsible outcomes.

Approaches will require continuous monitoring and ongoing investments. To act quickly against unanticipated outcomes, organizations should take the following actions:

1. Calculate baseline criteria for performance and risk

At the beginning of the project, we should establish baseline performance criteria for acceptable functioning of the AI. As one AI writer/practitioner described, just like a driving a new car off the lot, “the moment you put a model in production, it starts degrading.”1 If the AI “drifts” enough from its baseline, we may have to retrain or even scrap the model. Baseline performance criteria should be both mathematical and contextual, and criteria should include the perspectives of all affected stakeholders.

In parallel with performance criteria, risk assessment criteria should guide decisions about the AI’s suitability to a given application domain or intended use. Prior to deploying the system, we should determine the threshold of clarity that different stakeholders require, and how well the AI meets those requirements. Organizational guidance should be clear for higher stakes cases, when legality, ethics, or potential impact areas of concern.

2. Regularly monitor the AI’s impact and require prompt fixes

As part of a good project management plan, we should set up continuous, automated monitoring as well as a regular schedule for human review of a model’s behavior. We should check that the algorithm’s outputs are meeting the baseline criteria.2 This will not only help refine the model, but also help us act promptly as harms or biases emerge.

Because changes will have to be made to the model, the original development team should remain involved in the project after the AI is deployed.3,4 As the number of AI projects increases, that original development can train new maintainers.

3. Create a team that handles feedback from people impacted by the AI, including users

Bias, discrimination, and exclusion can occur without our even knowing it. Therefore, we should make clear and publicize how those affected by the AI can alert this feedback team. The organization can also create guidelines on how and when to act on this feedback.

In addition, this feedback team can be proactive. Some AI relies heavily on data; this team should broadcast how an individual’s data is used and implement processes for discarding old data.5 With its GPT-2 algorithm, Google set up an email address and guided other researchers looking to build off Google’s work6 – a particularly important step given the potential harmful outcomes of the application (see example 3 in the "You Told Me to Do This" Fail).

4. Experiment with different accountability methods

AI is a rapidly evolving technical field, and the interaction between AI and other applications creates a complex ecosystem. Therefore, accountability that works well today may not be equally effective as future technologies change that ecosystem. And as an organization’s structure and culture evolves, so too may its accountability efficacy.7

One example experiment comes from Microsoft, which established an AI, Ethics and Effects in Engineering and Research (AETHER) Committee in 2018. Wary of the suspicion that such a move would be viewed primarily as an attempt to improve public relations, Microsoft required direct participation in the committee by senior leadership. Microsoft also asked employees with different backgrounds to provide recommendations to senior leadership on challenging and sensitive AI outcomes and to help develop implementable policy and governance structures in conjunction with the company’s legal team. The committee also set up an “Ask AETHER” phone line for employees to raise concerns.8

The impacts from experiments like these are still being assessed, but their existence signals a growing willingness by organizations to implement oversight and accountability mechanisms.

AI has real consequences

AI has real consequences and is certain to continue to produce unintended outcomes. That is why we must explore all the possible perspectives to address this accountability challenge and to do our best to position our organizations to be proactive against, and responsive to, undesirable outcomes.

Envision Safeguards for AI Advocates

If ethical outcomes are part of our organization’s values, we need to devote resources and establish accountability among ourselves and our teams to ensure those values are upheld, and to protect those who fight to uphold those values.

Employees in AI organizations, both commercial and government, are organizing and protesting in response to perceived harmful outcomes arising from the products and organizational decisions of their leadership. Through walkouts,1 advocacy,2 and expressions of general concern3 these employees are representing and reinforcing the ethical principles that their organizations proclaim. When these employees are punished or fired,4,5 sometimes unlawfully,6 they need stronger safeguards and top cover.

What might those safeguards look like?

The AI Now Institute at New York University (a research institute dedicated to understanding the social implications of AI technologies) lays out specific approaches that organizations should adopt to avoid social, economic, and legal penalties, including “clear policies accommodating and protecting conscientious objectors, ensuring workers the right to know what they are working on, and the ability to abstain from such work without retaliation or retribution. Workers raising ethical concerns must also be protected, as should whistleblowing in the public interest.”7 Support for workers would also include assigning responsible parties and processes to administer changes at the deploying organization, and making clear how those affected by the AI can alert those parties.8

Require Objective, Third-party Verification and Validation

Because algorithms are making decisions that affect the livelihoods, finances, health, and the civil liberties of entire communities, the government has to protect the public, even if doing so may be initially detrimental to industry profit and growth. By incentivizing participation, the government could offset initial increased costs for AI in order to help promote the emergence of a new marketplace that responds to a demand signal for ethical AI.

What is objective, third-party verification and validation?

Objective, third-party verification and validation (O3VV) would allow independent parties to scrutinize an algorithm’s outcomes, both technically and in ways that incorporate the social and historical norms established in the relevant domain. For meaningful oversight, O3VV representatives need to understand the entire lifecycle of the AI-enabled system: from evaluating the origins and relevance of the training datasets, to analyzing the model’s goals and how it measures success, to documenting the intended and unintended deployment environments, to considering how other people and algorithms use and depend on the system after each update.1,2

Think of O3VV like an Energy Star seal – the voluntary program established by the Environmental Protection Agency that allows consumers to choose products that prioritize energy efficiency.3 Or think of “green energy” companies that respond to consumer preference for sustainable businesses and products, and enjoy more profits at the same time.4 Both models center on a recognized, consensual set of criteria, as well as an (ideally, independent) evaluative body that confirms compliance with the standard. ForHumanity, a non-profit group that advocates for increased human rights awareness and protection as AI spreads, describes what such a program might look like with its SAFEAI Seal.5

Because algorithms are making decisions that affect the livelihoods, finances, health, and the civil liberties of entire communities, the government has to protect the public, even if doing so may be initially detrimental to industry profit and growth

Making O3VV real

Following these examples, evaluators should come from a range of academic backgrounds and represent all the communities affected by the AI. O3VVs could take on consumer protection roles, placing emphasis on how the decisions affect real people’s lives6,7 and promoting truth in advertising requirements for AI products and services.8 O3VV agencies could take the form of government auditing programs, Federally Funded Research and Development Centers (FFRDCs), certified private companies, and a consensually developed “seal” program.

In order for O3VV to become established practice, the government needs to incentivize participation. Currently, there are no standards for using AI that have been certified by O3VV, nor are there incentives for companies to go through a certification process, or for professionals and academics to contribute to the process.9 One approach calls for a licensing program for O3VV professionals, and another calls for increasing monetary incentives for deploying certified systems.10 Another idea is to allow FFRDCs, which by law are not allowed to compete with industry and which work only in the public interest, access to proprietary AI datasets and model information in order to perform independent verification and validation. Especially if the government is a consumer, it can require that vendors adhere to these steps before the government will purchase their products.11,12

Entrust Sector-specific Agencies to Establish AI Standards for Their Domains

Sector-specific agencies already have the historical and legislative perspectives needed to understand how technology affects the domain under their responsibility; now, each of those agencies should be empowered to expand its oversight and auditing powers to a new technology

AI is increasingly integrated into more domains, including national defense, healthcare, education, criminal justice, and others. Establishing a global approach to AI governance is challenging because the legislative and social histories and policies in each domain differ drastically.1 New technologies will be more broadly adopted if they follow established practices, expectations, and authorities in a given domain.

Two Examples Illustrate Different Standards in Different Domains

First, a children’s hospital in Philadelphia deployed a black box AI that looks for a rare but serious infection (sepsis). The AI used patients’ electronic health records and vital-sign readings to predict which fevers could lead to an infection. The AI identified significantly more life-threatening cases than did doctors alone (albeit with many false alarms), but what made the story so compelling and the application so successful was that doctors could examine the identified patients as well as initiate their own assessments without alerts from the AI. In other words, doctors could use the AI’s queues while still employing their own judgment, decision making, and authority to achieve improved outcomes.2,3

Second, as introduced earlier, state and local jurisdictions in the US have deployed COMPAS, a black box tool that assesses the risk of prison inmate recidivism (repeating or returning to criminal behavior). COMPAS uses a combination of personal and demographic factors to predict the likelihood an inmate would commit another crime. The tool produced controversial results: the number of white inmates with a certain score re-offended at the same rates as black inmates with that score, but among defendants who did not re-offend, black inmates were twice as likely as white inmates to be classified as presenting medium or high risk. As in the hospital example, judges could ignore COMPAS’s input or refer to it, but final assessment and responsibility lay with the judge.4,5,6

In each of these cases, the expert could discount or act on the AI’s recommendation. The difference between these two examples lies in the historical and cultural norms, rules, and expectations that exist in the two domains. The public might be less at ease with using AI in the judicial context for any number of domain-specific reasons: because judges rule in “case of first impression” when a higher court has not ruled on a similar case before,7 or because the court uses twelve jurors rather than a single judge, a practice established as representative of a good cross-section of perspectives.8 In contrast, the public might be more at ease with AI offering predictions on medical diagnoses because doctors routinely use “evidence-based medicine”9 to integrate their own clinical experience with the latest research, established guidelines, and other clinicians’ perspectives, of which the algorithm could be considered a part. Doctors also take the Hippocratic oath, pledging to work for the benefit of the sick,10 whereas judges must weigh both individual and collective good in their decisions.

In short, different sectors have different expectations; therefore, institutional expertise should be central to determining the benefits and risks of incorporating each type of AI system.

Past Precedent and Future Principles

Sector-specific agencies already have the historical and legislative perspectives needed to understand how technology affects the domain under their responsibility; now, each of those agencies should be empowered to expand its oversight and auditing powers to a new technology. In early 2020, The White House called for the same process in its draft principles for guiding federal regulatory and non-regulatory approaches to AI: “Sector-specific policy guidance or frameworks. Agencies should consider using any existing statutory authority to issue non-regulatory policy statements, guidance, or testing and deployment frameworks, as a means of encouraging AI innovation in that sector.”11 It is incumbent on individual agencies to permit, regulate, temper, and even ban12 AI-enabled systems as determined by the experts and established practices in each domain.

The French Data Protection Authority (the government agency responsible for the protection of personal data)13 provides an example of two founding principles for AI standards:

• “A principle of fairness applied to all sorts of algorithms, which takes into account not only their personal outcomes but their collective ones as well. In other words, an algorithm… should be fair towards its users, not only as consumers but also as citizens, or even as communities or as an entire society.
• A principle of continued attention and vigilance: its point is to organize the ongoing state of alert that our societies need to adopt as regards the complex and changing socio-technical objects that algorithmic systems represent. It applies to every single stakeholder (designers, businesses, end-users) involved in ‘algorithmic chains.’”

Government legislation on AI standards means enacting a legal framework that ensures that AI-powered technologies are well researched, the AI’s impacts are tested and understood, and the AI is developed with the goal of helping humanity.14