Article body

Introduction

The risk of discriminatory bias is a central concern when it comes to the use of artificial intelligence (AI) technologies for automated decision-making (ADM).[1] Identification and mitigation of such bias is an important preoccupation of emerging laws and policies. Currently, public-sector automated decision systems (ADS) operate in lower risk contexts than the criminal justice system, although the use of such tools is evolving.[2] In the private sector, ADS are already deployed in higher impact contexts such as the selection of tenants for apartments,[3] the determination of creditworthiness,[4] and in hiring and performance evaluation.[5] Generative AI systems such as ChatGPT can also be used to support ADM in different ways, including in the preparation of briefing materials, translation, and drafting decisions.[6]

This paper explores discriminatory bias in ADM using the series of court decisions in R. v. R.D.S.[7] (RDS) (culminating in a 1997 decision of the Supreme Court of Canada) to illustrate some of the potential frailties in approaches to this issue. It is important to note at the outset that RDS addressed the issue of ‘reasonable apprehension of bias’, which differs significantly from the human-rights-based concept of discriminatory bias. Nevertheless, the case is important because in it, the concept of impartiality that underlies the doctrine of reasonable apprehension of bias becomes intertwined with the notion of discriminatory bias in complex and interesting ways. In RDS, the alleged apprehension of bias is tied to a Black woman judge’s perception of the credibility of two witnesses – one White and one Black. Credibility – something typically stripped from the targets of discrimination – is left to be determined by a decision-maker who is in turn challenged for bringing a racialized (i.e., non-White) perspective to the task. This complicated and messy case challenges risk-mitigation approaches to AI bias in which we identify risks, develop strategies to mitigate them, and monitor outcomes.[8] Risk-based approaches tend to assume that there is a social consensus about what bias is and how it is manifested. They also tend to lead us towards technological solutions. RDS teaches us that understanding, identifying, and addressing bias may be much messier.[9]

This paper begins with a brief overview of discriminatory bias in AI systems. Part 2 provides a summary of the dispute at the heart of RDS. Part 3 teases out four themes emanating from RDS that are relevant to the AI context: (1) the tension between facts and opinion, (2) transparency and explainability, (3) the issue of biased input and biased output and (4) the role of the human-in-the-loop. The paper concludes by considering that a statistical and technological approach to identifying and mitigating bias in ADM may unduly narrow the focus, and argues for a more robust approach to addressing bias in AI.

I. Bias and AI

It is well understood that AI technologies raise problems of harmful or discriminatory bias that must be addressed.[10] The US National Institute of Standards and Technology (NIST) Framework for AI Risk Management identifies “harmful bias” as an issue of fairness, linking it to concerns for equality and equity.[11] It identifies three broad categories of bias in AI: “systemic, computational and statistical, and human-cognitive,”[12] all of which can be present without any intention to discriminate. Systemic bias can be found “in AI datasets, the organizational norms, practices, and processes across the AI lifecycle, and the broader society that uses AI systems.”[13] Statistical or computational bias is “anything that leads to a systematic difference between the true parameters of a population and the statistics used to estimate those parameters.”[14] Errors that create statistical bias can arise from the collection of the data, its classification, the omission of certain variables, or choices made in study design or in the weighting of different variables. Human cognitive biases, according to the NIST Framework:

relate to how an individual or group perceives AI system information to make a decision or fill in missing information, or how humans think about purposes and functions of an AI system. Human-cognitive biases are omnipresent in decision-making processes across the AI lifecycle and system use, including the design, implementation, operation, and maintenance of AI.[15]

Both systemic and human cognitive bias are more complex than statistical bias, and neither can be eliminated through the correction of datasets or algorithms. For example, in considering systemic bias, the NIST Framework notes that “systems in which predictions are somewhat balanced across demographic groups may still be inaccessible to individuals with disabilities or affected by the digital divide or may exacerbate existing disparities or systemic biases.”[16]

A recent initiative between the EU and the US to harmonize AI terminology addresses both harmful bias and discrimination, and distinguishes between the two:

Harmful AI bias describes systematic and repeatable errors in AI systems that create unfair outcomes, such as placing privileged groups at systematic advantage and unprivileged groups at systematic disadvantage. Different types of bias can emerge and interact due to many factors, including but not limited to, human or system decisions and processes across the AI lifecycle. Bias can be present in AI systems resulting from pre-existing cultural, social, or institutional expectations; because of technical limitations of their design; by being used in unanticipated contexts; or by non-representative design specifications.[17]

The sources of bias are disparate, making it challenging to address. The EU-US definitions identify discrimination as a subset of harmful bias. Discrimination is defined as a form of unequal treatment that “can be a result of societal, institutional and implicitly held individual biases or attitudes that get captured in processes across the AI lifecycle, including by AI actors and organisations, or represented in the data underlying AI systems.”[18] The definition of discrimination goes on to note that discriminatory bias

can also emerge due to technical limitations in hardware or software, or the use of an AI system that, due to its context of application, does not treat all groups equally. Discriminatory biases can also emerge in the very context in which the AI system is used. As many forms of biases are systemic and implicit, they are not easily controlled or mitigated and require specific governance and other similar approaches.[19]

The essential difference between harmful bias and discrimination appears to be agency. Discrimination comes from human biases that infect AI processes or design, whereas harmful bias flows from flawed data or design issues, some of which may be impacted by the social context in which they are developed (which in turn suggests at least a degree of agency). Harmful bias can in part be attributable to discriminatory bias, but it can also result from the interaction of a variety of different factors within a system which then reproduces this bias or manifests it in new ways.

The NIST AI RMF and the EU-US definitions explore a complex understanding of bias in AI and its relationship to harm and discrimination – which are not interchangeable terms. Harmful bias does not necessarily fit within the grounds of discrimination typically identified in human rights legislation. For example, a system might embed bias based on whether one is resident in a rural or urban location, creating unfair outcomes that would not be recognized as discrimination under most Canadian human rights statutes. Nevertheless, Canada’s draft Artificial Intelligence and Data Act appears to conflate the two terms when it establishes obligations to identify and mitigate the risk of “biased output” from an AI system, defining “biased output” as:

content that is generated, or a decision, recommendation or prediction that is made, by an artificial intelligence system and that adversely differentiates, directly or indirectly and without justification, in relation to an individual on one or more of the prohibited grounds of discrimination set out in section 3 of the Canadian Human Rights Act, or on a combination of such prohibited grounds [. . . ].[20]

By pegging bias to specific categories of discrimination in human rights legislation, this approach narrows the range of bias addressed by the law. By focusing on biased output, it also narrows the focus of the bias inquiry and slants towards characterizations of bias as tied to machines and not the broader socio-technical context in which they are designed, deployed, and maintained.

A further concept relevant to this paper is the reasonable apprehension of bias, which is linked to fairness in administrative and judicial decision-making and addresses the decision maker’s impartiality. The test is framed in terms of whether a reasonable person would consider that the decision maker was able to act fairly.[21] The apprehension of bias need not be with respect to prohibited grounds of discrimination, it could be founded on pecuniary interests or personal relationships.[22] Further, it is unnecessary to demonstrate actual bias – just a reasonable apprehension of bias; the perception of fairness is important to the reputation of the justice system. In the case of ADM, the concept of reasonable apprehension of bias can align somewhat with automation bias. For example, a reasonable apprehension of bias might arise when a human regularly and unreflexively accepts the recommendations of a system designed to aid in decision-making.[23]

Clearly, the term ‘bias’ has different meanings in different contexts. There are also multiple factors that can contribute to biased outcomes in AI, and they are not limited to data and algorithms. Although RDS is a case about the reasonable apprehension of bias, it illustrates how the issue of reasonable apprehension of bias can become entangled in a broader discussion of discrimination. This is because specifically and pointedly, the case addresses whether a non-majoritarian understanding of the context in which a dispute arose reflects impartiality. In this sense, it resonates with the messiness of bias and discrimination in AI with which both the NIST AI RMF and the EU-US definitions struggle. The different decisions in the case are discussed in the next section.

II. R. v. R.D.S.

On November 10th, 1993, R.D.S., a 15-year-old youth from Nova Scotia’s Black community had a run-in with a White police officer on the streets of Halifax. The officer was arresting another youth in relation to a motor vehicle theft. The court heard two versions of the event. According to the police officer, he had detained the young man and was waiting for backup when R.D.S. cut across the road and ran his bicycle up against the officer’s legs. R.D.S. yelled at the officer and tried to push him away from the youth he was arresting. These facts led to R.D.S.’s arrest, who was charged with assaulting a police officer, assaulting a police officer with intent to prevent the lawful arrest of another person, and resisting arrest.[24]

According to R.D.S., he was cycling from his grandmother’s house to his own when he saw a crowd forming around a police car. He rode up to the scene and recognized the detained youth. He asked him what had happened and told him that he would call the youth’s mother. The police officer told R.D.S. to “shut up” or he would also be arrested. R.D.S. asked the detained youth again if he wanted him to call his mother, and the police officer put the former in a chokehold and arrested him. R.D.S. denied running into the officer with his bicycle.[25]

At trial, the officer and R.D.S. were the only witnesses. Judge Sparks found that the Crown had not met its burden of proving guilt beyond a reasonable doubt. All justices sitting in review of this case at all levels of court would have found no reasonable apprehension of bias had the decision ended there. However, Judge Sparks went on to say in oral reasons:

The Crown says, well, why would the officer say that events occurred the way in which he has relayed them to the Court this morning. I’m not saying that the constable has misled this Court, although police officers have been known to do that in the past. And I’m not saying that the officer overreacted but certainly police officers do overreact, particularly when they’re dealing with nonwhite groups. That, to me, indicates a state of mind right there that is questionable.

I believe that probably the situation in this particular case is the case of a young police officer who overreacted. And I do accept the evidence of R.D.S. that he was told to shut up or he would be under arrest. That seems to be in keeping with the prevalent attitude of the day.

At any rate, based upon my comments and based upon all of the evidence before the Court I have no other choice but to acquit.[26]

The Crown appealed the acquittal to the Nova Scotia Supreme Court, arguing that Judge Sparks’ decision was based on considerations and findings of credibility unsupported by evidence.[27]. Chief Justice Constance Glube agreed, noting that “judges must be extremely careful to avoid expressing views which do not form part of the evidence.”[28] Going further, the Chief Justice applied the objective test of reasonable apprehension of bias, “whether a reasonable right-minded person with knowledge of all the facts would conclude that the judge’s impartiality might reasonably be questioned.”[29] She concluded that “in spite of the thorough review of the facts and the finding on credibility, the two paragraphs at the end of the decision lead to the conclusion that a reasonable apprehension of bias exists.”[30]

The majority of a three-judge panel of the Nova Scotia Court of Appeal confirmed this decision. They found it “apparent” that Judge Sparks based her decision “at least in part, on her general comments with respect to the police.”[31] In response to arguments by counsel for R.D.S. that the comments “merely reflect an unfortunate social reality,”[32] the majority noted that the real issue was whether “the Youth Court Judge, considered matters not in evidence in arriving at her critical findings of credibility, and hence, acquittal.”[33] The majority criticized her “unfortunate use of these generalizations,” and found a reasonable apprehension of bias.[34]

Justice Freeman, in his dissent, noted that assessment of credibility is “a notoriously difficult and inexact exercise in adjudication in which the judge’s whole background experience plays a role in the assessment of demeanour and other intangibles.”[35] He highlighted the “racially charged” nature of the case, stating that “Judge Sparks was under a duty to be sensitive to the nuances and implications, and to rely on her own common sense which is necessarily informed by her own experience and understanding.”[36] Rather than finding that Judge Sparks’ comments were addressed to evidence not before the court, he treated the matter as one of determining credibility, which “draws upon all of the judge’s wisdom and experience.”[37] He observed that while Judge Sparks’ comments could have been clearer, they did not raise a reasonable apprehension of bias.

On further appeal, a majority of the Supreme Court of Canada ruled that Judge Sparks’ comments did not raise a reasonable apprehension of bias. Two of the six majority judges did so with reservations. Justice Cory, writing for himself and for Justice Iacobucci found that a judge is “obviously permitted to use common sense and wisdom gained from personal experience in observing and judging the trustworthiness of a particular witness on the basis of factors such as testimony and demeanour.” However, a judge “must avoid judging the credibility of the witness on the basis of generalizations or upon matters that were not in evidence.”[38] Although he found no reasonable apprehension of bias, Justice Cory characterized Judge Sparks’ remarks as “worrisome”[39] and “troubling.”[40]

Justices L’Heureux-Dubé and McLachlin wrote a separate opinion, with which Justices LaForest and Gonthier concurred. They agreed that there was no reasonable apprehension of bias, but disagreed with the conclusion that the remarks were inappropriate. They maintained the importance of judges bringing their experience to their role, and emphasized that while impartiality was important, judges were not to act as “neutral ciphers.”[41] They also stressed the importance of context to judicial decision-making. They noted that systemic racism was a reality in Nova Scotia, stating: “[t]he reasonable person is cognizant of the racial dynamics in the local community, and, as a member of the Canadian community, is supportive of the principles of equality.”[42] Further, they observed that an awareness of context is not a lack of neutrality; rather, it is “consistent with the highest tradition of judicial impartiality.”[43] On reviewing Judge Sparks’ comments, they found nothing to indicate pre-judgment. Rather, they ascertained that Judge Sparks’ comments showed she had “approached the case with an open mind, used her experience and knowledge of the community to achieve an understanding of the reality of the case, and applied the fundamental principle of proof beyond a reasonable doubt.”[44]

Three dissenting justices, under Justice Major’s pen, found that the facts raised a reasonable apprehension of bias. Justice Major stated: “A fair trial is one that is based on the law, the outcome of which is determined by evidence, free of bias, real or apprehended.”[45] He concluded that the decision was not based on the evidence before the court, but on “something else.”[46] He went so far as to state that Judge Sparks had stereotyped all police officers as liars and racists, declaring: “It would be stereotypical reasoning to conclude that, since society is racist, and in effect, tells minorities to ‘shut up,’ we should infer that this police officer told this appellant minority youth to ‘shut up’.”[47] It was an error in law for Judge Sparks “to infer that based on her general view of the police or society,”[48] the police officer’s actions and testimony were informed by racism. According to Justice Major, “[l]ife experience is not a substitute for evidence.”[49]

Although RDS was framed as an issue of reasonable apprehension of bias or judicial impartiality, the impartiality issue really turned on whether a racialized judge could publicly acknowledge the perspective that informed her assessment of the law and facts. Since her perspective was non-majoritarian, many judges involved did not consider it as “neutral” or impartial. This inability to distinguish between different lived experiences and bias—or the tendency to characterize non-mainstream perceptions as biased —raises important questions about how bias will be recognized and identified in the AI context, and by whom.

III. Themes from R. v. R.D.S.

This section explores four ways in which RDS is important to our understanding of discriminatory bias in automated decision making. It considers the tension between fact and opinion, issues of transparency and explainability, biased input and output, and the human-in-the-loop.

A. Fact is a matter of opinion?

A key issue in RDS is how to distinguish between “objective” fact and evidence on one hand, and “subjective” stereotype or opinion on the other. One of the asserted virtues of AI is its elimination of subjective opinion and focuses on objective data for decision-making.[50] Yet, although there is a tendency in some quarters to treat data as a kind of absolute truth,[51] critical data scholars remind us that data are not neutral. For example, Rob Kitchin describes data as “capta,” meaning “those units of data that have been selected and harvested from the sum of all potential data.” [52] Implicit are the choices that went into the decision to capture particular data about a specific subject matter.

When it comes to data about some groups or communities, there may be further issues: an absence of some key data and a lack of involvement of the community in how data are collected or used. The lack of adequate data about racialized communities, for example, is such a problem that Ontario and British Columbia have passed laws seeking to capture more data about these communities while ensuring that they will not be used in harmful ways.[53] Invisibility in data leads to poor outcomes in a data-driven society; yet visibility without control or input is dangerous.[54]

An important issue in RDS is what counts as fact in the first place. There are clear differences between the appellate justices as to whether Judge Sparks made findings of fact unsupported by evidence, or findings of credibility based on life experience that led to conclusions about what was or was not proven as fact (i.e., what happened). For the judges who found a reasonable apprehension of bias, Judge Sparks could only have made racism an issue if evidence about it were adduced and linked to a legal argument. For other judges, life experience supports assessments of credibility which can lead to conclusions about the facts. Thus, racism is either part of the life experience of a judge that informs her determinations of fact, or it is itself a social fact that must be proven before it can be relevant to a decision. By contrast, the role of dominant perspectives in shaping what constitute legal facts typically goes unquestioned.

This dispute over how facts are made is instructive in the AI context as it makes explicit how human judgment shapes the data on which we rely. It also challenges the neutrality of facts and data, and centralizes the issue of who gets to determine what constitute facts. The NIST Framework identifies human cognitive bias as an important factor in AI bias, and the EU-US definitions also make it clear that bias can be found not just in data assembled and curated by humans, but in human decisions and processes across the AI lifecycle. The issues that surfaced in RDS around what is fact and what is opinion are also implicit in the building and operation of AI systems. Rather than purely technical tools, AI systems are complex socio-technical systems that are deeply embedded within a framework that includes people, processes, rules, and norms. The challenge is how to surface, query, and challenge these issues in ADS.

B. Transparency and Explainability

Transparency and explainability are identified as core values in ethical AI.[55] In the US NIST AI RMF, explainability is defined as “a representation of the mechanisms underlying AI systems’ operation”.[56] In other words, interpretability refers to a kind of cause-and-effect logic. Transparency aids in scrutiny, but as the NIST AI RMF points out, “[a] transparent system is not necessarily an accurate, privacy-enhanced, secure, or fair system.”[57]

Rights to an explanation of ADM are often quite limited. For example, in Canada’s proposed Consumer Privacy Protection Act (CPPA), the right to an explanation of a “prediction, recommendation or decision” is available only where the decision may have a “significant impact” on the data-subject.[58] The content of an explanation includes “the type of personal information that was used to make the prediction, recommendation or decision, the source of the information and the reasons or principal factors that led to the prediction, recommendation or decision.”[59] Under article 13(2)(f) of the General Data Protection Regulation (GDPR), data subjects have the right to “meaningful information about the logic involved as well as the significance and the envisaged consequences of such processing for the data subject”.[60] Explainability operates on a systemic rather than an individual basis, in large part because it is presumed that the system processes data in an objective way—outcomes are adequately explained by a description of parameters and inputs.[61]

The right to reasons in administrative law is a principle of administrative fairness, yet the extent of this right varies considerably. For routine administrative decisions of little consequence, general and formulaic explanations will suffice. More detailed reasons may only be required where assessments of credibility are made, the decision has an important impact on the individual, or there is a statutory right of appeal.[62] The form and extent of these reasons may also vary.[63]

In the AI context, basic rights to an explanation or to interpretability may not help in exposing bias or discrimination. In RDS, a majority of justices across all levels of appeal found that it would have been better for Judge Sparks to say less rather than more in reaching her decision. For example, Justice Glube stated that “judges must be extremely careful to avoid expressing views which do not form part of the evidence,”[64] suggesting that it is the expressing, and not the holding of the views that matters. Justice Cory, for the majority of the Supreme Court of Canada agreed that “if the decision had ended after the general review of the evidence and the resulting assessments of credibility,”[65] there would have been no basis on which to impugn it. Judge Sparks’ principal problem, it would seem, was that she deviated from the standard script and provided insight into her thought process. Formalized rights to an explanation in ADM may ultimately offer little help in unpacking—or challenging—the assumptions and human-cognitive choices in the system design or data classification that led to outcomes.

C. Biased Input and Biased Output

Although RDS dates to the 1990’s, we do not have to look very far into the past to find examples of how the existence of bias and discrimination are still contested in our society. Acceptance of systemic bias is particularly challenging. For example, in 2021, the Premier of Quebec argued that there was no systemic bias in Quebec, relying on his own interpretation of a dictionary definition of ‘systemic.’[66] In 2020, the Commissioner of the Royal Canadian Mounted Police also denied the presence of systemic racism in that force.[67] Taking a data-driven approach, a recent think tank report used statistical methods to contest the existence of systemic discrimination in Canada.[68] The fundamental nature of some human rights claims are also flat-out contested. For example, equality rights claims from the LGBTQ+ communities have been consistently resisted by those claiming that such rights conflict with their religious views.[69]

These examples suggest that addressing bias and discrimination in AI may be more complex than typically presented, even if it is identified as an ethical and legal imperative.[70] Different approaches are being developed to identify and monitor for bias and discrimination in AI. These means include techniques to improve data quality, and to disrupt factors that may lead to biased correlations.[71] Canada’s AIDA requires the person responsible for a high-impact AI system to “establish measures to identify, assess and mitigate the risks of harm or biased output that could result from the use of the system.”[72] One mitigation measure could be determining if data are representative of the relevant community to which the decision system will apply and making necessary corrections if the data are biased. Yet, as noted earlier, data curation alone will not suffice. Problems may arise from how training data were classified or weighed.[73] There may also be issues around the decision to use ADM in this context rather than human decision-makers. Furthermore, issues may arise around how any human-in-the-loop interacts with and responds to the AI system.

Countering the concerns that AI systems can perpetuate bias, some argue that ADM has potential to greatly improve notoriously flawed human decision-making.[74] Some human decision-makers might not only be biased, but they might also have techniques that allow them to mask it (for example, by emphasizing other factors in reasons for decision). Risk mitigation measures that include scrutiny of training data and ongoing monitoring of decision-making outputs in ADS have the potential to identify bias and to correct it, thus, in theory, making decision-making fairer and more impartial.[75] These are serious arguments. Yet, they depend to some extent on the idea that the answers lie in better data and algorithms. The more complex definitions of harm and bias discussed earlier make it clear that humans and their decision-making processes are still deeply embedded in the choice, design, and implementation of ADS. If approaches to bias and discrimination in AI are reduced to an assessment and modification of algorithms and data, these machine-focused bias solutions may short-circuit the complex and difficult discussions needed to address broader manifestations of bias in AI.[76]

D. The human-in-the-loop

In RDS the different justices assess the same paragraphs of Judge Spark’s decision looking to see if there is a reasonable apprehension of bias. Their conclusions are as much about what each perceives as meeting the legal test for bias as they are about their understanding of how lived experience shapes the interpretation of evidence. In RDS, we see how the dominant group’s lived experience is the largely unquestioned norm. In this way, the case centres the role of the human decision-maker and the relevance of identity in the decision-making process.

Where concerns are expressed about ADM, a “human-in-the-loop” is typically proposed to ensure a degree of actual or potential human participation in the decision-making process. The EU’s GDPR implicitly requires a human-in-the-loop for ADM, providing that “[t]he data subject shall have the right not to be subject to a decision based solely on automated processing [...].”[77] Article 14 of the EU AI Act, requires human oversight for high-risk systems, stating:

Human oversight shall aim at preventing or minimising the risks to health, safety or fundamental rights that may emerge when a high-risk AI system is used in accordance with its intended purpose or under conditions of reasonably foreseeable misuse, in particular when such risks persist notwithstanding the application of other requirements set out in this Chapter.[78]

The human-in-the-loop humanizes the process and provides a backstop against harmful and discriminatory bias that may have escaped technological risk mitigation measures. Notably, Canada’s AIDA and proposed CPPA do not require a human-in-the-loop for automated decision systems,[79] although the federal Directive on Automated Decision-Making provides that in cases of high-impact ADM, a final decision must be made by a human.[80]

In spite of the backstop role of the human-in-the-loop, there is little consideration of who the human-in-the-loop is or their status in relation to the decision-making process.[81] The human-in-the-loop may not be a decision-maker in the more formal sense of a tribunal member or a judge. There are no examples of provisions for an appointment process that might speak to independence or impartiality, nor is it clear what, if any, consideration will be given to the person’s background or experience. It is similarly unclear whether a human-in-the-loop will be someone with understanding of the technical features of the system, or a person trained in the policy behind the decision-making system, or even in ethics. The genericness of humans-in-the-loop is particularly interesting considering RDS, where the identification and experience of the decision-maker(s) at every stage was a central factor.[82]

Certainly, the extent to which judges are representative of Canadian society remains an issue that is important to equity and fairness in the justice system.[83] Moreover, just as there is underrepresentation in adjudicative roles, there is substantial underrepresentation of many groups among those involved in the design and development of AI—including women and racialized persons.[84] In this context, it is uncertain whether a person affected by ADM will have any means of knowing the identity of the human-in-the-loop who reviewed or participated in the decision. Neither is it evident how humans-in-the-loop will be assessed for automation bias. It is possible that in some cases, their own performance will be monitored by an automated system, which could impact impartiality if repeated divergence from AI recommendations is seen as anomalous behavior.

Perhaps most importantly, though, it remains uncertain whether the role of the human-in-the-loop is just to be a so-called ‘neutral’ check on the operations of ADS or whether this is also a context in which we continue to seek diversity in perspectives to shape and inform decision-making. The authors of a paper on intersectionality and AI bias call for a different and more inclusive approach to assessing fairness in AI, including “a widening of AI fairness practice by centering marginalized people and valorizing critical knowledge production that makes room for their voices.”[85] This is an encouraging, although rare articulation of this view. The lack of attention to the identity of the human-in-the-loop – in other words, their presumed neutrality – deeply resonates with the issues of identity and bias that are surfaced by RDS.

Conclusion

Risk regulation, the dominant paradigm for AI governance, is premised on the existence of risks that must be mitigated. Such risks include harmful bias and discrimination, which will be disproportionately borne by those who have experienced generations of discrimination, compounding existing inequality. Furthermore, although bias and discrimination are often presented as issues of data quality or flawed assumptions in algorithms, RDS teaches us that the problems are more complex than merely biased or incomplete data. There may be fundamental differences as to how we are prepared to understand or interpret the data, how we build the systems to process the data, how we adopt, implement and oversee systems, and who is engaged in these processes. While the NIST AI RMF and the EU-US AI definitions attempt to capture this broader understanding of how bias and discrimination may be manifested in ADM, this approach is less evident in Canada. In all contexts, there is a real risk that risk-mitigation measures will be reduced to automated assessments of outputs and enhanced data curation. Even though these are important activities, they are not sufficient. Just as the problems are not solely in the machines, neither are the solutions.

More fulsome approaches to bias and discrimination in AI are not limited to issues of data quality or coded assumptions; they go so far as to include the very choices that are made about how to deploy AI and in what contexts. RDS reminds us that very experienced, highly-trained and well-paid and respected members of society can have profoundly different opinions about the constitution of facts and the existence of bias. It is a reminder that bias and discrimination in AI systems are fundamentally human issues, and artificial intelligence is still a fundamentally human technology from its inception to its deployment. This reasoning suggests that we have much work to do—and much more challenging and complex work at that—in order to address bias and discrimination in AI.