Richard T. Horton
Associate Editor
Loyola University Chicago School of Law, LLM 2021

There’s no doubt that remote work, brought on by the coronavirus pandemic, will accelerate the digital revolution already underway. Consumers’ growing appetite to conduct their business online, rather than in-person, has fueled the proliferation of digitally accessible products and services. For instance, movie theaters have closed their doors while content streaming services have experienced exponential growth. And while the restaurant industry, as a whole, has suffered, ‘virtual’ kitchens and grocery delivery apps have picked up steam. A critical question that arises from these trends is “what can be done to eliminate biases in the algorithms that drive these digital transactions?”

How artificial intelligence works

In the 2000’s, when a customer needed a taxicab, she might call the cab company’s phone number and provide her pickup location and destination address to the dispatcher. The dispatcher may check his map to determine which cars are assigned to the customer’s ‘zone,’ which of those cars are available at the moment, and which are properly equipped for the customer’s needs (e.g., handicap accessible, black sedan service). The dispatcher then decides which taxicab to dispatch to the customer. Once the customer is in the cab, the driver determines the best route to get to the destination.

Now, in 2021, a customer’s chosen ride-sharing app does all of the work. It performs the same driver/rider matching and route selection processes, but the humans are no longer a part of the equation. Each of these decisions is now made automatically by the technology itself. Engineers use AI approaches, including machine learning, search, and optimization algorithms, to simulate the human intelligence needed to make these decisions efficiently.

Over time, the app could theoretically ‘learn’ to optimize its decisions through various methods such as incentivizing drivers to serve high-volume areas or prioritizing high-volume riders over occasional ones. Perhaps it’s harmless to prioritize the frequent rider that happens to be a corporate exec or traveling salesperson over the first-time rider couple headed to the pier for a day of leisure, but what happens when this good intentioned ‘learning’ results in disparate impact that segments society by gender, age, race, or socio-economic classes?

The potential for AI to ‘learn’ to discriminate

Researchers and social activists have begun to disrupt the digital disrupters in a growing movement to identify discriminatory algorithms. AI systems may discriminate because they were trained on data that reflects discrimination by humans. The most prominent example is the research conducted on facial recognition technology by MIT Media Lab researcher Joy Buolamwini, which demonstrated both racial and gender bias in the software of several big tech company products. Buolamwini proved that the commercial products were much more accurate at identifying the faces of men than women, and white people than black people.

AI systems may perpetuate and reinforce social inequalities that already exist in society. A notable case is Amazon’s internal-facing human resources recruiting system which was intended to sift through hundreds of digital applications and rank the candidates, thereby automating the painstakingly subjective process of discovering top talent. Engineers created qualitative models reflective of select job functions and taught the AI to look for terms used in candidate resumes to satisfy the criteria. Some terms were given more weight than other terms. Unsurprisingly, there were some unintended consequences. Based on males’ historical dominance of the tech industry, the AI effectively ‘learned’ that men were more preferrable candidates than women. The AI began to downgrade candidates that had attended women’s colleges or that had joined women’s organizations.

Existing regulation of discriminatory algorithms

Researchers are still studying the ethics involved in fair algorithms, but not much of this thought leadership has resulted in actual legislation. The European Union pioneered transparency of automated decisions in its General Data Protection Regulation (“GDPR”). The law requires data controllers to obtain data subjects’ informed consent when their personal data is processed by AI programs and the decision is used to significantly affect them, especially legally. GDPR gives the data subject the right to contest the decision and have a human intervene. And even greater protections are required when the automated decision utilizes special categories of personal data such as racial or ethnic origin, political opinions, and religious or philosophical beliefs.

The State of Illinois was the first in the U.S. to act on this issue. In 2019, state lawmakers passed the Artificial Intelligence Interview Act, which went into effect January 1, 2020. The law bans companies from using AI facial analysis technology on video recordings of job candidates unless the candidate is informed that AI is being used, what AI is and how its being used, and obtain the candidate’s consent. Maryland’s version of this law has similar notice and consent requirements, but it is narrowly limited to AI facial recognition technology.

Proposals for new legislation

Even municipal lawmakers have taken aim at discriminatory algorithms. In New York City, a proposed bill regulating the ‘sale of automated employment decision tools’ requires software developers, prior to selling the tool, to conduct a bias audit on the tool and provide subsequent audits free of charge and on an annual basis. Employers that utilize the tool to filter candidates for employment are required to notify candidates of its use and the job qualifications that were evaluated.

Remediation recommendations

Based on an analysis of the few enacted laws, pending bills, and research in the area of AI ethics and privacy, the following steps should be taken by companies to self-regulate against discriminatory algorithms. Lawmakers poised to adopt strong regulations may ultimately opt for a self-regulatory model and less demanding regulations if companies demonstrate a willingness to police themselves in the absence of US federal legislation.

Step 1 – assign accountability

Companies must take accountability for managing the risks of discriminatory algorithms. To establish accountability, companies should assign responsibility to an employee, or committee, qualified to manage the risks. In this case, qualified likely means the personnel has a background in computer science, AI, and/or risk management. Companies should assign responsibility to their ethics & compliance program and/or diversity & inclusion program (i.e. product diversity office). Companies, that haven’t yet established either, should consider assigning responsibility to the privacy program because automated decisions are sometimes regulated by privacy laws.

Step 2 – define and identify ‘high risk’ algorithms

Companies will need to develop and adopt policies defining the terms “automated decisions” and “artificial intelligence” so that company employees will have a clear understanding of which technologies fall within these definitions. Employees should be trained on these policies. Once a workable definition is adopted, the applicable technology must be identified. Companies should create an inventory of these technologies and prioritize them by their potential effect on consumers. GDPR indicates that only algorithms that “significantly affect” data subjects or “produces legal affects” are applicable. These terms likely include employee hiring, compensation, promotions, annual reviews, and bonuses. Products and services related to consumer loans (e.g., mortgage, auto, etc.), lines of credit, education (i.e., university admissions), and consumer price differentiation will also likely meet the standards.

Step 3 – document the algorithms

One of the biggest challenges in regulating algorithms is that they are inherently opaque. Software engineering principles and object-oriented programming dictate that algorithms be hidden away and encapsulated in methods that operate like black boxes. Agile methodology of software development arguably discourages detailed documentation of system functionality in exchange for increased flexibility and efficiency. And an overarching goal of AI methods is to allow the algorithms to change and improve over time as it ‘learns’ from new data. These realities make it so that even the engineers themselves may not know exactly how the technology works in every instance, in addition to the lack of detailed documentation. Therefore, engineering teams should implement documentation practices that capture the functionality as it is being developed for new systems and conduct retroactive reviews to create documentation of legacy systems.

Step 4 – select a risk framework and manage the risks

Companies should either design a custom risk and control framework or adopt one of the standards issued by their industry. For example, companies may leverage FATML’s Principles for Accountable Algorithms, which is a high-level set of principles that may be used as foundational risk areas. It may be even more useful to use the Principles as a starting point for developing a customized low-level detail risk framework, complete with granularly defined risk descriptions and corresponding controls.

Among other safeguards and redress mechanisms, the framework should mandate the implementation of controls including “notice,” “consent,” and “right to contest.” At the minimum, data subjects should be informed when artificial intelligence is being used and provided a meaningful explanation of what decisions it is making and how the algorithm works. And the data subject’s “right to contest” the automated decision should include the right to obtain human intervention (i.e., a review of the decision that does not solely rely on automation).

Companies should then conduct a risk assessment by reviewing the documentation from Step 3 and conducting stakeholder interviews to gather the information necessary to ascertain the current state of the anti-discrimination controls in place within the company. The unaddressed risks should be appropriately mitigated through implementation of the appropriate recommended controls. Once the remediation work is complete, companies may opt to periodically conduct independent fairness audits to certify their tech products as fair, unbiased, and free from discriminatory algorithms.