Algorithms are increasingly governing our lives. Algorithmic decisions affect everything from digital spaces like streaming services and social media, to our credit scores, insurance options, and even career prospects (Andrejevic, 2019; Crawford 2015; Just, Latzer, 2019; Pasquale, 2015). Essentially, an algorithm is a set of instructions for problem solving or generating an outcome. Within information technology, they are crucial to data processing and automated reasoning (Flew, 2021, p. 82). Although advancing computer algorithms have facilitated the rapid growth of digital technology in the 21st century, they are not always necessarily effective or even appropriate. We must also remember that “artificial intelligence is not an objective, universal, or neutral computational technique that makes determinations without human direction” (Crawford, 2021, p. 211). In short, algorithms are simply statistical computations and are as good as their designers make them. And sometimes, their designers don’t do a particularly good job.
Algorithmic bias refers to the generation of unfair outcomes from the application of an algorithm. It might be a fundamental issue within its very formulation, or an unintended outcome due to the datasets being processed. Examples of algorithmic bias are far-reaching; algorithms have been accused of reinforcing damaging stereotypes within policing, the creation of online filter bubbles (Andrejevic, 2018, p. 50-58), and influencing voter behavior, to name a few.
What’s fascinating is that although algorithmic bias is widespread, everyday online algorithms (e.g., search engine results, social media content) have faced fewer consequences than those created and used by governments or companies for offline purposes, such as in hiring decisions or student grade allocation. Perhaps this is an issue of transparency; digital platforms can claim ‘trade secrets’ in order to keep their algorithmic decision-making processes close to their chest and “faulty data, invalid assumptions, and defective models can’t be corrected when they are hidden” (Pasquale, 2015, p. 18). Where the digital space seems so removed from human processing, it’s almost as though people are more likely to accept online algorithms as an inevitability, whether those algorithmic decisions are favourable or not (Just, Latzer, 2019). Furthermore, digital platforms’ global reach makes it hard to know where the fault came from or who to hold to account in cases of algorithmic bias. This is a smaller issue when specific institutions are behind a problematic system.
In 2020, the UK Government cancelled all end of year secondary school examinations. This included A-levels, final-year exams that determine a student’s university place. Instead, the nation’s exam watchdog, the Office of Qualifications and Examinations Regulation (Ofqual), created an algorithm that determined students’ grades primarily based on their school’s historic performance and teachers’ internal rankings of pupils. In theory, the algorithm was designed to avoid potential unfairness and consequent grade inflation that might’ve resulted from overly generous teachers had the final results been based on their predicted grades. In actuality, the algorithm proved to be ‘unfair’ in more overtly biased and discriminatory ways.
Photo: Henry Nicholls, Reuters. Retrieved from The Guardian (2020)
The algorithm was based on the following parameters and its details were only released after huge public pressure:
- CAGs, or ‘Centre assessed grades’ (i.e., teacher predicted grades within schools)
- Within each grade, teachers were asked to individually rank their students
- For cohorts of >15 in a class, the centre’s historic results were considered, and the distribution of final grades preferred to reflect these, rather than pupils’ CAGs
- For cohorts <15, students’ individual CAGs were not adjusted (Ofqual, 2020).
In the end, about 40% of A-level grades were below teachers’ original predictions, with downward adjustments disproportionately affecting students from lower socio-economic and ethnic minority backgrounds (BBC, 2020). As a result, many students missed out on their preferred university places. Cohorts of under 15 students per class / subject are almost exclusively found in private or selective schools, meaning that those in already privileged or advantageous positions did not suffer from algorithmic adjustments to their grades, though in theory, the algorithm had determined their grades. The algorithm was therefore accused of classism, racism, and enforcing a discriminatory status quo. It was eventually scrapped entirely following public outrage and teachers’ original predicted grades were used instead.
An algorithm is only as good as its design. This scandal is a perfect encapsulation of the concerns raised by algorithms. Terry Flew discusses several challenges (2021, p. 84), but two are particularly pertinent:
- bias, fairness, and transparency
- accountability and agency
Bias, Fairness, and Transparency
“The data that inform decisions are often biased in terms of sampling practices […] or reflect social biases […] problems of transparency and fairness arise from the massive imbalances that exist between the institutions and agencies that design algorithms […] and the people who are subject to the decisions based on those algorithms” (Flew, 2021, p. 84)
Fairness is a subjective moral term, and it has been highlighted that there is no one perfectly fair way to distribute opportunities (Elster, 1992; cited in Pasquale, 2015, p. 10). So, it’s arguable that any system that ranks, grades, excludes, etc. will always be vulnerable to claims of unfairness.
The use of terms like ‘(mis)judge’, ‘(mis)interpret’ etc. when talking about how algorithms make decisions humanise technological processes that don’t have capacity to reason like people. When Reuters reported that Amazon’s recruitment tool “did not like women” (Dastin, 2018), it ascribed an algorithm the power to have personal preferences, detracting from the fact that its programmers didn’t consider existing inequalities in the tech industry, and thus within the hiring data.
However, algorithms do make unfair decisions in the sense that they can create unfair outcomes. These are not morally loaded in the same way that human decision-making is, but an unfair outcome is an unfair outcome. Therefore, it might be more appropriate to refer to ‘unfair algorithmic design’ or ‘encoded bias’ rather than ‘algorithmic bias’ or ‘unfair algorithms’, to highlight the fact that the problems stem from the very top of the design chain, rather than shifting the blame onto an unconscious computer program through the language we use.
Ofqual’s sampling practices were inherently flawed. Choosing not to apply centres’ historic results to small cohorts within their algorithm meant that inequalities around types of schools most affected were inevitable, thus supporting common criticisms that algorithmic application has the potential to reflect social biases (Andrejevic, 2019; Crawford, 2021; Flew, 2021; Pasquale, 2015).
In her book Atlas of AI (2015), Crawford scathingly argues that AI is an engine for and fulfilment of existing power structures and social hierarchies. Algorithmic decision making is based on data relating to past and present and can therefore result in latent bias. Assuming future outcomes does not allow for progress or change. This is also the case in online algorithmic application like credit scores, loan repayment calculations, car insurance, etc. When applied to schoolchildren, and particularly those from underprivileged and underrepresented backgrounds, how was it morally sound to assume they will always perform like their peers before them? Media commentary at the time focused on exactly this, on students not being given an opportunity to prove themselves: “no matter how high a student was performing in the current year, they could only achieve what the algorithm determined was the top mark a student at that school could achieve based on historical data” (Fai, Bradley, Kirker, 2020).
Transparency is another major issue when it comes to algorithms, and a lack of it is especially rife among corporations (Pasquale, 2015). Governments (ideally) tend to be held to higher standards of accountability and transparency. A media storm following the release of grades. This, combined with lobbying from students and teachers, protests, and public outrage meant the government had very little choice but to firstly, explain how the algorithm worked and eventually, scrap it altogether.
What was really interesting from the fallout of the whole debacle were debates around social systems and hierarchies. Crawford argues that debates about bias within the realm of artificial intelligence don’t focus enough on “contending with underlying social, political, and economic structures” (2021, p. 128). She goes on to point out that “it is much rarer to have a public debate about why these forms of bias and discrimination frequently recur” (2021, p. 129). The discussions surrounding our case study is one of those rare instances.
Accountability and Agency
“If decisions are seen as being made ‘by machines’, then who is responsible for the resulting outcomes?” (Flew, 2021, p. 84)
Decisions are made by machines only as far as they have been designed to do so by people. As Crawford highlights, “AI systems are not autonomous, rational, or able to discern anything” without targeted training or predetermined rules (2021, p. 8). It’s hard to ascribe agency to a non-autonomous program, so accountability should lie with those that conceived of and created an algorithm. Even if it does not function in its intended way, as in the case of Amazon’s recruitment algorithm which favoured male candidates purely based on the fact that the company had more men in it and this had been interpreted as a desirable characteristic (Dastin, 2018), an algorithm is a tool that has been created by programmers with responsibilities to make sure their product functions as intended.
AI and algorithmic decision-making has often been seen as a solution to people’s inherent subjectivity (Andrejevic, 2019, p. 63). Technology analyst Nigel Rayner argues, “we humans are very bad at making decisions”, claiming that “when we look forward we are too influenced by what’s happened in the past” (Rayner, 2011; cited in Andrejevic, 2019, p. 63). This is highly ironic, given the fact that algorithms typically run on historic data to predict future outcomes. As we have seen, Ofqual’s insistence on considering historic school performance to avoid grade inflation (The Guardian, 2020) was criticised for the very issue Rayner’s conception of an ideal algorithm would supposedly solve.
Andrejevic points out that “while it is true that [AI, algorithmic] systems may reduce the element of human bias and emotion, in certain contexts, there are also good reasons for being circumspect about off-loading key-decisions onto them” (2019, p. 66). Indeed, human emotion can sway decisions, and it is understandable that the Department of Education and Ofqual were concerned about teachers being too lenient with their predicted grades, especially given the impact and disruption caused by the pandemic. However, as we’ve seen, the solution was even more problematic.
There is a point to be made about the UK government trusting a computer algorithm over a person to make decisions. This is a recurring issue in debates about technology, and its application in spaces like the internet; Pasquale discusses how societies function effectively because people have a high level of underlying trust (2015, p. 59). It’s something we’re not conscious of at any given moment, and our increased reliance on technology has us ascribe algorithmic decision-making a level of agency that is perhaps unwarranted.
Just as we “notice how much we rely on these routine forms of trust and the social infrastructures they support when they break down or are violated” (Pasquale, 2015, p. 59), so the UK population was angered when trusted decisions typically made by humans (aka teachers marking exams) were transferred to machines that seemed to butcher the job. The algorithm had not been trusted with agency, and the government that championed it was held to account by the public.
Closing Thoughts
Algorithms are incredibly useful tools, and in decision-making, they should be treated as such. Shoddy tools and poor workmanship often lead to ineffective and damaging results. Algorithms should be designed with as much forethought as possible. ‘What can go wrong? How might our worldview affect the outcome?’ are questions politicians, corporations, the public, and programmers should be asking themselves. We should be hyper-aware of the ways in which machines can generate results that we were not aiming for. Questions of unfairness, bias, accountability are all concerns that planners and designers of algorithms need to pay more attention to in a world where algorithmic decisions can quite literally affect the course of people’s lives, and are increasingly doing just that.
References
Adams, R., Elgot, J., Stewart, H., Proctor, K. (2020, August 20). Ofqual ignored exams warning a month ago amid ministers’ pressure. The Guardian. Retrieved from https://www.theguardian.com/politics/2020/aug/19/ofqual-was-warned-a-month-ago-that-exams-algorithm-was-volatile
Akec, A. (2020, August 17). The A-level algorithm chaos reveals society’s racist, classist biases. Dazed. Retrieved from https://www.dazeddigital.com/politics/article/50152/1/a-level-algorithm-grades-chaos-reveals-society-government-racist-social-biases
A-levels and GCSEs: How did the exam algorithm work?. (2020, August 20). BBC. Retrieved from https://www.bbc.com/news/explainers-53807730
Andrejevic, M. (2019). Automated Culture. In Automated Media. (pp. 44-72). London: Routledge.
Crawford, K. (2021). The Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence. New Haven, CT: Yale University Press.
Dastin, J. (2018, October 11). Amazon scraps secret AI recruiting tool that showed bias against women. Reuters. Retrieved from https://www.reuters.com/article/us-amazon-com-jobs-automation-insight-idUSKCN1MK08G
Fai, M., Bradley, J., Kirker, E. (2020, September 11) Lessons in ‘Ethics by Design’ from Britain’s A Level algorithm. Gilbert & Tobin. Retrieved from https://www.gtlaw.com.au/insights/lessons-ethics-design-britains-level-algorithm
Flew, T. (2021). Regulating Platforms. Cambridge: Polity.
Harkness, T. (2020, August 18). How Ofqual failed the algorithm test. UnHeard. Retrieved from https://unherd.com/2020/08/how-ofqual-failed-the-algorithm-test/
Inbuilt biases and the problem of algorithms. (2020, August 18). The Guardian. Retrieved from https://www.theguardian.com/education/2020/aug/17/inbuilt-biases-and-the-problem-of-algorithms
Just, N., Latzer, M. (2019). Governance by algorithms: reality construction by algorithmic selection on the Internet, Media, Culture & Society, 39(2), 238-258.
Mahadawi, A. (2020, August 19). It’s not just A-levels – algorithms have a nightmarish new power over our lives. The Guardian. Retrieved from https://www.theguardian.com/commentisfree/2020/aug/19/its-not-just-a-levels-algorithms-have-a-nightmarish-new-power-over-our-lives
Ofqual (Office of Qualifications and Examinations Regulation). (2020). Requirements for the calculation of results in summer 2020. Retrieved from https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/910614/6674_Requirements_for_the_calculation_of_results_in_summer_2020_inc._Annex_G.pdf
Pasquale, F. (2015). The Need to Know. In The Black Box Society: the secret algorithms that control money and information. (pp. 1-18). Cambridge: Harvard University Press.