When people think about AI, they often imagine objectivity. They imagine algorithms that soberly follow data and numbers, unaffected by personal opinions, emotions, or prejudice. But here’s the problem: AI systems don’t fall out of the sky. Humans develop them, they’re trained on human-generated data, shaped by human choices, and deployed in human contexts – all of which are far from neutral.

So whenever any AI system creates an output or takes a decision, the processes that lead to that outcome are deeply intertwined with the human experience and, hence, our biases and dispositions. That might not be a big issue if you merely ask ChatGPT to calculate 4 plus 4. The answer is objectively clear. But if you ask it to rank CVs for a job position, approve bank loans, or recommend a prison sentence for an individual, the subjective nature of these questions prohibits an objective answer. In these cases, concerns around bias, discrimination, and fairness become inevitable.

What is Fairness in AI, and Why Does it Matter?

Fairness is often viewed as an “Essentially Contested Concept” outside the realm of AI, due to its many definitions and interpretations across sectors (Mulligan et al., 2019). Although fairness is commonly described as the quality or state of being fair, what actually counts as fair remains highly contested and has served as the basis for long-lasting philosophical debates (Binns, 2018). 

Unsurprisingly, these debates also carry over into the world of Artificial Intelligence and Machine Learning (Mahoney et al., 2020). At the core, fairness, as a key principle in AI ethics, refers to the equitable and impartial treatment of individuals or groups by algorithmic systems, aiming to ensure that outcomes are free from unjust bias or discrimination (Mahoney et al., 2020; Mulligan et al., 2019). In practice, this means that decisions taken by AI systems, from hiring outcomes and credit approval to policing tactics or healthcare choices, should not systematically disadvantage certain people. However, fairness cannot be easily measured or optimised, unlike metrics such as accuracy or precision, as it is inherently multidimensional and context-dependent (Binns, 2018; Mehrabi et al., 2021). 

In purely technical terms, fairness often refers to the mitigation of algorithmic bias across datasets, models, and outputs. Technically, bias is frequently identified when one group experiences systematically different outcomes than another. Such disparities can stem from unrepresentative or incomplete training data, biased labeling practices, or opaque and unaccountable model design, making bias a “concern that continues throughout the AI lifecycle.” (UNESCO, 2024: 24). Addressing these issues is a big task in and of itself due to historical issues in data collection and often oversimplified approaches to labelling which often reproduce the very inequalities that fairness efforts aim to counter. For instance, in healthcare, existing AI models are often trained on biased datasets that, by predominantly involving male data, fail to capture the experiences of women and marginalized groups, leading to lower accuracy and higher rates of misdiagnosis (UNESCO, 2024). The challenge of countering historical biases in data is even more difficult in cases where there is intersectionality concerning groups and fairness. 

Importantly, fairness is not just a computational concern with a computational solution. It is also about how these systems are used and embedded into societal structures, historical trajectories, and institutional power. It is about how technology interacts with the lived experiences of humans. In other words, even a technically sound system can still be unjust.

This is especially problematic in sectors like healthcare, education, or criminal justice, where AI systems increasingly decide or support decisions about who gets care, access, or punishment. In the past decades, algorithmic biases and discrimination have been well-documented in several cases, from facial recognition systems that are less accurate for persons with darker skin (Buolamwini & Gebru, 2018), hiring algorithms that exacerbate gender and racial disparities (Dastin, 2022) to structural racism in healthcare data and algorithms (Obermeyer et al., 2019; Ferrara, 2023). Due to these direct examples of discrimination, unfair systems erode public trust (Ferrara, 2023). If people perceive AI systems and their outcomes to be unfair towards them, they will be less likely to engage with or accept these systems. This may result in a vicious cycle of self-reinforcing bias and discrimination, as people who are discriminated against might shy away from using certain technologies, thus being prevented from receiving the benefits they can provide. 

However, fairness is not just about preventing discrimination or harm. It should also be about inclusion, empowerment, and emancipation in an ideal world. Pursuing fairness shouldn’t end once unfair and discriminatory bias is removed. Instead, AI systems should be actively leveraged to promote a more just and equitable distribution of opportunities and resources and eventually enable a progression from fairness to justice. This is why it is imperative to ensure that the existing and developed systems don’t just work for most, but work for everyone. How might we achieve that? This will be the focus of next week’s blog post, so stay tuned!

References:

Binns, R. (2018, 21. Januar). Fairness in Machine Learning: Lessons from Political Philosophy. PMLR. https://proceedings.mlr.press/v81/binns18a.html

Buolamwini, J. & Gebru, T. (2018). Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Proceedings Of The 1st Conference On Fairness, Accountability And Transparency, 77–91. http://proceedings.mlr.press/v81/buolamwini18a/buolamwini18a.pdf

Dastin, J. (2022). Amazon Scraps Secret AI Recruiting Tool that Showed Bias against Women *. In Auerbach Publications eBooks (S. 296–299). https://doi.org/10.1201/9781003278290-44

Ferrara, E. (2023). Fairness and Bias in Artificial Intelligence: A Brief Survey of Sources, Impacts, and Mitigation Strategies. Sci, 6(1), 3. https://doi.org/10.3390/sci6010003

Mahoney, T., R. Varshney, K. & Hind, M. (2019). AI Fairness How to Measure and Reduce Unwanted Bias in Machine Learning (1st Edition). O’Reilly Media, Inc.

Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K. & Galstyan, A. (2021). A Survey on Bias and Fairness in Machine Learning. ACM Computing Surveys, 54(6), 1–35. https://doi.org/10.1145/3457607

Mulligan, D. K., Kroll, J. A., Kohli, N. & Wong, R. Y. (2019). This thing called fairness. Proceedings Of The ACM On Human-Computer Interaction, 3(CSCW), 1–36. https://doi.org/10.1145/3359221

Obermeyer, Z., Powers, B., Vogeli, C. & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453. https://doi.org/10.1126/science.aax2342

UNESCO. (2024). UNESCO Women for Ethical AI Outlook Study on Artificial Intelligence and Gender.

Further reading/watching/listening:

Books & Articles:

Barocas, S., Hardt, M. & Narayanan, A. (2023). Fairness and Machine Learning: Limitations and Opportunities. MIT Press.

Benjamin, R. (2019). Race after technology: Abolitionist Tools for the New Jim Code. Polity.

Eubanks, V. (2019). Automating inequality: How High-Tech Tools Profile, Police, and Punish the Poor. Picador.

O’Neil, C. (2016). Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. https://ci.nii.ac.jp/ncid/BB22310261

UNESCO. (2024). UNESCO Women for Ethical AI Outlook Study on Artificial Intelligence and Gender.

Videos & Podcasts:

“AI Bias and Fairness” by Ava Soleimany at MIT.
Watch on YouTube

“How I’m fighting bias in algorithms” by Joy Buolamwini at TEDTalks.
Watch on YouTube