Tag: EU Charter of Fundamental Rights

Bias in AI isn’t abstract. It shows up in real diagnoses, real treatment decisions, and real patients, particularly those who already face multiple layers of disadvantage. Discrimination in AI is an intersectional issue, overlapping with age, gender, race, sexual orientation, and health status, among other dimensions of identity. The patients most at risk of being harmed by biased AI tools are often those who already face the most barriers to quality care.

Decades of biased medical data

The biases we see in AI didn’t come from nowhere. They’re a direct consequence of medical data collected over decades, data shaped by systemic discrimination and structural inequalities in healthcare research and practice (Cirillo et al., 2020; Cross et al., 2024).

For much of medical history, clinical and experimental studies focused predominantly on male participants. This is what is known as the gender health gap. In 2020, only 5% of global health research funding went to women’s health research. This was split into 4% for women’s cancers and 1% for all other women-specific health conditions, with 25% of that further limited to fertility research (Nature Reviews Bioengineering, 2024). 

It’s worth noting this isn’t a one-way problem: in some areas, such as depression, men are underrepresented in clinical data, largely because they’re less likely to seek care, report symptoms, or receive a diagnosis (Smith et al., 2018).

Racial bias is equally well documented

Racial bias in medicine is also extensively documented, particularly in the United States where research shows that, for instance, Black patients and other minority groups receive fewer medical procedures, lower rates of surgical intervention, and fewer referrals to specialists than white patients, regardless of clinical need (Bowser, 2001; Williams & Wyatt, 2015). When AI systems are trained on data that reflects decades of existing inequalities in healthcare access and treatment, they risk encoding and perpetuating those inequalities at a much larger scale. While much of the evidence comes from the US, the structural conditions producing racial bias, including socioeconomic inequalities, barriers to access and underrepresentation in research, are present across Europe too.

LGBTQIA+ patients face their own layer of risk

It is also important to recognise the specific situation of LGBTQIA+ individuals, who experience discrimination in healthcare and are subject to stereotypes that affect the care they receive. These social and cultural factors perpetuate discrimination and have a measurable impact on health and healthcare. Research has shown that 16% of LGBTQIA+ individuals report discrimination in healthcare encounters, and 18% avoid seeking care altogether due to fear of mistreatment (Chang et al., 2025). Another large-scale study found that in emergency department scenarios, AI recommendations for LGBTQIA+ patients included mental health interventions six to seven times more often than was clinically appropriate (Chang et al., 2025).

How this plays out in three key areas

In cardiovascular care, women’s symptoms often differ from the “classic” presentation described in medical textbooks, itself largely based on male patient data (Fatunde et al., 2025). As a result, women are offered fewer diagnostic tests, less medication, and fewer specialist referrals (Al Hamid et al., 2024). Racial bias compounds the problem: pulse oximeters, devices routinely used to measure blood oxygen saturation, have been shown to produce less accurate readings for patients with darker skin tones (Sjoding et al., 2020), a bias that then feeds directly into AI-based triage systems.

In diabetes care, racial bias is extensively documented. Studies have shown that African American patients present systematically higher A1c values than white patients with the same average blood glucose (Karter et al., 2023). If an AI system uses A1c as a proxy for glycaemic control without accounting for this, it risks producing incorrect diagnoses for Black patients.

In depression, both gender and racial bias carry significant weight, especially in tools using natural language processing for screening. Men and women tend to express psychological distress differently (Pennebaker et al., 2003), so a system trained predominantly on one gender’s language patterns may screen inaccurately for the other. Most mental health AI tools also still operate on binary gender assumptions, excluding non-binary, transgender, and gender non-conforming individuals from both the data and the populations these tools are meant to serve (Hafner et al., 2024).

Why this matters

These biases translate into delayed diagnoses, inappropriate treatments, and unequal care, every day, for real patients. Understanding these patterns is exactly what equips healthcare professionals to ask better questions, challenge assumptions, and advocate for the patients most at risk, which is precisely what the AEQUITAS training is designed to do.

References:

Chang, C.T., Srivathsa N., Bou-Khalil, C., Swaminathan, A., Lunn, M.R., Mishra, K., Koyejo, S,. Daneshjou, R. (2025). Evaluating anti-LGBTQIA+ medical bias in large language models. PLOS Digit Health 4(9): e0001001. https://doi.org/10.1371/journal.pdig.0001001

Cirillo, D., Catuara-Solarz, S., Morey, C., Guney, E., Subirats, L., Mellino, S., Gigante, A.A., Valencia, A., Rementeria, M.J., Chadha, A.S., & Mavridis, N. (2020). Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare. npj Digit. Med. 3(81). https://doi.org/10.1038/s41746-020-0288-5

Cross, J. L., Choma, M. A., & Onofrey, J. A. (2024). Bias in medical AI: Implications for clinical decision-making. PLOS Digital Health, 3(11), e0000651. https://doi.org/10.1371/journal.pdig.0000651

Funding research on women’s health. (2024). Nature Reviews Bioengineering, 2, 797–798. https://doi.org/10.1038/s44222-024-00253-7

Hafner, F.S., Valdivia, A., Rocher, L. 2025. Gender Trouble in Language Models: An Empirical Audit Guided by Gender Performativity Theory. FAccT ’25: Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, 1677–1695. https://doi.org/10.1145/3715275.3732112

Karter, A. J., Parker, M. M., Moffet, H. H., & Gilliam, L. K. (2023). Racial and Ethnic Differences in the Association Between Mean Glucose and Hemoglobin A1c. Diabetes Technology & Therapeutics, 25(10), 697–704. https://doi.org/10.1089/dia.2023.0153

Pennebaker, J. W., Mehl, M. R., & Niederhoffer, K. G. (2003). Psychological Aspects of Natural Language Use: Our Words, Our Selves. Annual Review of Psychology, 54(1), 547–577. https://doi.org/10.1146/annurev.psych.54.101601.145041

Sjoding, M. W., Dickson, R. P., Iwashyna, T. J., Gay, S. E., & Valley, T. S. (2020). Racial Bias in Pulse Oximetry Measurement. New England Journal of Medicine, 383(25), 2477–2478. https://doi.org/10.1056/NEJMc2029240

Smith, D.T., Mouzon, D.M., & Elliott, M. (2018). Reviewing the assumptions about men’s mental health: An exploration of the gender binary. American Journal of Men’s Health, 12(1), 78–89. https://doi.org/10.1177/1557988316630953

Bowser, R. (2001). Racial bias in medical treatment. Dick. L. Rev., 105(3), 365.

Williams, D. R., & Wyatt, R. (2015). Racial Bias in Health Care and Health: Challenges and Opportunities. JAMA, 314(6), 555. https://doi.org/10.1001/jama.2015.9260