Tag: AEQUITAS

Fairness and bias correction in machine learning for depression prediction across four study populations

A study found that standard machine learning approaches often exhibit biased behaviours in predicting depression across different populations. It also demonstrated that both standard and novel post-hoc-bias mitigation techniques can effectively reduce unfair bias, though no single model achieves equality of outcomes.

The biases that were identified risk reinforcing structural inequalities in mental healthcare, particularly affecting underserved populations. This underscores the importance of analyzing fairness during model selection and transparently reporting the impact of debiasing interventions to ensure equitable healthcare applications.

Learn more about this study here: https://doi.org/10.1038/s41598-024-58427-7

Reference

Dang, V.N., Cascarano, A., Mulder, R.H. et al. Fairness and bias correction in machine learning for depression prediction across four study populations. Sci Rep 14, 7848 (2024).
Turning Evidence Into Action: The AEQUITAS Database

AI tools used in healthcare can carry hidden biases, sometimes performing less accurately for women, ethnic minorities, or other underrepresented groups. But knowing this risk exists in general is one thing. Being able to check whether a specific tool, in a specific clinical area, has documented bias issues is another, and that’s exactly the gap the AEQUITAS Database was built to fill.

The AEQUITAS Database is a structured, digital repository that systematically collects, organises, and provides access to evidence on gender and racial bias in biomedical AI. Rather than leaving healthcare professionals, researchers, and policymakers to piece together scattered studies and reports, it brings this dispersed evidence together into one coherent, accessible knowledge base.

The database is publicly accessible through the AEQUITAS project website, and it was built with a user-oriented approach, meaning both technical and non-technical users can navigate it effectively.

Built for multiple audiences

The Database was designed to serve several groups at once, each with different needs. For healthcare professionals, it’s a resource for identifying potential biases in AI-based diagnostic and decision-support tools, improving patient-centred care, and reducing the risk of misdiagnosis. For civil society organisations, it supports awareness-raising and advocacy for fair, inclusive healthcare systems. And for researchers and policymakers, it feeds into further scientific investigation, the development of clinical guidelines, and regulatory oversight at European level.

A practical tool

The Database is designed to be used, not just consulted occasionally. Before adopting a new AI tool, healthcare professionals can search for entries related to its disease area to check whether bias has been documented in similar tools. When an AI system produces unexpected outputs, the Database can help determine whether it’s a known pattern rather than an isolated anomaly. And when caring for patients from underrepresented groups, it can help clinicians understand whether the tools involved in their care carry documented disparities relevant to that patient’s profile.

The Database is also a living resource that will grow as new evidence becomes available. The more healthcare professionals, researchers, and organisations engage with it, the more valuable it becomes for everyone.
From Principles to Practice: How AEQUITAS Turns Policy into Action

AI systems used in healthcare are governed by strict ethical principles and legal requirements, from the EU AI Act to the EU Charter of Fundamental Rights. But knowing the rules isn’t the same as knowing how to apply them. Hospitals and civil society organisations (CSOs) need concrete, operational processes they can actually put into practice, day to day, to make sure these tools are used safely and fairly.

This is exactly what the AEQUITAS AI Regulatory Model was designed to do. It gives hospitals and CSOs a practical governance framework covering the full lifecycle of AI systems, from procurement and validation to deployment, monitoring, and accountability.

The model draws inspiration from approaches that have proven effective in other high-risk sectors. It echoes the logic of pharmacovigilance, the systematic tracking of adverse drug reactions, in what AEQUITAS calls an “AI-vigilance” system: a way to report and analyse incidents involving AI-supported clinical decisions, with a particular focus on equity outcomes. It also mirrors the patient safety culture familiar from aviation-inspired approaches in hospitals, built around identifying hazards, reporting incidents, and continuously improving.

At its core, the model is grounded in the EU Charter of Fundamental Rights (2009), particularly the principles of human dignity, equality and non-discrimination, and the right to healthcare, translating these rights into operational safeguards that follow an AI system throughout its use.

The five checkpoints

Rather than treating AI governance as a single box to tick, the model structures oversight around five sequential stages, each one a checkpoint designed to catch potential equity risks before they reach patients. Throughout this process, hospitals act as the main deployers, leading decisions and operational control, while CSOs play an independent oversight role, reviewing documentation, flagging equity-related risks, and conducting audits, without controlling day-to-day clinical decisions themselves.

These stages follow the natural lifecycle of any AI system in a hospital. It starts with scope and classification, determining an AI tool’s regulatory status before it’s even procured or developed. This sets the direction for everything that follows. Next comes procurement, often the moment hospitals have the greatest leverage, as it’s where suppliers can be required to commit to addressing bias risks throughout a system’s lifecycle. At this stage, CSOs are able to review procurement documentation and flag equity concerns. The third stage, validation, means independently evaluating how a system performs, not just overall, but across different patient groups, before it’s ever used with real patients.

Once a system passes validation, the deployment stage ensures the right safeguards are in place: trained oversight staff, mechanisms for clinicians to override AI recommendations, and clear communication with patients about when these tools are being used in their care. And because AI systems aren’t static, the final stage, ongoing monitoring, auditing, and accountability, keeps watching for emerging disparities long after a system goes live, with CSOs conducting independent audits and helping escalate persistent issues.

Building a shared network across Europe

Beyond the model itself, AEQUITAS is currently establishing a European network of CSOs and public hospitals. This shared governance layer means that insights, incidents, and mitigation strategies identified in one country contribute to a growing, Europe-wide knowledge base, one that benefits patients well beyond the institution where the issue was first spotted.

Together, these tools turn abstract principles into something hospitals and CSOs can actually act on, closing the gap between good intentions and equitable, everyday care.

References:

European Union. (2009). Charter of Fundamental Rights of the European Union. Official Journal of the European Union, C 303/1. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:12012P/TXT
Systematic review and meta-analysis of performance of wearable artificial intelligence in detecting and predicting depression

The systematic review and meta-analysis found that wearable AI systems demonstrate promising performance in detecting and predicting depression. However, substantial variability exists among algorithms and devices, thereby indicating that performance can vary significantly.

What this means is that disparities across different algorithms and devices were identified, suggesting that certain demographic groups may be underrepresented or inadequately served by current wearable AI systems. This variability underscores the need for further research to enhance the generalizability and fairness of these technologies in clinical practice.

Learn more about this review here: https://doi.org/10.1038/s41746 -023-00828-5

Reference

Abd-Alrazaq, A., AlSaad, R., Shuweihdi, F. et al. Systematic review and meta-analysis of performance of wearable artificial intelligence in detecting and predicting depression. npj Digit. Med. 6, 84 (2023).
Beyond Ethics: How Europe Is Regulating AI in Healthcare

Knowing that AI can be biased is one thing. Knowing what to do about it, legally and ethically, is another.

Medicine’s ethical foundations, extended to AI

Medicine has long been associated with high ethical standards, from the Hippocratic Oath to the Declarations of Geneva and Helsinki. Unfortunately, the history of medicine includes deeply troubling chapters, from eugenics, forced sterilisation of indigenous and marginalised populations, to non-consensual experimentation on people with disabilities. These historical failures are directly relevant today because the structural injustices behind them shaped decades of medical data collection, and their legacy is exactly what today’s AI systems risk inheriting when trained on that same data.

Biomedical ethics is built around four core ethical principles: autonomy (respecting a person’s right to make their own decisions), non-maleficence (avoiding harm), beneficence (actively promoting wellbeing), and justice (distributing benefits and risks fairly, especially given known disparities based on race, gender, and social status) (Beauchamp & Childress, 2019).

As AI entered clinical practice, ethicists added a fifth principle specifically for these technologies: explicability, the idea that the reasoning behind an AI system’s decisions must be understandable and open to scrutiny, not simply accepted at face value (Floridi et al., 2018).

The EU’s legal response

The European Union has built a multi-layered legal framework specifically to address these risks. At its centre is the AI Act, which classifies many healthcare applications, including diagnostic tools and clinical decision-support systems, as “high-risk.” This means they’re subject to strict requirements: providers must actively identify and mitigate risks of discriminatory outcomes, ensure meaningful human oversight, and maintain accuracy and transparency throughout the system’s lifecycle (EU AI Act, 2024).

Hospitals that deploy these systems have specific legal obligations too, including using AI in line with provider instructions, assigning competent human oversight, maintaining logs of system operation, and reporting serious incidents, particularly where there’s a risk of unequal outcomes across patient groups.

The AI Act doesn’t stand alone. It works alongside the Medical Device Regulation (MDR; European Union, 2017a) and In Vitro Diagnostic Regulation (IVDR; European Union, 2017b), which govern the safety of medical technologies, the GDPR (European Union, 2016), which protects the sensitive health data these systems rely on, and the emerging European Health Data Space (EHDS; European Union, 2025), designed to enable more representative datasets for future AI development. Together, they form a legal ecosystem, all ultimately grounded in the EU Charter of Fundamental Rights (2009), particularly its guarantees of non-discrimination, equality, and the right to healthcare.

All these policies translate into real responsibilities for hospitals and healthcare professionals, from understanding a tool’s limitations and knowing when to override its recommendations, to being transparent with patients about when and how these systems are used in their care.

Beyond raising awareness, the AEQUITAS project has developed a practical AI Regulatory Model that translates these legal and ethical principles into concrete steps hospitals and healthcare professionals can actually apply, turning complex EU legislation into something usable in everyday clinical practice.

References:

Beauchamp, T. L., & Childress, J. F. (2019). Principles of biomedical ethics (8th ed).

European Commission, Timeline for Implementation of the EU AI Act, Brussels: European Commission, 2024.

European Union. (2009). Charter of Fundamental Rights of the European Union. Official Journal of the European Union, C 303/1. https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:12012P/TXT

European Union, Regulation (EU) 2016/679 on General Data Protection Regulation (GDPR), Brussels: EU, 2016.

European Union, Regulation (EU) 2017/745 on Medical Devices (MDR), Official Journal of the European Union, L117, Brussels: European Union, 2017.

European Union, Regulation (EU) 2017/746 on In Vitro Diagnostic Medical Devices (IVDR), Official Journal of the European Union, L117, Brussels: European Union, 2017.

European Union, Regulation (EU) 2025/327 on the European Health Data Space (EHDS), Brussels: European Union, 2025.

Floridi, L., Cowls, J., Beltrametti, M., Chatila, R., Chazerand, P., Dignum, V., Luetge, C., Madelin, R., Pagallo, U., Rossi, F., Schafer, B., Valcke, P., & Vayena, E. (2018). AI4People—An Ethical Framework for a Good AI Society: Opportunities, Risks, Principles, and Recommendations. Minds and Machines, 28(4), 689–707. https://doi.org/10.1007/s11023-018-9482-5
When “Neutral” Data Isn’t: Gender and Racial Bias in Biomedical AI

Bias in AI isn’t abstract. It shows up in real diagnoses, real treatment decisions, and real patients, particularly those who already face multiple layers of disadvantage. Discrimination in AI is an intersectional issue, overlapping with age, gender, race, sexual orientation, and health status, among other dimensions of identity. The patients most at risk of being harmed by biased AI tools are often those who already face the most barriers to quality care.

Decades of biased medical data

The biases we see in AI didn’t come from nowhere. They’re a direct consequence of medical data collected over decades, data shaped by systemic discrimination and structural inequalities in healthcare research and practice (Cirillo et al., 2020; Cross et al., 2024).

For much of medical history, clinical and experimental studies focused predominantly on male participants. This is what is known as the gender health gap. In 2020, only 5% of global health research funding went to women’s health research. This was split into 4% for women’s cancers and 1% for all other women-specific health conditions, with 25% of that further limited to fertility research (Nature Reviews Bioengineering, 2024).

It’s worth noting this isn’t a one-way problem: in some areas, such as depression, men are underrepresented in clinical data, largely because they’re less likely to seek care, report symptoms, or receive a diagnosis (Smith et al., 2018).

Racial bias is equally well documented

Racial bias in medicine is also extensively documented, particularly in the United States where research shows that, for instance, Black patients and other minority groups receive fewer medical procedures, lower rates of surgical intervention, and fewer referrals to specialists than white patients, regardless of clinical need (Bowser, 2001; Williams & Wyatt, 2015). When AI systems are trained on data that reflects decades of existing inequalities in healthcare access and treatment, they risk encoding and perpetuating those inequalities at a much larger scale. While much of the evidence comes from the US, the structural conditions producing racial bias, including socioeconomic inequalities, barriers to access and underrepresentation in research, are present across Europe too.

LGBTQIA+ patients face their own layer of risk

It is also important to recognise the specific situation of LGBTQIA+ individuals, who experience discrimination in healthcare and are subject to stereotypes that affect the care they receive. These social and cultural factors perpetuate discrimination and have a measurable impact on health and healthcare. Research has shown that 16% of LGBTQIA+ individuals report discrimination in healthcare encounters, and 18% avoid seeking care altogether due to fear of mistreatment (Chang et al., 2025). Another large-scale study found that in emergency department scenarios, AI recommendations for LGBTQIA+ patients included mental health interventions six to seven times more often than was clinically appropriate (Chang et al., 2025).

How this plays out in three key areas

In cardiovascular care, women’s symptoms often differ from the “classic” presentation described in medical textbooks, itself largely based on male patient data (Fatunde et al., 2025). As a result, women are offered fewer diagnostic tests, less medication, and fewer specialist referrals (Al Hamid et al., 2024). Racial bias compounds the problem: pulse oximeters, devices routinely used to measure blood oxygen saturation, have been shown to produce less accurate readings for patients with darker skin tones (Sjoding et al., 2020), a bias that then feeds directly into AI-based triage systems.

In diabetes care, racial bias is extensively documented. Studies have shown that African American patients present systematically higher A1c values than white patients with the same average blood glucose (Karter et al., 2023). If an AI system uses A1c as a proxy for glycaemic control without accounting for this, it risks producing incorrect diagnoses for Black patients.

In depression, both gender and racial bias carry significant weight, especially in tools using natural language processing for screening. Men and women tend to express psychological distress differently (Pennebaker et al., 2003), so a system trained predominantly on one gender’s language patterns may screen inaccurately for the other. Most mental health AI tools also still operate on binary gender assumptions, excluding non-binary, transgender, and gender non-conforming individuals from both the data and the populations these tools are meant to serve (Hafner et al., 2024).

Why this matters

These biases translate into delayed diagnoses, inappropriate treatments, and unequal care, every day, for real patients. Understanding these patterns is exactly what equips healthcare professionals to ask better questions, challenge assumptions, and advocate for the patients most at risk, which is precisely what the AEQUITAS training is designed to do.

References:

Chang, C.T., Srivathsa N., Bou-Khalil, C., Swaminathan, A., Lunn, M.R., Mishra, K., Koyejo, S,. Daneshjou, R. (2025). Evaluating anti-LGBTQIA+ medical bias in large language models. PLOS Digit Health 4(9): e0001001. https://doi.org/10.1371/journal.pdig.0001001

Cirillo, D., Catuara-Solarz, S., Morey, C., Guney, E., Subirats, L., Mellino, S., Gigante, A.A., Valencia, A., Rementeria, M.J., Chadha, A.S., & Mavridis, N. (2020). Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare. npj Digit. Med. 3(81). https://doi.org/10.1038/s41746-020-0288-5

Cross, J. L., Choma, M. A., & Onofrey, J. A. (2024). Bias in medical AI: Implications for clinical decision-making. PLOS Digital Health, 3(11), e0000651. https://doi.org/10.1371/journal.pdig.0000651

Funding research on women’s health. (2024). Nature Reviews Bioengineering, 2, 797–798. https://doi.org/10.1038/s44222-024-00253-7

Hafner, F.S., Valdivia, A., Rocher, L. 2025. Gender Trouble in Language Models: An Empirical Audit Guided by Gender Performativity Theory. FAccT ’25: Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency, 1677–1695. https://doi.org/10.1145/3715275.3732112

Karter, A. J., Parker, M. M., Moffet, H. H., & Gilliam, L. K. (2023). Racial and Ethnic Differences in the Association Between Mean Glucose and Hemoglobin A1c. Diabetes Technology & Therapeutics, 25(10), 697–704. https://doi.org/10.1089/dia.2023.0153

Pennebaker, J. W., Mehl, M. R., & Niederhoffer, K. G. (2003). Psychological Aspects of Natural Language Use: Our Words, Our Selves. Annual Review of Psychology, 54(1), 547–577. https://doi.org/10.1146/annurev.psych.54.101601.145041

Sjoding, M. W., Dickson, R. P., Iwashyna, T. J., Gay, S. E., & Valley, T. S. (2020). Racial Bias in Pulse Oximetry Measurement. New England Journal of Medicine, 383(25), 2477–2478. https://doi.org/10.1056/NEJMc2029240

Smith, D.T., Mouzon, D.M., & Elliott, M. (2018). Reviewing the assumptions about men’s mental health: An exploration of the gender binary. American Journal of Men’s Health, 12(1), 78–89. https://doi.org/10.1177/1557988316630953

Bowser, R. (2001). Racial bias in medical treatment. Dick. L. Rev., 105(3), 365.

Williams, D. R., & Wyatt, R. (2015). Racial Bias in Health Care and Health: Challenges and Opportunities. JAMA, 314(6), 555. https://doi.org/10.1001/jama.2015.9260
The Hidden Ways AI Can Discriminate

When we talk about bias in AI, we are referring to “computer systems that systematically and unfairly discriminate against certain individuals or groups in favour of others” (Friedman & Nissenbaum, 1996). This is not about occasional errors, but about consistent, predictable patterns of unfair outcomes. Furthermore, biases in AI systems are complex, as they can enter the system at almost any stage, from the data it’s trained on to the way it’s ultimately used in a hospital setting. Recognising where bias originates, and how different types can reinforce each other, is the first step towards addressing it.

Three broad roots of bias

Researchers have identified three broad categories of bias that apply to all computer systems. The first is pre-existing bias, which comes from existing inequalities in society that get absorbed into the system, sometimes without anyone realising it (Friedman & Nissenbaum, 1996). The second is technical bias, which arises from the practical compromises made when translating complex human realities into clean computational form. And the third is emergent bias, which appears only after a system is deployed, as the world around it changes in ways the original design never anticipated.

Bias in the AI pipeline

These three categories give us a useful starting point. But for AI and machine learning specifically, researchers have identified seven more precise bias types that can affect these systems throughout their development and use (Suresh & Guttag, 2021):

Historical bias reflects prejudices and stereotypes already present in training data, even if the data is technically accurate. A 1990 study found that after coronary bypass surgery, male patients received pain medication significantly more often than female patients, who were instead given sedatives more frequently (Calderone, 1990). An AI trained on data like this would learn to replicate that same pattern, perpetuating the very inequality it inherited.

Representation bias happens when certain groups are underrepresented in training data. An AI trained mostly on data from one demographic will simply perform worse for everyone else. A well-known example is skin cancer detection tools that are significantly less accurate for patients with darker skin tones because the training datasets contained predominantly images from fair-skinned individuals (Guo et al., 2021).

Measurement bias creeps in when the way something is measured isn’t equally accurate or fair across groups. One striking case involved an algorithm that used healthcare costs as a proxy for how sick someone actually was, and that ended up disadvantaging Black patients, who, facing disproportionate levels of poverty, tended to spend less on healthcare than equally sick white patients (Obermeyer et al., 2019).

Aggregation bias occurs when a single, one-size-fits-all model is applied to genuinely diverse populations, ignoring the fact that the same data point can mean very different things depending on a person’s background.

Learning bias emerges from technical decisions made while building the model itself, choices that can amplify disparities already present in the data, sometimes without the developers being aware. Research has shown, for instance, that differential privacy, a technique meant to protect patient confidentiality, can end up reducing a model’s accuracy for underrepresented groups even further (Bagdasaryan & Shmatikov, 2019).

Evaluation bias happens when the datasets used to test and benchmark the model don’t reflect the real population it will actually serve, meaning a system can look accurate on paper while quietly failing specific groups of patients in practice.

Finally, deployment bias arises when a tool is used in a context very different from the one it was designed and tested for, undermining its reliability in ways that are easy to miss.

Why this matters for patients

Understanding where bias comes from is the first step to catching it, and to making sure the AI tools shaping modern medicine work fairly for everyone, not just the patients who happen to be well represented in the data.

This is exactly the kind of practical, structured understanding the AEQUITAS training equips healthcare professionals with, helping them recognise these patterns before they translate into real harm for real patients.

References:

Bagdasaryan, E., & Shmatikov, V. (2019). Differential Privacy Has Disparate Impact on Model Accuracy (arXiv:1905.12101). arXiv. https://doi.org/10.48550/arXiv.1905.12101

Calderone, K. L. (1990). The influence of gender on the frequency of pain and sedative medication administered to postoperative patients. Sex Roles, 23(11), 713–725. https://doi.org/10.1007/BF00289259

Friedman, B., & Nissenbaum, H. (1996). Bias in computer systems. ACM Transactions on Information Systems, 14(3), 330–347. https://doi.org/10.1145/230538.230561

Guo, L. N., Lee, M. S., Kassamali, B., Mita, C., & Nambudiri, V. E. (2021). Bias in, bias out: Underreporting and underrepresentation of diverse skin types in machine learning research for skin cancer detection-A scoping review. Journal of the American Academy of Dermatology, S0190-9622(21)02086-7. https://doi.org/10.1016/j.jaad.2021.06.884

Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453. https://doi.org/10.1126/science.aax2342

Suresh, H., & Guttag, J. (2021). A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle. Equity and Access in Algorithms, Mechanisms, and Optimization, EAAMO ’21, 1–9. https://doi.org/10.1145/3465416.3483305
AI Meets Healthcare: Understanding the Promise and the Risks

Artificial intelligence is no longer a distant, futuristic concept in medicine. It’s already here, and is becoming an everyday part of clinical practice, supporting diagnosis, clinical decision-making, treatment planning, and patient risk prediction. In fact, according to the World Health Organization, nearly three quarters of EU countries are now using AI-assisted diagnostics in some form, and that number is only expected to grow (Alowais et al., 2023; World Health Organization, 2026).

As the use of AI promises to further revolutionize the sector, what should we know before trusting it with our health?

How AI works

At its core, biomedical AI relies on three main approaches. Machine learning allows systems to learn patterns from data and make predictions without being explicitly programmed for every scenario. Deep learning, a more advanced form of machine learning, uses layered neural networks (loosely inspired by the human brain) to process complex information like medical images or clinical notes. And natural language processing (NLP) enables computers to understand and interpret human language, such as a doctor’s written notes or a patient’s description of their symptoms.

Real examples already in use

In cardiovascular care, AI tools are helping detect arrhythmias from ECG readings and estimating a patient’s individual risk of heart disease. In diabetes management, AI-powered retinal scans can catch signs of diabetic retinopathy earlier than ever before. And in mental health, natural language processing tools are being explored to help identify signs of depression in the way people write or speak, and to support patients between therapy sessions.

So what’s the catch?

Research has shown that AI applications can lead to misinterpretation and the promotion of biases and discrimination (Varona & Suarez, 2022). In essence, AI systems are only as good as the data they learn from. If that data reflects existing inequalities, for example, if it comes mostly from male patients, or predominantly from one ethnic group, the AI will inherit those biases. It will perform well for the people it “knows,” and poorly for everyone else.

Several documented cases have shown AI tools trained on biased datasets producing less accurate results for underrepresented patients, sometimes with real consequences for diagnosis and treatment.

What AEQUITAS is doing about it

This is exactly the challenge our AEQUITAS project, funded by the European Union, was designed to address. By developing training resources, a database documenting real cases of bias in biomedical AI, and a practical regulatory model for hospitals and CSOs, AEQUITAS is helping healthcare professionals and institutions across Europe use these powerful tools more safely, and more fairly, for every patient.

Understanding how AI works, and where it can go wrong, is the first step toward making sure it works for everyone.

References:

Alowais, S.A., Alghamdi, S.S., Alsuhebany, N. et al. Revolutionizing healthcare: the role of artificial intelligence in clinical practice. BMC Med Educ 23, 689 (2023). https://doi.org/10.1186/s12909-023-04698-z

Varona, D., & Suárez, J. L. (2022). Discrimination, Bias, Fairness, and Trustworthy AI. Applied Sciences, 12(12), 5826. https://doi.org/10.3390/app12125826

World Health Organization. 2026. “Artificial intelligence is reshaping health systems: state of readiness across the European Union.” World Health Organization.
Digital health tools for the passive monitoring of depression: a systematic review of methods

This systematic review examines studies linking passive data from smartphones and wearables to depression, identifying key methodological flaws and threats to reproducibility. It highlights biases such as representation, measurement, and evaluation bias, stemming from small, homogenous samples and inconsistent feature construction.

Although gender and race are not explicitly discussed, the lack of diversity in study populations suggests potential demographic bias. The review calls for improved reporting standards and broader sample inclusion to enhance generalizability and clinical relevance. These improvements are essential for ensuring that digital mental health tools are equitable and reliable across diverse populations.

Learn more about this review here: https://doi.org/10.1038/s41746-021-00548-8

Reference

De Angel, V., Lewis, S., White, K., Oetzmann, C., Leightley, D., Oprea, E., Lavelle, G., Matcham, F., Pace, A., Mohr, D. C., Dobson, R., & Hotopf, M. (2022). Digital health tools for the passive monitoring of depression: a systematic review of methods. NPJ digital medicine, 5(1), 3.
Bias Discovery in Machine Learning Models for Mental Health

This article examined how AI can unintentionally reproduce social and demographic biases when applied to mental health prediction. Using benzodiazepine prescriptions as a proxy for conditions such as depression and anxiety, a study analyzed machine learning models trained on patient data to identify systematic disparities.

It found that women are more frequently predicted to receive such treatments, reflecting gender bias, while the models perform less accurately for minority ethnic groups, indicating representation and evaluation bias. The AI models here are not used to prescribe drugs but rather to predict treatment likelihoods, revealing how bias in healthcare data can lead to inequitable AI performance in the context of depression-related care.

Learn more about the article here: https://doi.org/10.3390/info13050237

Reference

Mosteiro, P.J., Kuiper, J., Masthoff, J., Scheepers, F., & Spruit, M. (2022). Bias Discovery in Machine Learning Models for Mental Health. Inf., 13, 237.
Assessing Algorithmic Bias in Language-Based Depression Detection: A Comparison of DNN and LLM Approaches

A study found that large language models (LLMs) outperform traditional deep neural network (DNN) embeddings in automated depression detection and show reduced gender bias, through racial disparities remain. Among DNN fairness-mitigation techniques, the worst-group loss provided the best balance between overall accuracy and demographic fairness, while fairness-regularized loss underperformed.

The identified biases affect the fairness and diagnostic reliability of AI systems for mental health assessment, particularly by disadvantaging underrepresented racial and gender groups, mainly Hispanic participants in the case of this research. Such disparities risk perpetuating inequities in automated mental health screening and could undermine trust and validity in clinical or public health applications.

Learn more about the study here: https://doi.org/10.48550/arXiv.2509.25795

Reference

Junias, O., Kini, P., & Chaspari, T. (2025). Assessing Algorithmic Bias in Language-Based Depression Detection: A Comparison of DNN and LLM Approaches. 2025 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI), 1-7.
Developing personalized algorithms for sensing mental health symptoms in daily life

This study investigates algorithmic bias in AI tools that predict depression risk using smartphone-sensed behavioral data.

It finds that these tools underperform in larger, more diverse populations because the behavioral patterns used to predict depression are inconsistent across demographic and socioeconomic subgroups.

Specifically, the AI models often misclassify individuals from certain groups—such as older adults or those from different racial or gender backgrounds—as being at lower risk than they actually are. The authors emphasize the need for tailored, subgroup-aware approaches to improve reliability and fairness in mental health prediction tools. This work highlights the importance of addressing demographic bias to ensure equitable AI deployment in mental healthcare.

Learn more about this study here: https://doi.org/10.1038/s44184-025-00147-5

Reference

Timmons, A.C., Tutul, A.A., Avramidis, K. et al. Developing personalized algorithms for sensing mental health symptoms in daily life. npj Mental Health Res 4, 34 (2025).
Racial bias in AI-mediated psychiatric diagnosis and treatment: a qualitative comparison of four large language models

The article investigates racial bias in psychiatric diagnosis and treatment recommendations across four large language models (LLMs): Claude, ChatGPT, Gemini, and NewMes-15. The study evaluates the models’ responses to ten psychiatric cases representing five diagnoses (depression, anxiety, schizophrenia, eating disorders, and ADHD) under three conditions: race-neutral, race-implied, and race-explicitly stated (African American).

Key findings include:

1) Bias in Treatment Recommendations: LLMs often proposed inferior or divergent treatments when racial characteristics were explicitly or implicitly indicated, particularly for schizophrenia and anxiety cases. Diagnostic decisions showed minimal bias overall.

2) Model Performance: NewMes-15 exhibited the highest degree of racial bias, while Gemini demonstrated the least bias across conditions.

3) Statistical Analysis: A Kruskal–Wallis H-test revealed significant differences in bias among the LLMs, with Gemini being significantly less biased than ChatGPT and NewMes-15.

4) Challenges in AI Development: The study highlights that LLMs trained on biased datasets may perpetuate racial disparities in psychiatric care, even when specialized medical training data is used. Local LLMs, despite their cost and privacy advantages, showed higher susceptibility to bias compared to larger, online models.

Learn more about this study here: https://doi.org/10.1038/s41746-025-01746-4

Reference

Bouguettaya, A., Stuart, E.M. & Aboujaoude, E. Racial bias in AI-mediated psychiatric diagnosis and treatment: a qualitative comparison of four large language models. npj Digit. Med. 8, 332 (2025).
Domain Adversarial Training for Mitigating Gender Bias in Speech-based Mental Health Detection

A domain adversarial training (DAT) was developed in a study as a method to reduce gender bias in AI models for depression and PTSD detection using speech data (E-DAIC dataset).

DAT improved F1-scores up to +13% and reduced gender gaps in detection accuracy, improving generalization across male and female participants, specially addressing the effects of the latter’s underrepresentation.

Learn more about this study here: https://doi.org/10.48550/arXiv.2505.03359

Reference

Kim, J., Yoon, H., Oh, W., Jung, D., Yoon, S., Kim, D., Lee, D., Lee, S., & Yang, C. (2025). Domain Adversarial Training for Mitigating Gender Bias in Speech-based Mental Health Detection. 2025 47th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 1-7.
Minding the Gaps: Neuroethics, AI, and Depression

In this article, the author highlights the benefits and potential issues regarding the use of AI in depression diagnosis/treatment, focusing on the prevalent gender, racial and ethnicity biases.

It is mentioned that, given the historical, inherent biases in society generally and healthcare specifically, AI-driven advancements are not going to serve minority groups as a matter of course. Unless they are tailored to represent and serve all communities equally, they will exacerbate existing biases and disparities.

Learn more about this article here: https://nonprofitquarterly.org/minding-the-gaps-neuroethics-ai-and-depression/

Reference

Boothroyd, Gemma (2024), “Minding the Gaps: Neuroethics, AI, and Depression”, in Nonprofit Quarterly Magazine, winter 2024, “Health Justice in the Digital Age: Can We Harness AI for Good?”
Bias and Fairness in AI-Based Mental Health Models

The paper examines bias and fairness issues in AI-based mental health applications, including diagnostic tools, chatbots, and suicide risk prediction models. It reports how unrepresentative datasets lead to misdiagnosis and unequal outcomes across different socioeconomic, gender and racial groups – namely concerning women, local ethnic minorities or non-Western societies -, and presents mitigation strategies such as diverse datasets, fairness metrics, and human-in-the-loop approaches.

Learn more about this paper here: https://www.researchgate.net/publication/389214235_Bias_and_Fairness_in_AI-Based_Mental_Health_Models

Reference

Barnty, Barnabas & Joseph, Oloyede & Ok, Emmanuel. (2025). Bias and Fairness in AI-Based Mental Health Models.
AI and Mental Healthcare – ethical and regulatory considerations

This governmental report discusses the ethical and regulatory considerations of using artificial intelligence in mental healthcare in the UK.

Bias in AI tools (algorithmic bias) can stem from various places, including tools being trained on biased datasets and outputting discriminatory outcomes or developers making biased decisions in the design or training of such tools. For example, mental health Electronic health record (EHR) data is susceptible to cohort and label bias. This can occur because culture-bound presentations of mental disorders, combined with a lack of transcultural literacy among clinicians, often lead to both over- and under-diagnosis. People can also exhibit bias when using AI tools, such as over-relying on, or mistrusting AI outputs. All these biases can be conscious or unconscious.

Learn more about the report here: https://doi.org/10.58248/PN738

Reference

Gardiner, Hannah and Natasha Mutebi (2025), AI and Mental Healthcare – ethical and regulatory considerations, UK Parliament – POST, POSTnote 738, 31 January 2025
A Data-Centric Approach to Detecting and Mitigating Demographic Bias in Pediatric Mental Health Text: A Case Study in Anxiety Detection

This study examines classification parity across sex and finds that female adolescents have systematically under-diagnosed mental health disorders: their model’s accuracy was ~4 % lower and false negative rate ~9 % higher compared to male patients. The source of the bias resides in the textual data, namely notes corresponding to male patients tended to be on average 500 words longer and had distinct word usage. To mitigate this, the authors introduce a de-biasing method, based on neutralizing biased terms (gendered words and pronouns) and reducing sentences to essential clinical information. After correcting, diagnostic bias is reduced by up to 27%.

This emphasizes how linguistically transmitted bias—ensuing from word choice and gendered language—consistently leads to the under-diagnosis of mental health disorders among female adolescents, which critically undermines the impartiality of medical diagnosis and treatment.

Learn more about this study here: https://doi.org/10.48550/arXiv.2501.00129

Reference

Ive, J., Bondaronek, P., Yadav, V., Santel, D., Glauser, T., Cheng, T., Strawn, J.R., Agasthya, G., Tschida, J., Choo, S., Chandrashekar, M., Kapadia, A.J., & Pestian, J.P. (2024). A Data-Centric Approach to Detecting and Mitigating Demographic Bias in Pediatric Mental Health Text: A Case Study in Anxiety Detection.
The Role of Gender: Gender Fairness in the Detection of Depression Symptoms on Social Media

The study found that the BDI-Sen dataset used for depression symptom detection on social media exhibits gender bias, with machine learning models such as mentalBERT showing predictive disparities that generally favour male users. Although bias mitigation techniques like data augmentation reduced the bias, they did not eliminate it completely.

The existence of this bias affects the fairness and reliability of AI systems in detecting depression symptoms, leading to unequal predictive performance across genders. This can result in under- or over-identification of depression symptoms in certain groups, thereby compromising the validity of such systems for clinical or mental health monitoring.

Learn more about this study here: https://studenttheses.uu.nl/handle/20.500.12932/47734

Reference

Gierschmann, Lara (2024), The Role of Gender: Gender Fairness in the Detection of Depression Symptoms on Social Media, Utrecht University, unpublished Master Thesis
AEQUITAS
Proposal: Preparation of the CSOs and public healthcare sector to address gender and racial biases that might arise from the wide usage of AI in order to protect and promote fundamental rights

Implementation: 2025 to 2027

Call: CERV-2024-CHAR-LITI – Promote civil society organizations’ awareness of, capacity building and implementation of the EU Charter of Fundamental Rights

Proposed Budget: 1 061 761,00€

Keywords: gender and racial biases, biomedical AI, EU Charter of Fundamental Rights

Objective: The topic of the AEQUITAS project is the gender and racial biases that have been reported in biomedical AI and which can lead to misdiagnosis and mistreatment and how they pose a threat to the fundamental rights protected by the EU Charter. The aim of the project is 3-fold:

– to increase the capacity of the CSOs and human rights organizations in educating the public, monitoring the biases and advocating for the protection of the fundamental rights especially regarding the biomedical AI;

– to increase the knowledge of healthcare staff from public hospitals on the biases that biomedical AI can present due to the biased data from which they were fed and to help them approach and consult these AI systems with a critical mindset;

– to develop an AI Regulatory Model that will be used by CSOs and public hospitals in their practices;

– to develop a European network of CSOs and public hospitals that will collaborate and support each other in the effort to raise public’s awareness on the gender and racial biases of biomedical AI and on the applications of the EU Charter;

– to raise awareness of the EU Charter of fundamental rights and its application in the AI era;

– to develop and distribute policy recommendations in order to advocate for the need to regulate biomedical AI.

Partners:
- Innovation Hive – Kypseli Kainotomias
- Kentro Ginaikeion Meleton Kai Ereyvnon Astiki Mi K
- Universitat Zu Koln
- Center for the Study of Democracy
- C.I.P. Citizens in Power
- Moterų Informacijos Centras Asociacija Mic
- Lobby Europeo de Mujeres en Espana LEM España
- TIA Formazione Internazionale Associazione APS
- Health Citizens – European Institute
- Technologiko Panepistimio Kyprou
- Cyens Centre of Excellence
- Edex – Educational Excellence Corporation Limited
- Rite Research Institute for Technological Evolution
- Vucable
- TotalEU Production
Project Website: under development
Gender Bias in AI’s Perception of Cardiovascular Risk

The study investigated gender bias in GPT-4’s assessment of coronary artery disease risk and showed that there was a substantial shift in the perception of risk between men and women when a psychiatric comorbidity was added to the vignette, even when they presented identical complaints.

This resulted in women being assessed as having as lower risk of CAD when concurrently having a psychiatric condition.

Learn more about this study here: https://www.jmir.org/2024/1/e54242

Reference

Achtari M, Salihu A, Muller O, Abbé E, Clair C, Schwarz J, Fournier S
Gender Bias in AI’s Perception of Cardiovascular Risk
J Med Internet Res 2024;26:e54242
DOI: 10.2196/54242
Fairness in AI-Based Mental Health: Clinician Perspectives and Bias Mitigation

Considering how there is limited research on fairness in automated decision making systems in the clinical domain, particularly in the mental health domain, this study explores clinicians’ perceptions of AI fairness through two distinct scenarios: violence risk assessment and depression phenotype recognition using textual clinical notes.

Clinicians were engaged with through semi-structured interviews to understand their fairness perceptions and to identify appropriate quantitative fairness objectives for these scenarios. Then, a set of bias mitigation strategies were compared, developed to improve at least one of the four selected fairness objectives. The findings underscore the importance of carefully selecting fairness measures, as prioritizing less relevant measures can have a detrimental rather than a beneficial effect on model behavior in real-world clinical use.

Learn more about the article here: https://doi.org/10.1609/aies.v7i1.31732

Reference

Sogancioglu, G., Mosteiro, P., Salah, A. A., Scheepers, F., & Kaya, H. (2024). Fairness in AI-Based Mental Health: Clinician Perspectives and Bias Mitigation. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 7(1), 1390-1400.
Multimodal Fusion of EEG and Audio Spectrogram for Major Depressive Disorder Recognition Using Modified DenseNet121

Depression and anxiety are common, often co-occurring mental health disorders that complicate diagnosis due to overlapping symptoms and reliance on subjective assessments.

Standard diagnostic tools are widely used but can introduce bias, as they depend on self-reported symptoms and clinician interpretation, which vary across individuals. These methods also fail to account for neurobiological factors such as neurotransmitter imbalances and altered brain connectivity.

Similarly, clinical AI/ML models used in healthcare often lack demographic diversity in their training data, with most studies failing to report race and gender, leading to biased outputs and reduced fairness. EEG offers a promising, objective approach to monitoring brain activity, potentially improving diagnostic accuracy and helping address biases in mental health assessment, as this study found.

Learn more about it here: https://doi.org/10.3390/brainsci14101018

Reference

Yousufi, M., Damaševičius, R., & Maskeliūnas, R. (2024). Multimodal Fusion of EEG and Audio Spectrogram for Major Depressive Disorder Recognition Using Modified DenseNet121. Brain sciences, 14(10), 1018.
Deconstructing demographic bias in speech-based machine learning models for digital health

This study investigates algorithmic bias in AI tools that predict depression risk using smartphone-sensed behavioral data.

It finds that the model underperforms across several demographic subgroups, including gender, race, age, and socioeconomic status, often misclassifying individuals with depression as low-risk. For example, older adults and Black or low-income individuals were frequently ranked lower in risk than healthier younger or White individuals.

These biases stem from inconsistent relationships between sensed behaviors and depression across groups. The authors emphasized the need for subgroup-specific modeling to improve fairness and reliability in mental health AI tools.

Learn more about this study here: https://doi.org/10.3389/fdgth.2024.1351637

Reference

Yang M, El-Attar AA and Chaspari T (2024) Deconstructing demographic bias in speech-based machine learning models for digital health. Front. Digit. Health 6: 1351637.
Key language markers of depression on social media depend on race

A recent U.S. study published in PNAS found that artificial intelligence models analyzing social media posts can detect signs of depression in white Americans but are far less accurate for Black Americans, underscoring the dangers of using AI trained on non-diverse data in healthcare.

According to co-author Sharath Chandra Guntuku from Penn Medicine, these differences suggest that prior AI models and language-based assessments have largely overlooked racial diversity. While the researchers noted that social media analysis should not be used for diagnosis, it may still help assess risk or monitor mental health trends in communities

Learn more about the study here: https://www.pnas.org/doi/10.1073/pnas.2319837121

Reference

S. Rai et al (2024), Key language markers of depression on social media depend on race, Proc. Natl. Acad. Sci. U.S.A. 121 (14)
Artificial Intelligence in mental health and the biases of language based models

In this literature review of the uses of Natural Language Processing (NLP) models in psychiatry, an approach that “systematically evaluates each stage of model development to explore how biases arise from a clinical, data science and linguistic perspective” was employed to find existing patterns.

The result was that significant biases were found, with respect to religion, race, gender, nationality, sexuality and age.

Learn more about this review here: https://doi.org/10.1371/journal.pone.0240376

Reference

Straw, I., & Callison-Burch, C. (2020). Artificial Intelligence in mental health and the biases of language based models. PloS one, 15(12), e0240376.