2 min read

Addressing discrimination in AI

Tshedimoso Makhene Jan 15, 2025 5:23:45 PM

AI in healthcare

Discrimination in AI is the unfair or unequal treatment of individuals or groups by AI systems, often stemming from biases in data or algorithmic design. This can manifest in outcomes that disadvantage specific demographics, perpetuate societal inequities, or create new forms of injustice.

What causes discrimination in AI?

Discrimination in AI stems primarily from two sources: biased data and biased algorithms.

Biased data: AI systems learn from historical data. If this data reflects societal biases—such as gender, racial, or socioeconomic disparities—the AI can replicate and even amplify these biases. For instance, if a recruitment algorithm is trained on data from a company with a history of underrepresenting women, the algorithm may favor male candidates.
Algorithmic bias: Even when the data is relatively balanced, the way algorithms are designed and optimized can introduce bias. For example, prioritizing certain metrics over others, such as maximizing engagement on social media, can lead to unequal outcomes for different demographic groups.

Impacts of AI discrimination in healthcare

AI discrimination is not just a theoretical concern; it has tangible consequences in healthcare. Some notable examples include:

Diagnosis and treatment: AI models used in diagnostics may underperform for marginalized groups if these groups are underrepresented in training datasets. For instance, studies have found that dermatology algorithms often fail to accurately identify conditions on darker skin tones because the data used to train them lacks sufficient diversity.
Resource allocation: Predictive models in healthcare resource management may prioritize certain demographics over others, leading to unequal access to critical treatments or interventions. This can perpetuate existing disparities in healthcare outcomes.
Personalized medicine: AI-driven tools designed for personalized medicine may provide suboptimal recommendations for minority groups, as their unique genetic or environmental factors are less likely to be adequately represented in datasets.
Health insurance: Research has found that AI systems used by insurers to assess risk or determine premiums can inadvertently discriminate against individuals from certain socioeconomic or ethnic backgrounds, reinforcing systemic inequalities.

Addressing discrimination in AI

To address discrimination in AI, the HHS Office for Civil Rights (OCR) issued a letter outlining guidelines for the responsible use of AI tools in healthcare. The letter addresses the importance of adhering to Section 1557 of the Affordable Care Act, which prohibits discrimination by healthcare providers and insurers when using AI-driven tools for making patient care decisions.

The final rule under Section 1557 broadens nondiscrimination protections to include AI and other new technologies used in patient care, referred to as "patient care decision support tools." It requires covered entities to prevent discrimination based on race, color, national origin, sex, age, or disability when implementing these tools within health programs or activities. By applying civil rights principles in this context, the rule ensures that technological advances promote equity rather than hinder it in healthcare.

The rule requires covered entities to:

Identify risks of discrimination: Covered entities must take the necessary steps to assess whether their tools use input variables that could lead to discriminatory outcomes. Efforts may include:
- Reviewing guidance in the Section 1557 final rule.
- Researching peer-reviewed studies or industry recommendations.
- Developing or using AI safety registries.
- Gathering input variable details from tool vendors.
Mitigate risks of discrimination: After identifying risks, entities must implement reasonable measures to minimize these risks. Mitigation efforts may include:
- Establishing policies for tool usage and monitoring outcomes.
- Training staff to recognize and address potential bias.
- Auditing tools in real-world applications.
- Ensuring “human-in-the-loop” oversight to override biased outputs.

FAQs

Can AI systems be designed to prevent discrimination?

Yes, through the use of fairness-focused algorithms, diverse and representative datasets, and regular bias audits, AI systems can be designed to minimize discriminatory outcomes.

What is the Affordable Care Act?

The Affordable Care Act (ACA) is a comprehensive healthcare reform law enacted in 2010. Its primary goals are to expand access to affordable health insurance, improve the quality of care, and reduce healthcare costs.

Are there penalties for non-compliance with Section 1557?

Entities found in violation of Section 1557 can face legal and financial penalties, including loss of federal funding and enforcement actions by the Office for Civil Rights (OCR).