CS5720 - Week 13
Slide 243 of 260

Fairness Metrics and Evaluation

Key Fairness Metrics

Fairness Metrics provide mathematical ways to measure whether an AI system treats different groups equitably.
  • ⚖️
    Demographic Parity
    Equal positive prediction rates across groups
  • 📊
    Equalized Odds
    Equal true positive and false positive rates
  • 👤
    Individual Fairness
    Similar individuals get similar predictions
  • 🎯
    Calibration
    Prediction confidence matches actual outcomes

Fairness Tradeoffs

⚠️ The Impossibility Theorem
It's mathematically impossible to satisfy all fairness criteria simultaneously!
Choose Your Fairness: Different contexts require different fairness definitions.
  • Accuracy vs. Fairness
    Optimizing for overall accuracy may hurt fairness
  • Group vs. Individual Fairness
    Treating groups equally vs. individuals fairly
  • Short-term vs. Long-term Fairness
    Immediate equity vs. systemic change
  • Competing Group Interests
    What's fair for one group may harm another

Interactive Fairness Scenarios

AI Hiring System
Group A: 60% hired (200/333 applicants)
Group B: 40% hired (80/200 applicants)
Overall Accuracy: 85%
Is this fair? Click to explore different perspectives.
Loan Approval AI
High-income: 90% approval, 5% default
Low-income: 30% approval, 15% default
Profit Maximization: Achieved
Should income determine loan access?
Medical Diagnosis AI
Urban patients: 95% accuracy
Rural patients: 78% accuracy
Overall Performance: 92%
When are accuracy differences acceptable?
Risk Assessment Tool
Group X: 45% labeled high-risk
Group Y: 23% labeled high-risk
Recidivism Prevention: 12% improvement
Can public safety justify disparate impact?
← Previous Next →
Prepared by Dr. Gorkem Kar