Skip to content

Bias and Fairness Testing in Practice

In one line: Test an AI system for unequal outcomes the way you test for bugs: define who could be harmed, build test sets across groups, measure outcomes and error rates by group, decide an acceptable gap in writing, and re-test after every change. Last reviewed: 2026-11-07

Judgment guide, not legal advice. What counts as unlawful discrimination depends on the law and the setting.

The short version

  • Bias is not only a data problem. NIST notes that biases "remain endemic across technology processes and can lead to harmful impacts regardless of intent" (NIST SP 1270 abstract, read 2026-11-07). Intent is not a defence in practice; measure outcomes.
  • Fairness has no single definition. You must choose what to measure (for example, selection rates, error rates, or calibration by group) and be able to justify the choice for your use case.
  • You usually cannot fix what you cannot measure. That means sometimes needing group data to test, collected and handled lawfully. See privacy and data protection.
  • Regulators already expect it for consequential decisions. The EEOC said in 2021 that AI tools in employment decisions may perpetuate bias and that anti-discrimination laws still apply (read 2026-11-07). Colorado replaced its 2024 AI act with SB26-189 in May 2026, which covers automated decision-making in consequential decisions (see US state AI laws).
  • Test generative systems too: chat answers, summaries, and image output can differ in quality or tone by name, dialect, gender or language.

The method

  1. Scope the harm. Who is affected, what decision or output matters, what is the worst realistic outcome.
  2. Pick groups that matter for the use and the law (for example protected classes in hiring, language or dialect in support).
  3. Build a test set. Real data where lawful, plus synthetic paired examples that differ only in a group signal (a name, a pronoun, a dialect).
  4. Choose metrics:
Metric Question it answers
Selection or approval rate by group Does one group get favourable outcomes more often?
False positive / false negative rate by group Are mistakes spread unevenly?
Score or quality gap on paired prompts Does the output change when only the group signal changes?
Refusal or toxicity rate by group Does the system treat some groups' requests differently?
  1. Set thresholds in advance. In US employment practice, a common screening heuristic compares group selection rates (the "four-fifths" or 80 percent rule). 29 CFR 1607.4(D) says a selection rate for any group below four-fifths (80 percent) of the highest group's rate will generally be regarded as evidence of adverse impact; confirm its use with counsel. A heuristic is a flag to investigate, not a pass or fail.
  2. Investigate gaps: data coverage, labels, features that act as proxies (such as postcode), prompt wording, thresholds.
  3. Mitigate and re-test: rebalance data, change features, adjust thresholds, add human review, or decide not to automate.
  4. Document and monitor. Keep results, decisions and owners. Re-test on every model, prompt or data change, and on a schedule.

A worked example (paired prompts)

You run a resume-summary tool. Take 100 resumes, create two copies of each that differ only in the candidate's first name (chosen to signal different backgrounds), run both, and compare: summary length, strength of language, and recommended-to-interview rate. If the rate differs materially, you have a flag to investigate before launch. Use an LLM or human rater with a fixed rubric, and have people spot-check.

Common mistakes

  • Testing only overall accuracy. A 95% average can hide a 70% group.
  • "Removing" the sensitive column and calling it done; proxies remain.
  • One audit at launch, never repeated.
  • Treating a vendor's fairness claim as your evidence.
  • Measuring only the model, not the whole process with humans in it.

Next steps

Sources

  • NIST, Towards a Standard for Identifying and Managing Bias in AI (SP 1270): https://www.nist.gov/publications/towards-standard-identifying-and-managing-bias-artificial-intelligence (fetched 2026-11-07; abstract only, the page did not give the bias categories)
  • EEOC, AI and Algorithmic Fairness initiative: https://www.eeoc.gov/newsroom/eeoc-launches-initiative-artificial-intelligence-and-algorithmic-fairness (fetched 2026-11-07)
  • Colorado SB26-189: https://leg.colorado.gov/bills/sb26-189 (fetched 2026-11-07)
  • eCFR, 29 CFR 1607.4: https://www.ecfr.gov/current/title-29/part-1607/section-1607.4 (fetched 2026-11-07)