AI2 Releases FairStress Benchmark to Measure Demographic Fairness in LLM Decision-Making

AI2 has released FairStress, a benchmark designed to measure demographic fairness in large language model decision-making, and integrated it into Olmo-eval, its open evaluation workbench. The benchmark includes 300 two-candidate decision scenarios across 29 domains, with variations in how demographic identity is presented and adversarial prefixes that expand the dataset to more than 13 million test cases. It covers a range of open-weight and frontier models, including Olmo, Llama, Qwen, Gemma, Claude and the GPT family.

The findings show that models behave nearly symmetrically when no demographic information is present, but become less symmetric as identity cues become explicit. When one candidate is clearly better qualified, models were more likely to abandon the correct choice under persuasive arguments favoring a minority-coded candidate, a pattern the researchers describe as anti-stereotypical over-correction. Differences also appeared in how models explained decisions and in their internal activation patterns.

AI2 also introduced FairStress-Core, a smaller subset selected using item response theory. Tests of several debiasing methods found that most did not reliably reduce the asymmetry, while counterfactual chain-of-thought narrowed the gap in how readily models abandoned correct decisions while largely preserving accuracy. The researchers say FairStress aims to push evaluation beyond final choices toward a fuller understanding of model decision-making.