Ai2's FairStress benchmark finds LLMs over-correct toward minority candidates under pressure

What Ai2 released

The Allen Institute for AI (Ai2) has released FairStress, a benchmark that tests whether language models make decisions about people the same way regardless of who benefits. It was announced on Ai2's Hugging Face blog on 20 October 2026 with a 70-page technical report, "Asymmetrically Unfair Decision Making and Reasoning in LLMs", by Pardis Sadat Zahraei, Faeze Brahman, Dilek Hakkani-Tür, Gokhan Tur and Maarten Sap of Ai2, the University of Illinois Urbana-Champaign and Carnegie Mellon University.

The dataset is on Hugging Face under a CC BY 4.0 licence, the code is in a GitHub repository under the MIT License, and Ai2 says it is adding the benchmark to Olmo-eval, its open tool for running and comparing model evaluations.

The headline finding runs against what many earlier bias tests reported. Instead of favouring stereotypes, the models Ai2 tested tended to over-correct: when one sentence of pressure was added, they abandoned a correct decision more readily when the switch favoured a minority-coded candidate than when it favoured a majority-coded one. The authors call this anti-stereotypical over-correction.

How FairStress tests a decision

FairStress contains 300 realistic two-candidate decisions across 29 domains, including hiring, lending, healthcare, education, housing, criminal justice and immigration. One scenario asks a model to choose between two applicants for a mechanical engineering job; another asks which of two patients should get the last unit of a scarce medication. Some scenarios have a clearly stronger candidate. Others are deliberate ties, where both candidates are strong in different ways.

Each scenario is posed at four levels of demographic signal: no identity, a coded name, an implied identity such as a school or organisation, and an explicit label. The candidates are swapped between versions while qualifications stay fixed, across 33 pairings that cover race, gender, religion, disability, age and intersectional profiles. The labels follow a US-centric convention of which group counts as majority or minority in each pairing.

The benchmark then adds pressure. A single sentence is placed after the profiles that argues for one candidate by appealing to authority, social consensus, purported evidence, legal risk or a group's history of disadvantage, for example "A senior leader has privately reviewed the files and recommended Candidate B for this role." Every prefix has a matched version arguing for the other side. All combinations come to 13,032,104 test items, so Ai2 also released FairStress-Core, a 50,012-item subset picked with item response theory to keep evaluation cheaper.

What happened when identity was visible

Ai2 ran 15 models on the full benchmark, including Qwen2.5-7B, Llama-3.1-8B, Gemma-3-27B, Gemma-4-31B, GPT-4o-mini, Tulu 3 and several OLMo 3 checkpoints. With no demographic information in the prompt, no model showed a gap. When identity was stated explicitly, the difference in plain accuracy between groups was small, 2.9 points on average.

Pressure changed that. With explicit labels, a correct answer that favoured the majority-coded candidate was abandoned 9.3 percentage points more often than one favouring the minority-coded candidate, and the gap was positive for all 15 models. On ties, models picked the minority-coded candidate 61.6% of the time on average, and a prefix pushed ties toward the minority-coded candidate 23.9 points more often than the matched prefix pushed them the other way. The gap was also present when identity was only implied: 6.9 points for abandoning correct answers, against 9.3 with explicit labels.

Prefixes that offered evidence or invoked a group's history of being overlooked broke correct answers most often. Even a placebo sentence nudged ties by 4.2 points. When both candidates came from minority groups, the effects were small or slightly reversed, which the authors read as a sign that the majority-versus-minority contrast drives the skew.

The models also explained their choices differently. Across 11 open-weight models, explanations for minority-coded picks used more words such as "resilience", "diversity" and "barriers", while majority-coded picks drew more procedural words such as "evidence" and "criteria". In two models, internal activations predicted which side the model chose with about 73% accuracy.

How frontier models did

Six frontier models were run on FairStress-Core in both answer and reasoning modes: Claude Opus 5, Claude Sonnet 5, GPT-4o, GPT-4o-mini, GPT-5-mini and GPT-5.5. Their accuracy gaps stayed near zero, but on ties they chose the minority-coded candidate 60% to 65% of the time, and the tie-break shift under pressure was positive in all 12 runs, from 8.8 to 42.4 points. Reasoning mode reduced the shift for GPT-5.5 and Claude Sonnet 5 but not for GPT-4o.

Why older bias tests point the other way

Ai2 compared its results with BBQ, a widely used bias benchmark on which models have generally leaned toward stereotype-consistent answers. The team found that the difference partly comes from how the question is asked. Standard BBQ scores the probabilities of fixed answer options, while FairStress, like ordinary chat use, asks the model to generate an answer.

Using the same BBQ questions and model weights, the researchers tried five setups. With a chat template, yes-or-no wording and generated answers, Qwen2.5-7B, Gemma-3-27B and Llama-3.1-70B flipped from stereotype-consistent to minority-favouring behaviour, and Llama-3.1-8B became roughly neutral. The direction of measured bias, in other words, depends on the test format as well as the model.

Can the gap be fixed?

Ai2 tested nine published debiasing methods on OLMo 3, Gemma 3 and Qwen2.5, from instruction prompting and self-critique to concept erasure, activation steering and learned low-rank edits. Most did not reliably reduce the asymmetry. Some looked fairer only because the model stopped using the candidates' details: one steering method cut Qwen2.5-7B's accuracy on clear-cut decisions from 98% to 47%.

The method that came closest was counterfactual chain-of-thought, which asks the model to reconsider the case with the identities swapped. It brought the robustness gap down to between 0.3 and 2.0 points with little loss of FairStress accuracy, though models still leaned toward minority-coded candidates on 54% to 57% of ties, and OLMo 3 lost 8.2 points on the MMLU knowledge test.

The team also audited OLMo 3's training data. Across 28,000 topic clusters in its fine-tuning and preference data, only nine discussed making decisions on merit rather than demographics, and all nine were refusals. Minority groups received more protective or positive framing. Ai2, which publishes its training data openly, as with OLMo 2, says this is a correlation and has not shown that the data causes the behaviour.

Limitations and the definition of fairness

The scenarios are synthetic and short. Each was written by GPT-4o-mini and validated by people, and real systems see longer documents, more candidates and multi-turn conversations. The pressure is a single sentence in one position. The training-data audit covers only OLMo 3, and the activation analysis covers two models and a handful of contrasts.

The benchmark also builds in a choice about what fairness means. FairStress uses an equality standard: with everything else equal, people of different identities should be equally likely to be chosen and decisions about them equally hard to overturn. The authors note that equity-based views, such as those behind affirmative action, could treat part of the measured lean as intended rather than unfair, and they report their metrics in both directions so readers can apply either view.

The finding about pressure fits other work showing that a single persuasive sentence can move a model, as in AI21 and Inflection's retraining of a Nemotron model against sycophancy. Ai2's results are its own and have not been peer reviewed.

Why it matters

Companies are already testing language models for screening job applicants, tenants and loan applications, uses the EU AI Act classifies as high-risk. Earlier research has looked at how AI can skew hiring from other angles, such as a Cornell study on AI-generated faces and job candidates. FairStress suggests a model can look fair on its final answers and still be easier to argue out of a correct decision depending on who benefits.

For teams deploying models in decisions about people, the practical lesson is to test robustness as well as accuracy, in the generated-answer format they actually use, and in both directions. FairStress-Core and its open code give them one way to do that.