The Return on People

Videos · AI in HR · 10 min

AI hiring bias: why a human check is not enough.

Four sentences that make AI in hiring feel safe, tested against two studies, three real cases and the EU AI Act. Then five checks you can run this month.

The video loads from YouTube only when you press play. What that means for your data · Watch on YouTube

Chapters

  1. 0:00Four sentences that feel safe
  2. 0:39Sentence one: “The tool only looks at skills”
  3. 1:45Sentence two: “Our vendor tested it”
  4. 3:43Sentence three: “That’s the vendor’s problem”
  5. 5:08Sentence four: “A human checks every decision”
  6. 7:30What the law now expects
  7. 8:12Five checks you can run this month
  8. 9:18Your four sentences, rewritten

Try this month

Five checks for any AI hiring tool

  1. Ask your vendor for selection results by gender, age and ethnicity, on candidates like yours, and who ran the test.
  2. Each quarter, where the law allows you to hold the data, compare pass rates per group at every stage.
  3. Decide first, then look: reviewers rate a sample before they see the AI score. Track how often they disagree.
  4. Remove what the tool does not need, like birth dates and photos. Schools and places can carry the same clues.
  5. Tell candidates and employees where AI is used, and offer a real human review.
Template4 questions for your HR tech vendorPrint it, or paste it into your next vendor email

The honest limit

The experiment at the centre of the video was online, with a simulated AI, and the people taking part were not professional recruiters. The Workday case is still open: no court has found discrimination. This is orientation, not legal advice.

Go deeper

Read the full transcriptTaken from the video's subtitles

0:00 Four sentences that feel safe

If your company uses AI to sort job applications, you've probably heard these four sentences. "The tool only looks at skills." "Our vendor tested it." "If it goes wrong, that's the vendor's problem." And the one everybody leans on: "A human checks every decision." Each one sounds reasonable. Each one runs into a study or a real case. Take the last one. Researchers at the University of Washington tested that safety net. The people in the study didn't catch the AI's bias. They copied it. In this video: two studies, three real cases, what the law now expects, and five checks you can run this month.

0:39 Sentence one: "The tool only looks at skills"

In October 2018, Reuters reported that Amazon had scrapped an experimental recruiting tool. The tool had learned to mark down CVs that contained the word "women's", as in "women's chess club captain". It also downgraded graduates of two all women's colleges. Nobody wrote that rule. The tool learned from CVs sent to the company in the past, and most of those came from men. Amazon said the tool was never used by its recruiters to evaluate candidates. An AI tool doesn't know what a good employee is. It only knows what your past decisions looked like. If the past was tilted, the tool reads the tilt as a sign of quality. It's a mirror mistaken for a judge. Nobody told it a candidate's gender. It found clues instead, like a club name or the college someone went to. So deleting the gender field doesn't fix it. So sentence one becomes a question: what did this tool learn from? And that was 2018. In 2024, researchers tested newer models.

1:45 Sentence two: "Our vendor tested it"

Kyra Wilson and Aylin Caliskan took more than 550 real CVs and more than 500 real job ads across nine occupations. They used three AI models which turn a CV and a job ad into numbers and measure how close the two are. Then they changed one thing: the first name at the top. 120 names, linked to white or Black men and women. More than three million comparisons. For each model and each job they asked: among the best matches, does one group show up more often? That's 27 tests. White associated names came out ahead in 85 percent of those tests. Black associated names, in 9 percent. When Black men's names went up against white men's names, the white names won every single test. And Black women's names beat Black men's in 67 percent of tests, the reverse in 15. Now the why. They also tried Black and white names that are about equally common in the online text these models learn from. The pattern flipped: Black names came out ahead in about half the tests. So the model wasn't judging the candidate. It was reacting to how familiar a name looked. Two honest limits. These were general models in a research setup, not one vendor's product. So it doesn't prove your vendor is biased, only that you can't assume it isn't. And in August 2026 the authors corrected the paper: a bug in their code had flipped the results for gender alone, so we've left those out. The race results were rechecked by a team at Salzburg University, and held. And scale matters. One biased recruiter affects the people they see. A biased system tilts every application the same way, and no single decision looks wrong. You only see it when you count. So sentence two becomes: tested on what, by whom, and can we see results for each group?

3:43 Sentence three: "That's the vendor's problem"

Here's where this becomes your problem. In September 2023, the EEOC, the US agency that enforces laws against job discrimination, settled a lawsuit with a tutoring company, iTutorGroup. According to the EEOC, the company's application software was programmed to automatically reject women aged 55 or older and men aged 60 or older. More than 200 qualified applicants in the US were turned away. The company agreed to pay 365,000 dollars, and it is now barred from asking applicants for their birth dates. No clever model. A setting. Someone typed in a rule, nobody stopped it, and the employer paid. The next case is still open. Mr Mobley, a job applicant over 40, says he applied to more than 100 jobs at companies using Workday's hiring software, and was rejected every time. In May 2025, a federal judge in California gave preliminary approval for his age claim to go forward as a nationwide collective action: a group case for applicants aged 40 and over. Workday says the claims lack merit. No court has found discrimination. But look at who's being sued. Not the employers. The software provider. So the safe assumption isn't "the vendor carries it". It's "we might both carry it". Which leaves the last sentence.

5:08 Sentence four: "A human checks every decision"

Let's try something. Two candidates. Same experience, same skills. Your screening tool has put a small green tick next to one of them. Take a second. Which one would you call first? Most of us would say: on merit. The tick doesn't decide for me. Hold on to that answer. In 2025, the same research group ran an experiment with 528 people in the US. They screened candidates for 16 jobs. Each round: five CVs, pick three to interview. Four were equally qualified men: two with names that signalled white, two with names that signalled Black, Hispanic or Asian. The fifth was a decoy who didn't fit the job. Sometimes there was no AI, sometimes a simulated AI made recommendations, and the researchers set its bias: neutral, a realistic level based on the 2024 study, or severe, where it recommended only one group. Without AI, or with the neutral AI, people picked white and non white candidates at equal rates. With the realistic bias, their choices mirrored the AI's. Whichever way the AI leaned, toward white candidates or away from them, people leaned the same way. With the severe bias, people pushed back only a little. In the strongest case, their picks still followed the AI around 90 percent of the time. Even people who rated the AI's advice as low quality or unimportant were still swayed in some conditions. As lead author Kyra Wilson put it: "Unless bias is obvious, people were perfectly willing to accept the AI's biases." The study didn't test why. But psychology has a name for this: automation bias, relying on what a system tells us more than we should. The first suggestion becomes the anchor we judge everything else against. Now picture a normal week: forty CVs, deadline Friday, a score already on the screen. That's when the green tick is hardest to ignore. The honest limit: this was an online experiment with a simulated AI, and the participants weren't professional recruiters. But the authors' conclusion is direct: human in the loop setups, as they're used today, can't reliably undo AI bias.

7:30 What the law now expects

Europe has written this risk into law. Under the EU AI Act, AI used to recruit, select, promote, dismiss, allocate tasks or monitor workers is high risk. After an amendment this July, those rules apply from 2 December 2027: a delay, not a cancellation. And the Act is precise about the human. The people overseeing the system must be able to stay aware of automation bias, that exact term, and to override it. Employers must give that job to people with the competence, training and authority to do it, and tell workers before the system is used. The law assumes the safety net can fail, and asks you to build it properly.

8:12 Five checks you can run this month

So what can you do on Monday? Five checks. One. Send your vendor one email: "Please show us selection results by gender, age and ethnicity, on candidates like ours, and tell us who ran the test." A vague answer is also an answer. Two. Every quarter, where the law allows you to hold that data, compare pass rates per group at each stage: applied, screened, interviewed, hired. Better you find a gap before a candidate's lawyer does. Three. Decide first, then look. Have reviewers rate a sample of candidates before they see the AI's score, and track how often they disagree. If nobody ever disagrees, you don't have a check. You have a rubber stamp. Four. Remove what the tool doesn't need, like birth dates and photos. And remember the clues: schools and locations can carry the same information. Five. Tell candidates and employees where AI is used, and offer a real human review when they ask. None of these removes bias on its own. Together, they make it visible.

9:18 Your four sentences, rewritten

Here are the four sentences, rewritten. "The tool learns from our past, so we check our past." "Our vendor showed us results for each group." "If it goes wrong, it's our problem too." And: "A human checks every decision, decides first, and is allowed to disagree." Next time someone says "it's fine, a human checks it", ask one question. When did that human last disagree with the tool? So AI isn't the problem on its own. The problem is a system that learns your past, repeats it at scale, and quietly steers the person meant to catch it. For the practical side of working with AI without wearing your team out, the free guide "AI at Work Without the Burnout" is on the return on people dot com. So is the free Team Check, to see where your own team stands.

Sources

After the video

One study like this a week, by email, plus the free guide AI at work, without the burnout.

Never more than one email a week. Unsubscribe in one click. Privacy