The Limits of Machine Learning in Recruiting: Garbage In, Garbage Out?

1200 900 Paul Breloff
Hardly a day goes by without someone asking: “So Shortlist is basically using artificial intelligence to screen CVs faster?” My response: “Well, not really…”

There’s no doubt that AI, neural nets, natural language processing, and machine learning are having a moment. They’re the shiny new toys in Silicon Valley right now, and with good reason — the capabilities are powerful, and if the vision is realized, the world will change, big-time. And no doubt there are a number of companies looking at ways to apply AI, machine learning and related concepts to recruiting.

It’s true that we incorporate sophisticated data analysis into our screening. We collect hundreds of data points from each candidate, and we’re starting to engineer even more features or variables that can predict who gets hired, who performs well, and who sticks around. These data feedback loops can get even more powerful as we do more and more hiring with a single employer, learning what they like and who does well at their company.

That said, we see glaring limitations to applying machine learning and AI to screen candidates based on their CVs and social media profiles alone — summed up in the old computer science adage of “garbage in garbage out.” GIGO is the concept that any output can only be as good as the inputs, and ultimately, any predictive algorithm is only as good as the data going in and the outcomes you’re predicting. In this case, most fancy new machine learning-based solutions in the recruiting space are using CV data as the primary input, and “similarity to other candidates/employees” or to a job description as the outcome to be predicted.

But this is myopic: CVs are shown to have only modest predictive relevance to performance in a job when considered on their own. Of course prior experience can matter, but CVs alone fail to capture individual performance and contribution, raw talent, actual competence, motivation, and a host of other factors that are important in assessing quality.

There’s also the pesky reality that they’re often embellished and written to be picked up by keyword-driven screening engines, which distorts analysis and results. Think back to your own hiring experiences; how many times has that candidate who looked so great on paper disappointed in reality? And have you ever had great colleagues who didn’t go to great schools or work at fancy corporates but who have shined in the real world? So don’t ignore a CV, but don’t rely on it exclusively!

And there are limits on the availability of “outcome data” — i.e., how do people actually perform once on the job? Ultimately, you need good “training data” and time to build a good algorithm, which means you need a data set with outcomes you care about — i.e., what was the person’s productivity on the job, did they stick around, were they a great culture fit? Unfortunately, that data is rarely available when training an algorithm in recruiting contexts.

Unless you’re running predictions about actual performance, actual retention, and actual outcomes that matter — then you run a great risk that you’re just pattern matching the status quo, which may entrench the same hiring mistakes, the same biases, the same lack of diversity that we already see.

But if CV data is all we’ve got, what’s a recruiter to do? Well, we can start to generate new data and new signals. Our technology automates the collection of dozens of new, user-generated data points — raw data about experience and salary expectation, performance data drawn from cognitive and competency tests, and the meta-data about how a candidate goes through the process which can be mined for motivation, speed, curiosity. As we use this treasure trove of new data to supplement traditional CV data, we become even more excited about the promise of machine learning approaches to make sense of it, yielding better matches for companies and candidates alike.

So, it’s an exciting time for AI (particularly as my brilliant brother joins one the coolest AI companies out there — congrats Tom!), but I don’t think it will be the standalone silver bullet in recruiting for some time to come. Humans are just too darn complicated.

Can we shift the recruiting paradigm from pedigree to potential?

1200 900 Paul Breloff
All around the world, companies big and small are facing a similar problem: Hiring is so much harder than it should be.

While India adds a million people to its job market every month and Africa is set to add more people to its workforce by 2020 than the rest of the world combined, over half of emerging market companies still can’t fill the roles they have open. Startups consistently rank talent acquisition as a top barrier to growth. What gives?

I saw this dilemma firsthand while investing in financial technology startups around the world for the last five years, as the founder of seed venture fund Accion Venture Lab. Once an investment was closed and cash was in the bank, the company’s problem shifted from not having financial capital to not having the human capital they needed to be successful.

The picture is even bleaker on the jobseeker’s side. Even skilled professionals often can’t get hired because they didn’t go to the “right” school, didn’t work at the “right” company, don’t know the “right” people, or fall victim to unfair biases during the application process. They are left lobbing their CV into job board black holes, never able to show potential employers what they can do.

This needs to change. We believe that talent is equally distributed, but opportunity is not. What often appears to be a lack of talent supply in markets is more often a failure of not knowing where to look or what to look for. At Shortlist, we want to level the job search playing field, shifting the recruiting paradigm from one based on pedigree and prejudice to a new version grounded in competency and potential.

How are we doing it?

1. Bringing intelligence to technology

Technology has burst on the scene to flatten access to job opportunities and broaden candidate pools (thank you LinkedIn and Monster). But without intelligent intermediation, more tech creates more noise, more decision fatigue, more work, and more despair for companies and jobseekers — not better outcomes. Just ask any of our employers who have received 2,000+ applications to a single job posting.

We combine a chatbot questionnaire with online assessments and phone screens to help us decide who is most likely to be great in a job. This filtering layer combines technology, data, and a human touch to ensure that talented candidates don’t slip through the cracks, particularly those who risk being overlooked based on CV alone.

2. Creating signals beyond the CV

Most companies have been hiring the same way for centuries (seriously): source and skim a lot of CVs, speak with some of the candidates, then make a decision — and regret those decisions more often than they would like. Not only is it hard to discern genuine ability and fit through a CV and unstructured interview alone, but this mode of decision-making is also often riddled with bias and prejudice.

Companies often do this not because they think it’s best, but because, frankly, there’s nothing else to go on. It’s like the joke about the economist looking for his keys under a streetlamp, not because that’s where he lost his keys, but because that’s where the light is better. At Shortlist, we engage candidates digitally to user-generate more accurate signals. We screen not only for basic experience fit but layer on additional data points for cognitive ability, competencies, and motivation. To be Shortlisted for a job, it’s more important to show us what you can do, not just tell us what you’ve done.

3. Refocusing on what matters

Let’s be clear: many people who went to great schools and worked at impressive companies are great and impressive. But for the vast majority of job-seekers, particularly in emerging markets like India and Kenya (where we work), prior experience paints an incomplete and often misleading picture of a candidate’s capabilities.

Schooling and subsequent corporate experience is — in all countries — more often determined by “birth lottery” than by merit. And we all hold biases, positive or negative, about certain schools or corporate brands. Looking past pedigree and refocusing on potential is the first step towards a world where everyone gets a shot at fulfilling professional experiences. Further, reconceiving the nature and focus of talent screening matters not only for hiring fairness, but also for hiring effectiveness. Building a team based on merit and performance instead of connections and pedigree is not only the right thing to do — it’s good for the bottom line.

The Shortlist mission

At Shortlist, we are on a mission to unlock professional potential and help great companies succeed in building great teams. We’re starting with a new way to match talent with opportunity, but we’re just getting started.

We want to level the talent playing field, but we can’t do it alone! We want to learn from each of you about what you think works to find and understand great talent, and what makes a great team. Visit our website, email us, or tweet at us — we’d love to talk with you about how we can help you hire. We’ll be using this blog as one of the ways we share the ideas behind what we do and how we do it, so stay tuned…

unstructured interviews

Unstructured Interviews: Less Predictive Than We Think

1080 651 Simon Desjardins

Interviewing effectively is surprisingly tough. It’s a process that sometimes seems straightforward and yet often leaves us feeling like we haven’t quite gotten the clarity we were hoping for out of an hour-long session, or that we’ve simply made a gut decision. The main reason for this feeling lies in fact that many people use unstructured interviews, which makes it much harder to reach a conclusion about whether the candidate would actually perform well or not.

In this series, we will explore some central challenges we face in the interview process, and highlight best practices and tools to make immediate improvements.

The statistics about interviews are both counterintuitive and somewhat alarming: the chances that unstructured interviews will accurately predict a candidate’s performance is less than 25 percent. You’d actually be better off flipping a coin, and would have saved several hours in the process!

Anatomy of unstructured interviews

  • Candidates are asked different questions by each interviewer.
  • Questions are not necessarily linked to key competencies required to do the job.
  • Candidates are not assessed using a standardized rating scale.
  • Interviewers haven’t aligned on acceptable answers beforehand.

With dozens of studies across multiple geographies and timelines showing them to be one of the worst ways to predict on-the-job performance, the evidence against using unstructured interviews is overwhelming. Yet we continue to rely on them almost exclusively to make hiring decisions. What’s going on?

Limiting the effects of bias

Bias plays a huge role in how we rate candidates in an interview setting. Humans are naturally and strongly predispositioned to favour people who are like them, which is a hazard when our objective is to build diversity on our teams.

Compounding this problem is our susceptibility to “first impression” bias, where we consistently end up asking easier questions of people who form a strong first impression, and harder questions for those who don’t. In their now-famous study, Tricia Pricket and Neha Gada-Jain showed how snap judgements made in the first 10 seconds of an interview could predict the outcome of that interview.

Mere awareness of these biases does little to counteract them, even for experienced interviewers. Asking candidates to complete a competency-based assessment process before the candidate reaches the interview stage adds a layer of objectivity to the screening process and can help put interview results in context.

Linking interview questions to competency requirements

Unlike unstructured interviews, great interviews start with the interviewers aligning on what competencies and other requirements will actually drive performance in a given role. It may sound obvious, yet we often see a majority of an interview being spent asking questions without a clear competency in mind.

Brain teasers, for example, are a perennial favourite, though they have been shown (most conclusively by Google’s HR team) to have no correlation to performance. By a similar token, academic scores (and by extension — questions about them) have equally little bearing as predictors of performance, unless we’re hiring people immediately after they graduate.

Asking questions that actually force a candidate to reveal a key trait can be risky without preparation and thought. For instance, if we wanted to understand a candidate’s “motivation to join,” we might be tempted to ask the basic question, “Why do you want to join our company?” This question, however, is easily answered by a clever candidate who has done their research. A less obvious but more revealing question, such as “What preparation did you do in the time between when this interview was scheduled and today?” might give us a much more meaningful data point on the same subject.

Interviews and the Stanford prison experiment

Back in 1971, Stanford University psychologist Philip Zimbardo led a set of experiments that changed what we know about human behaviour. Zimbardo arbitrarily assigned participants to play the role of “prisoner” and “guard” in a role play exercise, and the inhumane behavior of the “guards” revealed the extreme psychological effects of perceived power. The results were replicated many times across multiple countries and cultures.

Interview rooms aren’t prison cells, of course, though much of what was learned in those studies applies to this context. Given that employers have a job to offer, we sometimes assume a position of authority in relation to the candidate, and act accordingly, without realising it. Imagine, however, that we’re interviewing a top performer who has multiple employment options. In this situation, we’re being interviewed as much as we’re interviewing. Leaving five minutes at the end to answer a candidate’s questions won’t be enough to properly address concerns and communicate why the candidate should join us.

Interviews driving improved performance

Admittedly, interviews tend to be high stakes environments for both parties. We’re under pressure to make the perfect hire. The candidate is nervous. Skepticism, hope, and bias are at risk of permeating every exchange. If you leave time to “sell” the candidate at the end, we’re down to 40 minutes at most to ask five to eight questions. Given the exponential impact that high performers have on organisations, these 40 minutes are crucial.

Putting more thought and structure to that time will separate you from the vast majority of other hiring companies, including your competitors how are using unstructured interviews.

If unstructured interviews don’t work, then what’s the answer? In the next article in this series, we outline the basics of the “structured interview” and when combined with competency-based assessments, will save time and significantly improve the outcomes of the interview process.

About Shortlist Advisory

Shortlist Insights helps companies build capacity to improve how they recruit and manage talent. We combine best practices from industry experts, research, and our experience to deliver practical and tested solutions and thought leadership. Ultimately, we help our clients build a competitive people advantage.

Related: Everyone should be using structured interviews — here’s why