AI Interviews vs Human Interviewers: What the Evidence Says
AI interviews vs human interviewers: what the evidence says
Can an AI interviewer do the job of a trained human one? The honest answer from the research so far is “nearly, for some jobs, with some people”. This post walks through the five studies we lean on most, what each one does and doesn’t show, and how to read them when you are planning a real programme study. For the wider picture of where AI interviews fit and where they don’t, start with our guide to AI interviews for research.
The five studies side by side
| Study | Setting | What it found | What to keep in mind |
|---|---|---|---|
| [Chopra & Haaland](https://www.ifo.de/en/cesifo/publications/2026/working-paper/conducting-qualitative-interviews-ai) (CESifo WP 10666, first released 2023) | AI-led text interviews in several economics contexts | Thematically richer answers than other at-scale qualitative methods, mostly because of dynamic probing. Surfaced themes that would otherwise be missed. Answers predicted behaviour six months later. Reached thematic saturation in samples comparable to human-led studies. | Online, text-based participants |
| [Geiecke & Jaravel](https://cepr.org/publications/dp19705) (CEPR DP 19705, 2024) | Open-source AI interviewer; 466 respondents randomised between an AI interview and open text fields | About 460 words in the AI interview against 190 in text boxes. 76% would not have preferred the text box. For future interviews, 43% would prefer an AI, 19% a human, 38% didn't mind. Experts graded AI voice transcripts 3.4 against 3.6 for face-to-face human interviews (1-5 scale). | Online panels; small expert-graded sample (15-17 transcripts per type) |
| [Wuttke et al.](https://arxiv.org/html/2410.01824v1) (2024-25) | 11 interviews on Zoom (6 human-led, 5 AI-led) | Overall quality broadly comparable. The AI was better at active listening and worse at its follow-up rule, after a prompt change that pre-tests had not caught. Respondents found the AI interviews less interesting. | Very small student sample |
| [Barari et al., NORC](https://ojs.ub.uni-konstanz.de/srm/article/view/8624) (*Survey Research Methods*, 2026) | 1,800 web respondents with probing chatbots | More detailed, more informative open-ended answers. Live coding moderately accurate but slightly over-coded, which the authors link to respondents agreeing too readily. A slight cost to respondent experience. | Text chat, web panel |
| [NORC, earlier experiment](https://www.norc.org/research/library/generative-ai-can-enhance-survey-interviews.html) (2024) | 1,200 participants, probing vs no probing | Probing increased specificity and explanation. Mobile users saw no quality gain, a worse experience and higher attrition. | Text, not voice |
Three patterns across the studies
1. Probing does most of the work. The gain over a survey comes from follow-up questions, not from the AI being clever. If your study has no room for probing, you probably need a survey, not an interview.
2. Prompts are research instruments. The Wuttke team changed a prompt and the AI stopped following up properly. In a human study you would retrain an enumerator. With AI you have to pilot the guide again, every time you edit it.
3. The evidence is thin exactly where nonprofits need it most. None of these studies phoned first-generation smartphone users in Odia or Kannada. None tested voice AI on shared phones with a household listening in. J-PAL’s guidance on phone surveys still applies: keep calls under about 30 minutes, plan for low response rates (it suggests 50% as a rule of thumb for new surveys), and drop questions on mental health, sex or domestic violence where you can, because they are harder to ask by phone. An AI interviewer inherits every one of those constraints.
What “match” should mean for your study
When a programme team asks us whether AI interviews “match” human ones, we split the question into four:
- Depth of answers. Strongest evidence. AI interviews reliably beat open-text boxes and come close to human transcripts on expert grading.
- Accuracy of coding. Moderate. NORC found live coding moderately accurate but slightly over-coded. Treat AI themes as a first draft.
- Respondent experience. Mixed. Slightly worse in NORC’s work, less interesting in Wuttke’s, and worse on mobile in NORC’s 2024 test.
- Who you reach. Not tested in any of these studies for phone-based, low-literacy, multilingual samples. This is the gap you fill with your own pilot.
Worked example: reading the evidence for a dropout study
Illustrative scenario, not a client case. A livelihoods nonprofit wants to understand why trainees left jobs within six months. It has 600 alumni across two states, speaking Tamil and Kannada, most reachable only by basic phone.
- Depth: the studies support using AI here. The question is “why”, the sample is large, and probing is exactly where AI interviews outperform a survey.
- Coding: given NORC’s over-coding finding, the team plans a human coder to review every theme against at least a sample of audio, and to look hard at answers that look like agreement for its own sake.
- Experience: the NORC mobile result is a warning. The team keeps the interview to around 20 minutes, inside J-PAL’s 30-minute ceiling, and lets participants call back at a better time.
- Reach: no study answers this, so the pilot must. The team pilots 20-30 calls per language and reports, by district and gender, who answered and who didn’t.
- What stays human: 10-15 in-depth interviews with alumni who left after workplace harassment or family pressure. Those topics need a person who can slow down and offer support.
The result is a hybrid design: AI for breadth, humans for the hardest cases, and a pilot that tests the one thing the literature can’t. That is roughly the “hybrid” result our hub’s fit checker gives for this kind of study.
Where this leaves you
The research is good enough to take AI interviews seriously and not good enough to skip your own pilot. If your study is multilingual, the next question is transcription accuracy, which we cover in voice AI interviews in Indian languages. Field teams already running phone back-checks will recognise the logic from our guide to field data quality checks. And if you plan to analyse transcripts with a general-purpose assistant, read Claude for nonprofits first.
Bring your study design to a free 30-minute AI Clinic → We’ll tell you plainly whether AI interviews fit, and what to keep human.