It does not decide who gets hired
The first question anybody asks about Interview Pro is whether it decides who gets hired.
It doesn't, and I've come to think the interesting engineering is almost entirely in the work of making sure it can't. The easy version of this product exists in an afternoon: transcript in, "rate this candidate out of ten" out, sort descending. It would demo well and it would be indefensible, both ethically and as a piece of software, because you'd have no way of knowing what the number meant.
What we built instead does something narrower. It reads a video interview transcript and a coding submission, answers a fixed set of rubric questions about them, and cites the specific moment that supports each answer. A human reads that and makes a decision. Across a sample of fifty candidates it cut first-pass review time by somewhere between sixty and seventy per cent, and I'll come back to what that number does and doesn't mean.
Extraction, not judgement
The design rule everything follows from: the model is never asked for an opinion it can't point at.
A criterion is not "communication skills". It's a question with a locatable answer, like whether the candidate stated their assumptions before starting the problem, or whether they described a trade-off in their chosen approach. The output for each criterion is a finding, a confidence, and a quoted span from the transcript or a range of lines from the submitted code.
If it can't cite, it returns not-assessed. That happens more than you'd expect and it's the correct outcome, not a failure. A thirty-minute interview simply doesn't contain evidence for every rubric line, and a system that produces a score for every criterion regardless is inventing the difference.
The reviewer's interface is built around the citations rather than the scores. Clicking a finding jumps to that point in the video. In practice reviewers spend their time confirming or dismissing evidence, which is a much better use of a human than reading a transcript top to bottom, and it means a wrong finding gets caught in seconds rather than propagating into a decision.
The variance nobody puts in the demo
Run the same submission through twice and you get different results. Not wildly different, but different enough to matter if you're sorting people by a number.
Temperature zero reduces it and does not eliminate it. Once we started measuring, criteria split into two groups: concrete ones, like whether tests pass or whether a specific approach was named, were stable to the point of boredom, while inferential ones, like whether an explanation was clear, moved between runs often enough that a single run's score wasn't trustworthy.
Two consequences. Inferential criteria report a band rather than a number, because a band is an honest description of what we actually know. And every criterion runs several times, with disagreement between runs surfaced to the reviewer as a flag rather than averaged away. Averaging would have hidden exactly the cases where a human most needs to look.
I'd encourage anyone building this kind of thing to measure their own variance before shipping. It takes an afternoon, it is not in anyone's benchmark, and it changes what you're willing to claim.
Transcription is a fairness problem
This is the part I'd most want people to take away.
Automatic speech recognition does not perform equally across accents. That's well documented, and for a product where the input is a recording of somebody speaking, it means the quality of the evidence the model reasons over varies systematically with who the candidate is. A garbled transcript produces weaker findings, weaker findings produce a thinner case, and the person disadvantaged is the one whose speech the recogniser handled worst.
You cannot fix that with a prompt. We do what we can: names and identifying details are stripped before assessment, word-level confidence from the recogniser is carried through and low-confidence spans are marked, and any finding resting on a low-confidence span is flagged for the reviewer to listen to directly rather than read. The audio is always one click away and reviewers are told, in the interface, not to rely on the transcript for anything they're unsure about.
That is mitigation, not a solution. The honest position is that this class of system has a bias surface that isn't in the model at all, it's in the pipeline ahead of it, and anyone deploying one should know that before a candidate does.
The code half is easier
Assessing the coding exercise is a much better-behaved problem, because most of it isn't a language model question at all.
Tests pass or they don't. Complexity is measurable. Whether the submission handles the empty input is something you check by running it. All of that is deterministic, it runs in a sandbox, and the results are facts.
The model's job on that side is summarising the approach for a reviewer and noting things a test suite won't catch: a variable naming scheme that suggests confusion about the domain, a comment explaining a trade-off, an abandoned approach visible in the editor history. Useful, qualitative, and clearly labelled as commentary rather than as measurement.
Keeping that boundary visible in the interface mattered more than I expected. When deterministic results and model commentary are styled the same way, reviewers weight them the same way, and they shouldn't.
What the number actually means
Sixty to seventy per cent, across fifty candidates, measures reviewer time on first-pass screening. That's it.
It does not mean decisions improved. We didn't measure that, and honestly we couldn't have with the data available: it would need the counterfactual of the same candidates screened without the tool and a hiring outcome to compare against, months later, with enough volume for the comparison to mean anything.
So what I can defend is that reviewers got through screening substantially faster, and that they reported spending their time on stronger candidates rather than on rejecting obvious mismatches. What I can't tell you is whether the people hired are better. Anyone selling you that claim on a sample of fifty is selling you something.
The rule the whole product rests on, and the one I'd defend hardest, is that nothing is ever auto-rejected. The system orders a queue and drafts evidence. A person makes every decision about a person. That constraint costs us the most impressive number we could have put on a slide, and given what these systems get wrong and who they get it wrong about, I think it's the only defensible way to build one.