Toptal fine-tunes 9B AI model to reduce talent screening errors by 50%, matching proprietary models 10x its size.

Toptal’s existing screening process relied on a two-stage human interview: an initial conversation assessing basic communication and high-level technical fluency, followed by a deeper technical interview. Screening was fully dependent on a human reviewer for every candidate, a potential bottleneck on how far and quickly Toptal can scale its talent network without adding headcount. Off-the-shelf generalist models from major AI labs are built to be broadly capable, not tuned to judge a candidate against Toptal’s specific screening criteria or grading standards. Building something custom was the only way to automate the process without missing qualified candidates or waving through unqualified ones. What made it possible was something most companies don’t have: years of expert-graded interview transcripts, with human-assigned pass/fail decisions and competency scores across dimensions like leadership, creativity, and technical skill, the kind of proprietary ground-truth data that third-party evaluation services have to generate from scratch by hiring their own graders.

Created a strategy based on small, open-weight base models (9 billion parameters, including Qwen) rather than larger proprietary alternatives. The hypothesis: domain-specific fine-tuning on Toptal’s proprietary screening data would let a much smaller model approach the capabilities of far larger generalist models. The team plans to scale toward 30 billion parameters as the research matures.

Evaluated a wide range of fine-tuning and reinforcement learning techniques against each other, including supervised fine-tuning (SFT), direct preference optimization (DPO), proximal policy optimization (PPO), group relative policy optimization (GRPO), parameter-efficient tuning methods LoRA and QLoRA, and on-policy and off-policy self-distillation, to determine which approach best adapts a generalist base model to Toptal-specific screening data.

Built the fine-tuning pipeline around Toptal’s Certified Data Repository, drawing on historical interview transcripts with expert-assigned pass/fail grades and 1-to-5 competency scores across skill dimensions, giving the models a training signal grounded in real hiring decisions rather than synthetic or generic labels.

Solved the distinct grading and conversational screening problems by splitting the evaluation process into two models. A grading model was evaluated against historical ground truth using balanced accuracy, F1, and precision. A separate conversational model, evaluated without ground truth, since there’s no single “correct” interview conversation, used an LLM-as-a-judge framework across metrics, including single-question rate, specificity, coherence, adaptiveness, sycophancy rate, and coverage of the skills listed in a candidate’s target job description.

Designed a preference-labeling interface where an interviewer would be shown two AI-generated candidate questions and choose which one to ask, generating labeled preference data to feed into future rounds of model refinement.

The fine-tuned 9-billion-parameter open-weight model (Qwen 3.5) now identifies more than twice as many qualified candidates as before fine-tuning. In its base state, the model correctly flagged qualified candidates only 29.4% of the time, overlooking roughly 71% of them. Balanced accuracy rose from 61.4% to 82%, putting the model on par with proprietary closed models more than ten times its size (Gemini 3.1 Pro and 3.5 Flash, at approximately 82%). That’s the accuracy bar the team needed to clear before removing a human reviewer from the grading step becomes viable at scale.

In the conversational task, early results show that supervised fine-tuning with self-distillation outperforms the base model on grounding, specificity, and hallucination metrics, though these results are preliminary. A pilot study comparing the system’s performance directly against human screeners is also soon to be deployed.