Toptal develops a multimodal AI assessment pipeline that combines speech recognition, large language models, and human-in-the-loop validation to automate high-confidence English proficiency assessments.
English proficiency screening for job applicants relied entirely on manual review, requiring human evaluators to assess every recorded interview. The process was difficult to scale and inherently subjective, while variation between human assessors made it challenging to establish consistent criteria for model training. Toptal needed an AI-powered solution capable of delivering reliable assessments at scale without compromising screening quality.
Designed a modular assessment pipeline that decomposed English proficiency evaluation into separate video, audio, and transcript analysis stages. The system combined linguistic features extracted from transcripts, like vocabulary use, with acoustic features such as pronunciation and pauses, as well as visual cues like eye gaze, to identify potential scripted responses. These signals together produced more accurate assessments than a single input modality.
Generated 258 linguistic, acoustic, and pronunciation features from candidate interviews using Whisper, Praat, librosa, and openSMILE. The pipeline extracted fluency, vocabulary, prosody, and language-detection signals, enriching each transcript with confidence scores, hesitation markers, pause annotations, and filler word counts before passing structured inputs to the assessment models.
Processed each candidate interview using four Whisper models (tiny, base, small, and medium) to strengthen transcription reliability. The system treated disagreement between model transcriptions as a signal of potentially unclear pronunciation, while agreement increased confidence in the transcription before it was passed to the English assessment model.
Combined six specialized LLM evaluators, each focused on assessing a different aspect of English competence, such as grammatical accuracy, business communication, pronunciation clarity, or overall proficiency. Individual assessments were aggregated using an ensemble model to produce a final English proficiency score, along with separate scores for fluency, grammar, vocabulary, and pronunciation.
Engineered a vendor-agnostic model architecture capable of integrating and replacing commercial and open-source AI models without modifying the surrounding pipeline. The team evaluated multiple ASR models, including OpenAI Whisper, alongside LLMs from OpenAI, Anthropic, and Google to identify the highest-performing combination for transcription and candidate evaluation.
The solution enabled AI to assess the majority of English proficiency screenings while maintaining performance comparable to that of human reviewers.
In 93% of assessments, the system matched human screening decisions within one scoring band. The platform reduced screener workload by up to 70%, with human reviewers reserved for borderline cases.
By combining multimodal speech analysis with LLM-based assessment, the platform established a reusable and scalable foundation for automating additional talent-screening workflows and enabling reliable same-day candidate evaluation.
