Today, Askable Labs is releasing VOICE-H, a new human evaluation benchmark for text-to-speech (TTS) models. Evaluating voice quality is challenging: judgments are inherently subjective, and aggregate preference scores rarely reveal what actually drives them. VOICE-H is grounded in real human speech at both ends. It supplies both the prompt and the baseline sample that every model is judged against. This approach coupled with capturing the qualitative nuances behind preference decisions is essential to turning benchmark results into actionable insights. VOICE-H goes beyond measuring which voices people prefer to identify the specific qualities that make speech sound natural, engaging, and human.
We selected 100 quotes and corresponding audio clips from an Askable Labs dataset of over 40,000 real user interviews spanning topics that naturally elicited emotional expression. Participants then compared TTS-generated versions of each quote and rated them for naturalness, accuracy, emotion/tone, and overall quality. We also asked them to explain the reasoning behind their ratings.
Analysing the quantitative responses in conjunction with the qualitative data we can uncover several factors that consistently shaped preferences. Raters strongly valued how well a voice's emotional delivery matched the content. They also rewarded samples where the naturalness felt appropriate rather than performative, and where hesitations and imperfections sounded intentional instead of synthetic. Interestingly, we also found that raters preferred some TTS models to the original human recordings, preferring the crisp quality and audio clarity over the variability of real-world speech, including background noise, disfluencies, and unfamiliar accents.
These findings show that advancing TTS models requires more than improving accuracy and speed, and that models must also sound emotionally appropriate, natural, and authentic. Understanding the qualitative judgments that shape human preferences is therefore essential to building better voice models.
Methodology
Prompt curation
The prompts and human audio were sourced from a dataset of over 40,000 Askable user interviews lasting ~30 minutes each. For this first version, we focused on user interviews conducted in English, but plan to expand to more languages in subsequent versions. We curated samples of at least 10 seconds in length and at most three minutes of contiguous speech. From these samples, we filtered for clips that contained emotional responses, such as stories of frustration or excitement. We then picked the top 100 of these options from unique participants. Lastly, we manually reviewed each prompt to ensure quality and accuracy of the transcription.
Participant population
We performed this evaluation using a population of 300 raters from the Askable Labs panel. The panel covered four countries, 12 industry categories and a range of language backgrounds.
Model choice
We chose 9 models from across the state of the art and smaller models for comparison, including:
- Google: Gemini 3.1 Flash Voice
- Cartesia: Sonic 3.5
- Deepgram: Aura 2
- Mistral: Voxtral Mini
- Eleven Labs: Eleven v3
- Eleven Labs: Eleven v2
- OpenAI: GPT-4o Mini TTS
- Kyutai: Pocket TTS
- Smallest: Lightning V3.1
These models were chosen to cover a range of sizes and performance levels to ensure the evaluation spans a meaningful spread.
Comparisons
Using 9 models plus the human sample gives us 45 unique comparisons per prompt and a total of 4,500 comparisons across the entire dataset. A small number of these (26) were removed by rater-quality filtering. Each rater was shown 15 random comparisons for a single prompt. We also randomise which side each model is shown on to prevent position bias. The raters were asked to rate each model in the following categories:
- Naturalness (How natural the voice sounds)
- Accuracy (How accurate is the pronunciation to the text)
- Emotion and Tone (How does the emotion and tone of the voice match the context of the text)
Each rater was then asked to rate their overall preference on the following scale:
- Strongly Prefer A
- Prefer A
- Tie
- Prefer B
- Strongly Prefer B
The rater was then asked to provide qualitative information about their category and preference ratings.
Elo score
We use these ratings to compute an Elo score for both the individual ratings and overall preferences in each category. We use ordinal Bradley-Terry preference with weighting, taking into account the strength of the preference between each model.
Results
Scores
Gemini 3.1 Flash Voice is the clear winner. It exceeds the pack significantly on all quality metrics as well as preference in Elo score. In particular, it has a large lead in emotion and tone.
It also significantly outperforms human voice samples in accuracy and emotion/tone, and slightly outperforms them in naturalness (though not statistically significantly).
Sonic 3.5 comes in third, within error of human quality, in terms of preference. However, it loses out to humans in its naturalness and emotion scores, but the subpar performance of the human clip on accuracy drives the overall preference of the human clip down.
Looking at the correlation between category ratings and preference, we have derived a weighting of the relative importance of each category. In the chart below, we can see that emotion and tone outrank both accuracy and naturalness particularly when in reference to human responses where tone is even more emphasised. This is likely because most TTS models are already quite accurate.
Another key aspect of TTS models is cost and price per performance. For the most part, the rest of the models follow a fairly linear trend between cost and performance. However, we can see that Gemini is not only the best overall but is even particularly good for its cost. The two other outliers are both the Eleven Labs models which are significantly more expensive without being top performers. GPT-4o also slightly underperforms for its cost. We also note that Kyutai's model is open-source and able to be hosted on a very light machine, so the cost has been set to zero.
Findings
In addition to the results presented above, we conducted a qualitative analysis of the free-text responses of our raters, and present the following five key findings:
The emotional affect has to match the content
As shown above, the emotion / tone score had one of the highest correlations with overall preference. Flat, monotone delivery was actually the most common complaint; however, it is not enough for the model to just “be emotional”, the emotion also needs to match the content of the prompt. This was a consistent theme in some of the quotes from our raters:
“B [Lightning V3.1] sounds like someone reading from a script and unfamiliar with the context. A [Gemini 3.1 Flash] sounds like someone living the experience.”
Gemini 3.1 Flash vs Lightning V3.1 · prompt 33
“This [Gemini 3.1 Flash] sounded the most human, by far. The cadence and emotion was what I would have expected to align with the content being described.”
Gemini 3.1 Flash vs Eleven v3 · prompt 7
One interesting phenomenon is that sometimes the models' inferred emotionality was preferred over the real human response. Several raters identified that the human sample sounded like the voice was reading a response rather than really speaking it. In these instances, raters preferred the models' assumed emotional response.
“I prefer sample B [Aura 2] over sample A [Human Clip] because sample A [Human Clip] sounds really bad. It sounds like someone is obviously reading something and like they are confused or irritated by what they are reading. The emotion and tone don't match at all and they say "um" multiple times. Sample B [Aura 2] sounds more natural and the emotion and tone is fitting, but it does sound like someone is exaggerating or trying to sell the listener on something.”
Human Clip vs Aura 2 · prompt 95
This is partly because the original dataset comes from AI-moderated interviews, where participants are often speaking about emotional things that previously occurred in a matter-of-fact tone. We plan to correct this in future versions by using human clips elicited from more natural conversations rather than interviews.
Voices can be “too” natural
Naturalness had the second highest correlation to preference, but it can be a double-edged sword. Participants clearly preferred more natural models: sounding robotic was the single biggest complaint in the whole dataset with 590 negative mentions across nearly every model, with Lightning V3.1 and Eleven v2 flagged at the highest rates. However, models can also overcorrect in the opposite direction: voices that sounded excessively emotional or performative drew complaints of their own. For example, raters complained that Sonic 3.5 sounded too “sexy/breathy”. Sonic accounted for over half of these breathy comments, with raters split almost evenly on whether it was pleasant or inappropriate for the content.
“B [Lightning V3.1] sounded super robotic. Completely emotionless and deadpan.”
Aura 2 vs Lightning V3.1 · prompt 87
“here is the raunchy novel again with A [Sonic 3.5], terible! felt like she was reading erotica. B [Voxtral Mini] was great”
Sonic 3.5 vs Voxtral Mini · prompt 44
“I just find BotA [Sonic 3.5] to have a good sex appeal that i know is effective in real world settings”
Sonic 3.5 vs Pocket TTS · prompt 69
Our results suggest that naturalness is the largest area for improvement, but it's important to tread carefully in order to reduce the risk of overemphasising traits that read as inappropriate or performative rather than genuinely natural.
Accuracy alone doesn't win preference, but it can lose it
Raters often forgive accuracy issues if the voice is more natural and emotive, however wrong or missing words are much more likely to lead to an automatic dispreference. Mispronunciations were excused almost half the time in our tagged comments, while wrong or missing words led to negative results in 5 out of 6 mentions. The human clips were forgiven the most, with raters excusing garbled words and awkward grammar because the voice sounded real.
“B [Human Clip] grammar may not be good but sounded more natural”
Lightning V3.1 vs Human Clip · prompt 31
“A [Gemini 3.1 Flash] had little errors, B [Pocket TTS] had many errors but the voice would be good if those were ironed out.”
Gemini 3.1 Flash vs Pocket TTS · prompt 80
Disfluencies need to be executed well
Raters often mention preferring samples that have disfluencies (ums, ahs and hesitations); however, poorly executed disfluencies were a clear negative. In particular, Lightning V3.1's pauses and hesitations drew 52 negative mentions and no positive ones, contributing to 41 lost votes. More broadly, raters responded poorly when models awkwardly reproduced disfluencies written into the source text, but often liked disfluencies that models introduced spontaneously and naturally. Even genuine human hesitation was punished when it came across as rambling, losing 80% of the votes where raters explicitly called this out.
“The ums especially were really bad and more like someone was being told to say um, rather than saying it naturally as a pause.”
Human Clip vs Lightning V3.1 · prompt 9
Recording quality cost the humans votes
Almost half of all recording quality complaints were about the human clips, roughly double the rate of the next worst model. Quiet or muffled audio and background noise (crying babies, traffic, echo) were the main offenders, and when flagged the human clip lost the vote around 70% of the time. However, if the human voice was significantly more natural, it could still sway the preference decision in favor of the human clip:
“Despite bad audio (like a bad phone line) this [Human Clip] sounded exactly like a real person - the voice, flow, emotion. Best of the lot by far”
Lightning V3.1 vs Human Clip · prompt 6
This is another artifact of sourcing audio from real interview recordings, and something the higher quality samples in future versions should fix.
What's next?
We believe that TTS models have a lot of headroom in languages other than English. To start, we are currently expanding VOICE-H to 3 additional languages: German, French and Spanish.
In response to our findings on audio quality, we are also collecting more naturalistic, emotionally expressive conversations to strengthen the human baseline and expand the prompt set.
Conclusions
VOICE-H shows that voice quality can't be reduced to a single measure of realism: the strongest models are not simply the ones that sound the most human or the most polished, but the ones that adapt their delivery to the context, balancing emotion, accuracy and restraint. Gemini 3.1 Flash Voice was the clearest example of this, outperforming all other models and even the original human recordings by producing speech that raters found more emotionally appropriate and consistently reliable.
More broadly, VOICE-H offers a richer way to evaluate TTS performance. By grounding the prompt set in real human speech and collecting comprehensive quantitative and qualitative data, we can start to identify not only which models people prefer, but why. The remaining challenge for frontier voice models is thus not to simply sound less robotic: it is to produce speech whose emotion, rhythm, and level of polish feel right for the words being spoken.
Explore the full interactive results: head-to-head records, preference distributions, and more →