Updated October 13, 2026. We first published this article on July 24, 2026, with nine models and 300 raters. It now reports the English study after three rounds: 17 TTS models and the original human recordings, 15,217 comparisons and 901 rater sessions. See what changed.
Today, Askable Labs is releasing VOICE-H, a new human evaluation benchmark for text-to-speech (TTS) models. Evaluating voice quality is challenging: judgments are inherently subjective, and aggregate preference scores rarely reveal what actually drives them. VOICE-H is grounded in real human speech at both ends. It supplies both the prompt and the baseline sample that every model is judged against. This approach coupled with capturing the qualitative nuances behind preference decisions is essential to turning benchmark results into actionable insights. VOICE-H goes beyond measuring which voices people prefer to identify the specific qualities that make speech sound natural, engaging, and human.
We selected 100 quotes and corresponding audio clips from an Askable Labs dataset of over 40,000 real user interviews spanning topics that naturally elicited emotional expression. Participants then compared TTS-generated versions of each quote and rated them for naturalness, accuracy, emotion/tone, and overall quality. We also asked them to explain the reasoning behind their ratings.
Analysing the quantitative responses in conjunction with the qualitative data we can uncover several factors that consistently shaped preferences. Raters strongly valued how well a voice's emotional delivery matched the content. They also rewarded samples where the naturalness felt appropriate rather than performative, and where hesitations and imperfections sounded intentional instead of synthetic. Interestingly, eight of the 17 TTS models score above the original human recordings: raters preferred the crisp quality and audio clarity over the variability of real-world speech, including background noise, disfluencies, and unfamiliar accents.
These findings show that advancing TTS models requires more than improving accuracy and speed, and that models must also sound emotionally appropriate, natural, and authentic. Understanding the qualitative judgments that shape human preferences is therefore essential to building better voice models.
What's new in this update
Since July we have added eight models in two rounds. Round two, in September, added Inworld TTS-2, Sonic 3.6, Flux, Simba 3.2 and Speechmatics TTS. Round three, in October, added Google's Gemini 3.8 Flash TTS, Alibaba's Qwen Audio 3.1 Flash and ElevenLabs' Eleven v4. VOICE-H also runs in French and Spanish, each with 50 prompts and 14 TTS models against a native human recording. Across the three languages it now holds 26,251 comparisons from 1,701 rater sessions.
| New model | English / 17 | Spanish / 14 | French / 14 |
|---|---|---|---|
| Eleven v4 | 1 | 1 | 1 |
| Gemini 3.8 Flash TTS | 2 | 2 | 4 |
| Qwen Audio 3.1 Flash | 5 | 7 | 12 |
Rank among TTS models by overall preference, excluding the human recording. Elo values from separate language fits are not on a shared scale, and nearby ranks are not a reliable ordering.
Eleven v4 has the highest score in all three languages. In English its leads over Gemini 3.8 Flash TTS, Sonic 3.6 and Gemini 3.1 Flash Voice are within the uncertainty (see Results). In French it leads clearly: 97.1 points ahead of Eleven v3, with a 95% interval of 62.5 to 135.7 when we resample whole raters.
“the voice of A [Eleven v4] is way more natural, less robotic”
Original: la voix du A est beaucoup plus naturel, moins robotique
Eleven v4 vs Eleven v3 · French · prompt 8 · listen
Gemini 3.8 Flash TTS places second in English and Spanish and fourth in French. It scores above Gemini 3.1 Flash Voice in each language, but none of those differences is clear of zero. Timing is where listeners hear room to improve: in English its hesitations and disfluencies drew 40 negative tags in 1,631 appearances, against 14 in 1,630 for Eleven v4.
“Model A [Gemini 3.8 Flash TTS] makes stops that are too long and emphasized at commas. Model B [Aura 2] has a certain fluidity in its voice and fewer unexpected stops.”
Original: Le modele A fait des arrets trop longs et accentues au niveau des virgules . Le modele B a une certaine fluidite dans sa voix et moins d'arrets inopines.
Gemini 3.8 Flash TTS vs Aura 2 · French · prompt 41 · listen
Qwen Audio 3.1 Flash is fifth in English, seventh in Spanish and twelfth in French, and at about $0.001 per English prompt it is the cheapest hosted model in the study. French listeners were much harsher on it: negative voice-character tags run at 22.3 per 100 appearances in French against 9.3 in English, and accuracy complaints at 15.1 against 2.2.
“Clip A [Human Clip] is low quality, there's a lot of background noise, audio is stuttering constantly, however, the voice sound natural and pronunciation is on point. Clip B [Qwen Audio 3.1 Flash] has a good quality audio but the voice sounds artificial and very flat. It also mispronounced "5e" and "donc".”
Human Clip vs Qwen Audio 3.1 Flash · French · prompt 11 · listen
The rest of this article covers the English study. French and Spanish results, including head-to-head records, are on the VOICE-H leaderboard.
Methodology
Prompt curation
The prompts and human audio were sourced from a dataset of over 40,000 Askable user interviews lasting ~30 minutes each. For the English study we focused on user interviews conducted in English; the French and Spanish studies each use 50 prompts with human recordings from separate native-language datasets. We curated samples of at least 10 seconds in length and at most three minutes of contiguous speech. From these samples, we filtered for clips that contained emotional responses, such as stories of frustration or excitement. We then picked the top 100 of these options from unique participants. Lastly, we manually reviewed each prompt to ensure quality and accuracy of the transcription.
Participant population
We performed this evaluation using 901 rater sessions from the Askable Labs panel, completed by 725 distinct people: some raters returned in later rounds. The panel covered four countries, 28 industry categories and a range of language backgrounds.
Model choice
We chose 17 models from across the state of the art and smaller models for comparison, including:
- Google: Gemini 3.8 Flash TTS (round three)
- Google: Gemini 3.1 Flash Voice
- ElevenLabs: Eleven v4 (round three)
- ElevenLabs: Eleven v3
- ElevenLabs: Eleven v2
- Alibaba: Qwen Audio 3.1 Flash (round three)
- Cartesia: Sonic 3.6 (round two)
- Cartesia: Sonic 3.5
- Deepgram: Flux (round two)
- Deepgram: Aura 2
- Inworld: Inworld TTS-2 (round two)
- Speechify: Simba 3.2 (round two)
- Speechmatics: Speechmatics TTS (round two)
- Mistral: Voxtral Mini
- OpenAI: GPT-4o Mini TTS
- Kyutai: Pocket TTS
- Smallest: Lightning V3.1
These models were chosen to cover a range of sizes and performance levels to ensure the evaluation spans a meaningful spread.
Comparisons
Using 17 models plus the human sample gives us 153 unique comparisons per prompt and a total of 15,300 comparisons across the entire dataset. We fielded them in three rounds as models were added: the original 45 pairs per prompt in round one, the 60 pairs involving the round-two models, and the 48 pairs involving the round-three models. After rater-quality filtering, 15,217 comparisons remain. Each rater session covered a single prompt, with 15 random comparisons in round one, 20 in round two and 16 in round three. We also randomise which side each model is shown on to prevent position bias. Even so, the right-hand sample won 53.9% of decisive votes; the fits do not adjust for this. The raters were asked to rate each model in the following categories:
- Naturalness (How natural the voice sounds)
- Accuracy (How accurate is the pronunciation to the text)
- Emotion and Tone (How does the emotion and tone of the voice match the context of the text)
Each rater was then asked to rate their overall preference on the following scale:
- Strongly Prefer A
- Prefer A
- Tie
- Prefer B
- Strongly Prefer B
The rater was then asked to provide qualitative information about their category and preference ratings.
Elo score
We use these ratings to compute an Elo score for both the individual ratings and overall preferences in each category. We use ordinal Bradley-Terry preference with weighting, taking into account the strength of the preference between each model. All three rounds are pooled into one fit, so every model is on the same scale. Pooling assumes that earlier comparisons still apply; round three repeated no earlier pairs, so it cannot test for drift directly.
Uncertainty
Chart intervals are 95% intervals from 500 bootstrap draws over comparisons. Because each rater and each prompt contributes many votes, we also resample whole raters and whole prompts, separately. Those checks matter at the top of the table. Eleven v4's 2.0-point lead over Gemini 3.8 Flash TTS spans −24.3 to 31.7 when we resample raters and −34.1 to 35.5 when we resample prompts, and its leads over Sonic 3.6 (3.5 points) and Gemini 3.1 Flash Voice (20.0 points) also span zero in both checks.
Download all model scores · Head-to-head results · Theme counts · Clustered gap checks
Results
Scores
Four models share the lead. Eleven v4 has the highest overall score (1,109.1), followed closely by Gemini 3.8 Flash TTS (1,107.1), Sonic 3.6 (1,105.5) and Gemini 3.1 Flash Voice (1,089.0). None of Eleven v4's leads over the other three is clear of the uncertainty, so we do not name a single winner.
All four outperform the human voice samples on overall preference and on each of naturalness, accuracy and emotion/tone, with no overlap between their 95% intervals and those of the human recordings.
The human recordings rank ninth. Eight TTS models have higher overall scores, and the top seven are clear of the human recordings' interval. The human clips score well on naturalness (1,025.5) but third-lowest of the 18 systems on accuracy (905.2), and that weak accuracy pulls their overall preference down.
Looking at the correlation between category ratings and preference, we have derived a weighting of the relative importance of each category. In the chart below, we can see that emotion and tone carries the most weight, at 46%, against 35% for naturalness and 19% for accuracy. In comparisons against the human recordings, accuracy counts for more (27%) and naturalness for less (28%), but emotion and tone still leads at 45%. Accuracy otherwise ranks last, likely because most TTS models are already quite accurate.
Another key aspect of TTS models is cost and price per performance. For the most part, the models follow a fairly linear trend between cost and performance, with a few outliers. Qwen Audio 3.1 Flash ranks fifth at about $0.001 per prompt and Simba 3.2 sixth at $0.004, both far cheaper than the models around them. Gemini 3.8 Flash TTS sits level with the leaders at $0.011 per prompt, though that is a promotional rate that doubles in January 2027. Eleven v4 has the top score but costs $0.052 per prompt, 4.6 times as much as Gemini 3.8 Flash TTS, and Eleven v3 and Eleven v2 remain the most expensive at $0.065 per prompt without being top performers. GPT-4o Mini TTS also slightly underperforms for its cost. We also note that Kyutai's model is open-source and able to be hosted on a very light machine, so the cost has been set to zero.
Findings
In addition to the results presented above, we conducted a qualitative analysis of the 14,723 free-text responses from our raters, and present the following five key findings:
The emotional affect has to match the content
As shown above, emotion and tone carries the most weight in overall preference. Flat, monotone delivery was the most common emotional complaint, with 1,128 negative mentions against 500 for the wrong emotion and 452 for over-emoting; however, it is not enough for the model to just “be emotional”, the emotion also needs to match the content of the prompt. This was a consistent theme in some of the quotes from our raters:
“B [Lightning V3.1] sounds like someone reading from a script and unfamiliar with the context. A [Gemini 3.1 Flash] sounds like someone living the experience.”
Gemini 3.1 Flash vs Lightning V3.1 · prompt 33
“This [Gemini 3.1 Flash] sounded the most human, by far. The cadence and emotion was what I would have expected to align with the content being described.”
Gemini 3.1 Flash vs Eleven v3 · prompt 7
One interesting phenomenon is that sometimes the models' inferred emotionality was preferred over the real human response. Several raters identified that the human sample sounded like the voice was reading a response rather than really speaking it. In these instances, raters preferred the models' assumed emotional response. When raters criticised the human recording's emotional delivery, it lost 69 of 87 comparisons.
“I prefer sample B [Aura 2] over sample A [Human Clip] because sample A [Human Clip] sounds really bad. It sounds like someone is obviously reading something and like they are confused or irritated by what they are reading. The emotion and tone don't match at all and they say "um" multiple times. Sample B [Aura 2] sounds more natural and the emotion and tone is fitting, but it does sound like someone is exaggerating or trying to sell the listener on something.”
Human Clip vs Aura 2 · prompt 95
This is partly because the original dataset comes from AI-moderated interviews, where participants are often speaking about emotional things that previously occurred in a matter-of-fact tone. We plan to correct this in future versions by using human clips elicited from more natural conversations rather than interviews.
Voices can be “too” natural
Naturalness had the second highest weight in preference, but it can be a double-edged sword. Participants clearly preferred more natural models: sounding robotic was the single biggest complaint in the whole dataset with 2,066 negative mentions across nearly every model, with Speechmatics TTS, Lightning V3.1 and Eleven v2 drawing the most voice-character complaints per appearance. Gemini 3.8 Flash TTS and Eleven v4 drew the fewest of any TTS model: 2.9 and 3.6 per 100 appearances, against 24.8 for Speechmatics TTS. However, models can also overcorrect in the opposite direction: voices that sounded excessively emotional or performative drew complaints of their own. For example, raters complained that Sonic 3.5 sounded too “sexy/breathy”. Sonic 3.5 appeared in 80 of the 179 comparisons whose comments used words like “breathy”, “sexy” or “sultry”, nearly three times as many as the next model, and raters were split on whether that delivery was pleasant or inappropriate for the content.
“B [Lightning V3.1] sounded super robotic. Completely emotionless and deadpan.”
Aura 2 vs Lightning V3.1 · prompt 87
“here is the raunchy novel again with A [Sonic 3.5], terible! felt like she was reading erotica. B [Voxtral Mini] was great”
Sonic 3.5 vs Voxtral Mini · prompt 44
“I just find BotA [Sonic 3.5] to have a good sex appeal that i know is effective in real world settings”
Sonic 3.5 vs Pocket TTS · prompt 69
Eleven v4 shows a milder version of the same risk. Of the comments on its 420 English losses, 34 call it dramatic or over the top and 6 call it flat; for Gemini 3.8 Flash TTS the split is 13 and 13 across 466 losses. That is a keyword count, so treat it as approximate.
“The tone of A [Lightning V3.1] was good and the level of emotion matched the piece being read. B [Eleven v4] was too dramatic.”
Lightning V3.1 vs Eleven v4 · prompt 9 · listen
Our results suggest that naturalness is the largest area for improvement, but it's important to tread carefully in order to reduce the risk of overemphasising traits that read as inappropriate or performative rather than genuinely natural.
Accuracy alone doesn't win preference, but it can lose it
Raters often forgive accuracy issues if the voice is more natural and emotive, however wrong or missing words are much more likely to lead to an automatic dispreference. Mispronunciations were excused almost half the time in our tagged comments (274 of 609 mentions), while wrong or missing words led to negative results in roughly 4 out of 5 mentions (255 of 326). The human clips were forgiven the most, with raters excusing garbled words and awkward grammar because the voice sounded real.
“B [Human Clip] grammar may not be good but sounded more natural”
Lightning V3.1 vs Human Clip · prompt 31
“A [Gemini 3.1 Flash] had little errors, B [Pocket TTS] had many errors but the voice would be good if those were ironed out.”
Gemini 3.1 Flash vs Pocket TTS · prompt 80
Disfluencies need to be executed well
Raters often mention preferring samples that have disfluencies (ums, ahs and hesitations); however, poorly executed disfluencies were a clear negative, with 515 negative mentions. In particular, Lightning V3.1's pauses and hesitations drew 82 negative mentions and only 2 positive ones, and it lost 58 of those comparisons. The best-executed TTS disfluencies now outscore the human recordings: Sonic 3.6 drew 55 positive mentions to 19 negative and Eleven v4 35 to 14, against 101 to 98 for the human clips. Whether the source text contained fillers made little difference for TTS models: raters liked 38% of the TTS disfluencies they mentioned when the text had fillers, and 40% when it did not. Even genuine human hesitation was punished when it landed badly: when raters criticised the human recording's disfluencies, it lost 70 of 98 comparisons.
“The ums especially were really bad and more like someone was being told to say um, rather than saying it naturally as a pause.”
Human Clip vs Lightning V3.1 · prompt 9
Recording quality cost the humans votes
About two in five recording quality complaints (180 of 458) were about the human clips, roughly double the rate of the next worst model: 11.4 mentions per 100 appearances against 5.8 for Voxtral Mini. Quiet or muffled audio was the main offender, followed by crackle, background noise (crying babies, traffic) and echo, and when flagged the human clip lost the vote around three-quarters of the time (138 of 180). However, if the human voice was significantly more natural, it could still sway the preference decision in favor of the human clip:
“Despite bad audio (like a bad phone line) this [Human Clip] sounded exactly like a real person - the voice, flow, emotion. Best of the lot by far”
Lightning V3.1 vs Human Clip · prompt 6
This is another artifact of sourcing audio from real interview recordings, and something the higher quality samples in future versions should fix.
What's next?
We believe that TTS models have a lot of headroom in languages other than English. French and Spanish results are now on the leaderboard, and German is next.
In response to our findings on audio quality, we are also collecting more naturalistic, emotionally expressive conversations to strengthen the human baseline and expand the prompt set.
Conclusions
VOICE-H shows that voice quality can't be reduced to a single measure of realism: the strongest models are not simply the ones that sound the most human or the most polished, but the ones that adapt their delivery to the context, balancing emotion, accuracy and restraint. The leading group of Eleven v4, Gemini 3.8 Flash TTS, Sonic 3.6 and Gemini 3.1 Flash Voice shows this most clearly: all four outperform the original human recordings by producing speech that raters found more emotionally appropriate and consistently reliable.
More broadly, VOICE-H offers a richer way to evaluate TTS performance. By grounding the prompt set in real human speech and collecting comprehensive quantitative and qualitative data, we can start to identify not only which models people prefer, but why. The remaining challenge for frontier voice models is thus not to simply sound less robotic: it is to produce speech whose emotion, rhythm, and level of polish feel right for the words being spoken.
Explore the full interactive results: head-to-head records, preference distributions, and more →