AI music, voice and sound, from the sources

Analysis

Zero-shot TTS and gender expansive voices: human ratings and metrics disagree

A paper on six zero-shot TTS models finds mixed human results for gender expansive voices and conflicting automated metrics.

Cover: LinaSpeech, d=.704, largest effect size where GE voices did worse, for LinaSpeech

In a paper on queer and transgender voices, the authors report that human ratings found 2 of the 6 zero-shot TTS models synthesize less similar gender expansive (GE) voices than non gender expansive (N-GE) voices, while 3 have improved performance for GE voices (arXiv). Automated metrics gave conflicting results, according to the authors. For practitioners who rely on automatic scores to pick or validate a voice-cloning model, the second research question, on whether evaluation algorithms are biased compared with listener-based measures, is the one that bears on that choice (arXiv).

What the study tested

The authors evaluate six zero-shot TTS models on two datasets. One dataset holds non gender expansive voices (N-GE), and the other holds gender expansive (GE) voices from self-identified speakers. Zero-shot systems clone a voice from a short reference sample, so the way each reference is built shapes everything downstream.

According to the paper, a reference sample was created for all 14 MAGES speakers and 14 GLOBE speakers. Here the MAGES speakers are the GE group and the GLOBE speakers are the N-GE group. The comparison therefore rests on two speaker groups of equal size. Groups of that size make each individual speaker count for a lot, which is a reason to read the model-level contrasts below as indications for these speakers rather than as universal rules.

Human ratings split by model

The subjective metrics varied across models: two models showed worse performance for GE speakers, and three models showed better performance. The largest effect size where GE voices did worse was for LinaSpeech (d=.704), followed by XTTS (d=.419). Those two models are the ones that account for the worse-for-GE result in the human ratings. The split matters because a single average over all six systems would blur a real disadvantage for GE voices in some models with an advantage in others.

The word error rate (WER) test points the same way for one model. Significant WER differences appeared only for LinaSpeech, with higher WER for GE outputs. F5TTS came near significance at p=.051. Intelligibility and perceived similarity therefore agree for LinaSpeech, which makes it the clearest case of poorer service to GE voices in this paper.

Automated metrics disagree

For the ZONOS model, the three speaker similarity metrics disagree on which dataset performs better. ECAPA-TDNN predicts improved performance for GE voices, and TitaNet-L predicts improved performance for N-GE voices. ReDimNet-M found no difference between the datasets. A team that picked one embedding model to score ZONOS would reach a different conclusion depending on the choice.

UTMOS, a predictor of quality scores, predicted significantly higher AMOS scores for N-GE voices for all models. That result runs against the human ratings, where three models did better for GE voices. A quality predictor that favors N-GE voices across every model would hide the models whose human-rated results favor GE voices.

What it means for evaluation

The findings cover six zero-shot models and the speakers in the two datasets, so they describe those systems and those groups. The authors report that automated metrics gave conflicting results, so no single automated score can be treated as a stand-in for listener ratings. The practical reading is that evaluating a zero-shot TTS model on GE voices needs listener ratings alongside speaker-embedding and quality-predictor scores, and that model-by-model results matter more than a family-wide verdict.

What the sources we read do not state

We check six facts on every story in the full text of each source. The pages listed under Sources do not state a price.

Sources

The sources this article was written from. Links open the original page.

  1. arXivNo-Shot Text-to-Speech: Limitations of Zero-Shot TTS and its Evaluation Methods in Representing Queer and Transgender Voicesarxiv.orgPrimary source · accessed November 19, 2026
  2. arXivNo-Shot Text-to-Speech: Limitations of Zero-Shot TTS and its Evaluation Methods in Representing Queer and Transgender Voicesarxiv.orgPrimary source · accessed November 19, 2026
  • tts
  • zero shot tts
  • evaluation
  • voice cloning
  • bias