Audio Samples
🔥 CapSpeech is a new benchmark designed for style-captioned TTS (CapTTS) tasks, including style-captioned text-to-speech synthesis with sound effects (CapTTS-SE), accent-captioned TTS (AccCapTTS), emotion-captioned TTS (EmoCapTTS) and text-to-speech synthesis for chat agent (AgentTTS).
📍 CapSpeech comprises over 10 million machine-annotated audio-caption pairs and nearly 0.36 million human-annotated audio-caption pairs. 3 new speech datasets are specifically designed for the CapTTS-SE and AgentTTS tasks to enhance the benchmark’s coverage of real-world scenarios.
Links shown in darker buttons represent new captions or recordings recently released by us.
| Description | Audio | Caption |
|---|---|---|
| Pretraining set for CapTTS | Emilia-EN MLS-EN GigaSpeech CommonVoice-EN |
CapSpeech-PT
|
| Pretraining set for CapTTS-SE | CapSpeech-PT-SEDB |
CapSpeech-PT-SEDB
|
| High-quality pretraining subset for CapTTS-SE | CapSpeech-PT-SEDB-HQ |
CapSpeech-PT-SEDB-HQ
|
| SFT set for CapTTS | LibriTTS-R VCTK VoxCeleb VoxCeleb2 EARS Expresso |
CapTTS-SFT
|
| SFT set for EmoCapTTS | EARS Expresso |
CapTTS-SFT
|
| SFT set for AccCapTTS | VoxCeleb VoxCeleb2 VCTK |
CapTTS-SFT
|
| SFT set for CapTTS-SE | CapSpeech-SEDB |
CapSpeech-SEDB
|
| SFT set for AgentTTS | CapSpeech-AgentDB |
CapSpeech-AgentDB
|
Audio Samples
| Transcription | Caption | Audio Recording |
|---|---|---|
| So what happens is when you click on suggest page and you choose | A middle-aged woman delivers her speech with a subtle expressiveness, her voice deep and resonant due to its low pitch. Her pace is unhurried, lending an air of thoughtfulness and deliberation to her tone. | |
| This adventure of ours was destined to differ in one respect from others which i have narrated. | In a somber, unwavering tone, a middle-aged man delivers his words with a low pitch, his steady pace slightly accelerating, yet maintaining a sense of controlled cadence. | |
| But it really seems like they skipped a whole bunch of stuff and went straight to the trial.Notwithstanding, louis should probably be talking to the jury, if there is a jury present at this time, and not to the judge. | A young adult male speaks with dynamic inflection and a brisk, energetic pace. His slightly high pitch lends a sense of urgency and concentration to his delivery. | |
| It immediately triggered a deep sense of sadness and grieving. And my phone lit up with people who are sobbing, | In a room with minimal background noise and reverberation, an elderly man delivers his speech in a monotone manner, his slightly high-pitched voice resonating with a sense of calm and wisdom. His pace is measured, with a slight slowness that allows his words to echo with depth and significance. | |
| So you've got a number of options, so what it does, it works in this idea of creating a graph projection. | The speech emanates from a young adult woman, her voice, a melodious symphony of high-pitched exuberance. Her words flow rapidly, each syllable a testament to her animated spirit, imbued with a subtly expressive tone. |
Audio Samples
| Transcription | Caption | Audio Recording |
|---|---|---|
| montmorency made up his mind to defend on the whole coast of provence only marseilles <door_wood_knock> <I> </I> and arles he pulled down the ramparts of the other towns which were left exposed to the enemy | A young adult male delivers his speech with a monotone, measured tone, maintaining a consistent rhythm. His moderate pitch, carries a sense of calm assurance. | |
| <applause> <I> </I> i have neither head nor arm do not despise but help me | A middle-aged male delivers his speech with a moderately paced rhythm, his tone characterized by a strong monotone. His voice, with a moderate pitch, carries an understated yet deliberate tone, allowing the content of his words to take center stage. | |
| the man from <coughing> <B> the spacer camp </B> was using the stream as his road | In a slightly fast-paced flow, a young woman articulates her thoughts in a subdued, slightly low-pitched tone. Her speech is characterized by a monotone quality, imbuing it with an air of calm and measured deliberation. | |
| jane is now a common grown up with a daughter called margaret and every spring cleaning time except when he forgets peter comes for margaret and takes <writing> <B> her to the neverland where she tells him stories about himself to which he listens eagerly </B> | A middle-aged man delivers his speech with a slightly fast tempo and a slightly high-pitched vocal tone. His tone remains fairly monotone. |
Audio Samples
| Transcription | Caption | Audio Recording |
|---|---|---|
| Oh, how I long for the taste of victory! Ha ha ha! | An enthusiastic laughter filled with longing, as if craving for something delightful. | |
| Um, yeah, I think I saw them somewhere. It's just that I'm not sure if I understand all these technical terms. | Uncertain and hesitant voice, reflecting a state of bewilderment. | |
| I can't believe we stumbled upon this hidden gem of a roadside attraction! | Breathless panting, fueled by curiosity and intense interest. | |
| I despise every strand of hair on your wretched head, and I'll make sure you leave this salon in ruins! | Aggressive, seething delivery reflecting deep-rooted hatred. | |
| How can I assist you today with our delicious bakery treats? | Warm, supportive tone indicating a willingness to help. |
Audio Samples
| Transcription | Caption | Audio Recording |
|---|---|---|
| <clock_alarm> <I> </I> if he could only be there first and catch bob cratchit coming late | An elderly man, his voice subtly deep and steady, delivers his words in a slightly brisk pace, the monotone tone of his speech lending a sense of wisdom and gravitas, despite the swift delivery | |
| he offered his hand to the baroness who descending <clapping> <I> </I> took it with a peculiarity of manner imperceptible to every one but monte cristo | A young female's speech is characterized by an even, steady delivery. Her voice carries a moderate pitch, and her pace is a comfortable, moderate speed, creating an overall impression of calm and measured articulation. | |
| loose indentures and harsh laws put them at the mercy of their masters <keyboard_typing> <B> it would not be unfair </B> to add that such was their lot in all other colonies | A young adult male delivers his speech with a monotone, measured tone, maintaining a consistent rhythm. His moderate pitch, carries a sense of calm assurance. | |
| in the first chapter of his geography <door_wood_knock> <B> moses tells us there were only two people in the world </B> | In a moderately paced and subdued manner, a middle-aged man delivers his speech with a slightly low-pitched, monotone tone, presenting an understated yet deliberate air. |
All transcriptions and captions are unseen during training. CapSpeech-NAR performs better than CapSpeech-AR.
| Transcription | Caption | CapSpeech-AR | CapSpeech-NAR |
|---|---|---|---|
| Use "hir" or "ze" as gender neutral pronouns? | A male speaker delivers his words in a measured pace, exhibiting a high-pitched, happy, and animated tone in a clean environment. | ||
| From these genres and from these spaces, you know, and the feelings of what these games can bring. | An elderly woman, with a low-pitched voice, delivers her speech in a slow, yet expressive and animated manner. Her words flow like a captivating story, each sentence filled with emotion and wisdom, resonating deeply with her audience. | ||
| in germany they generally hock the kaiser | Her voice, a combination of feminine allure and intellectual brilliance, resonates with a sense of calm and elegance, making her every word a testament to her cool sophistication. | ||
| i want to see what he was when he was bright and young before the world had hardened him | A mature male voice, rough and husky, ideal for public speaking engagements. |
All transcriptions and captions are unseen during training. CapSpeech-AR performs better than CapSpeech-NAR.
| Transcription | Caption | CapSpeech-AR | CapSpeech-NAR |
|---|---|---|---|
| at this moment miss brandon entered with her brilliant cousin rachel the blonde and the dark it was a dazzling contrast <dog> <I> </I> | A young woman speaks at a moderate pace, her voice carrying a hint of monotone. Remarkably, she maintains a high pitch, giving her speech an air of focused determination. | ||
| it did not beckon or indeed move at all <coughing> <I> </I> it was as still as the hand of death | A middle-aged man delivers his words in a monotone manner, maintaining a steady, moderate pace. His slightly elevated pitch, while not overly dramatic, imparts a subtle sense of gravity to his speech. | ||
| <cat> <I> </I> he stood still in deference to their calls and parried their banter with easy words | The speech emanates from a middle-aged man, his voice resonating with a slight, low-pitch timbre. His delivery is steady, devoid of emotive inflections, yet not entirely monotonous. The moderate pace of his speech lends an air of thoughtful consideration to his words. | ||
| i know said margaret bolton with a half anxious smile <clapping> <B> the chafes against all the ways of friends </B> but what will thee do | An old female speaker, her voice subtly animated, adopts a slightly low pitch. Her speech pace is moderate. | ||
| <keyboard_typing> <B> at the inception of plural marriage among the latter day saints </B> there was no law national or state against its practise | A middle-aged man, his voice unwavering and steady, delivers his speech in a monotone manner. The moderate pitch, neither too high nor too low, adds a touch of gravitas to his words. The slight fast pace hints at a sense of urgency or importance, yet his tone remains consistently measured and controlled. |
All transcriptions and captions are unseen during training. CapSpeech-NAR performs better than CapSpeech-AR.
| Transcription | Caption | CapSpeech-AR | CapSpeech-NAR |
|---|---|---|---|
| This is a sector in overall deficit and urgent action is required. | An Indian-accented professional woman's voice for client and public interaction. | ||
| When the sunlight strikes raindrops in the air, they act as a prism and form a rainbow. | A mature, deep, and rough Scottish-accented female voice, with a hint of weakness, ideal for client and public interaction in customer service or community relations. | ||
| If we live forever, can we really be said to live? | Bright teenage girl's voice, with an American accent, ideal for client and public interaction. | ||
| It will decide today whether he should be struck off the register. | A young girl's English-accented voice, suitable for customer service and public engagement roles. |
All transcriptions and captions are unseen during training. CapSpeech-NAR performs better than CapSpeech-AR.
| Transcription | Caption | CapSpeech-AR | CapSpeech-NAR |
|---|---|---|---|
| The distant hum of traffic passed unnoticed in the background. | She speaks in a calm, measured tone, conveying a sense of quiet, unassuming presence. | ||
| Why does your car smell like a dead RAT? It's absolutely vile. | A middle-aged woman speaks in a low, monotone voice, her words dripping with disgust and annoyance. | ||
| The cat, usually aloof, suddenly curled up in his lap, purring contentedly. | A male speaker, speaking with a slightly high-pitched, slow, and slightly expressive tone, conveying surprise with a gentle, bemused warmth. | ||
| Dark smoke billowed from the engine as the car refused to start. | A middle-aged woman speaks in a low, slow, monotone tone, her words dripping with a palpable, simmering frustration. |
All transcriptions and captions are unseen during training. CapSpeech-NAR performs better than CapSpeech-AR.
| Transcription | Caption | CapSpeech-AR | CapSpeech-NAR |
|---|---|---|---|
| If only I had pursued my passion for dance earlier, I could have become a professional dancer. | A voice tinged with regret, conveying a sense of longing for what could have been. | ||
| Please, step lightly on the path ahead, respecting nature's delicate balance as we make our way forward. | Quiet, submissive intonation reflecting a sense of yielding and compliance. | ||
| The intricate patterns and vibrant colors of each quilt showcase the love and dedication poured into every stitch. | Sincere and soft-spoken voice filled with kindness and compassion. | ||
| Just reboot the robot and step aside, clearly, this is beyond your expertise. | Cool, dismissive tone conveying disdain. |
@misc{wang2025capspeechenablingdownstreamapplications,
title={CapSpeech: Enabling Downstream Applications in Style-Captioned Text-to-Speech},
author={Helin Wang and Jiarui Hai and Dading Chong and Karan Thakkar and Tiantian Feng and Dongchao Yang and Junhyeok Lee and Laureano Moro Velazquez and Jesus Villalba and Zengyi Qin and Shrikanth Narayanan and Mounya Elhiali and Najim Dehak},
year={2025},
eprint={2506.02863},
archivePrefix={arXiv},
primaryClass={eess.AS},
url={https://arxiv.org/abs/2506.02863},
}