OV-InstructTTS: Towards Open-Vocabulary Instruct Text-to-Speech

Abstract

Instruct Text-to-Speech (InstructTTS) leverages natural language descriptions as style prompts to guide speech synthesis. However, existing InstructTTS methods mainly rely on a direct combination of audio-related labels or their diverse rephrasings, making it difficult to handle flexible, high-level instructions. Such rigid control is insufficient for users such as content creators who wish to steer generation with descriptive instructions. To address these constraints, we introduce OV-InstructTTS, a new paradigm for open-vocabulary InstructTTS. We propose a comprehensive solution comprising a newly curated dataset, OV-Speech, and a novel reasoning-driven framework. The OV-Speech dataset pairs speech with open-vocabulary instructions, each augmented with a reasoning process that connects high-level instructions to acoustic features. The reasoning-driven framework infers emotional, acoustic, and paralinguistic information from open-vocabulary instructions before synthesizing speech. Evaluations show that this reasoning-driven approach significantly improves instruction-following fidelity and speech expressiveness. We believe this work can inspire the next user-friendly InstructTTS systems with stronger generalization and real-world applicability.

Dataset and Framework

To enable open-vocabulary InstructTTS, we constructed the OV-Speech dataset. Built upon the 476.8-hour ContextSpeech corpus, our dataset construction follows a five-stage pipeline. The stages are: 1. Contextual Information Extraction from novels; 2. Open-Vocabulary Instruction Generation using an LLM to mimic a director's instructions; 3. Consistency Filtering to ensure alignment between instructions and audio; 4. Reasoning Process Annotation to bridge high-level instructions with low-level acoustic features; and 5. Paralinguistic-Aware Transcription Annotation to enrich the transcription with paralinguistic tags.

Our proposed framework, OV-InstructTTS-TEP, first generates a textual reasoning process from the open-vocabulary instruction, then produces an interleaved sequence of enriched text and audio tokens for synthesis. This reasoning-driven approach is key to interpreting complex instructions and generating expressive speech.

Demo