Paper Detail

WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning

Sang-Hoon Lee, Ha-Yeong Choi

huggingface Score 8.4

Published 2026-11-09 · First seen 2026-11-11

General AI

Abstract

This work proposes WaveGen, a fully end-to-end diffusion-based text-to-waveform framework based on Spectral Forcing, which jointly learns an internal spectral trajectory and the target waveform trajectory within a single model, enabling direct end-to-end text-to-waveform generation. Unlike recent TTS pipelines, WaveGen does not rely on separately trained or externally decoded intermediate representations, such as neural audio codecs, Mel-spectrogram vocoders, or VAE-based latent models. In addition, WaveGen performs internal text-speech alignment within a single model, eliminating the need for external alignment modules such as duration predictors. To preserve a truly end-to-end formulation, our framework further avoids dependence on self-supervised representations, such as BERT-based text embeddings and Wav2Vec 2.0-based semantic representations, as well as semantic distillation methods for training optimization. To achieve this, we carefully design Spectral Forcing, a model architecture with disentangled diffusion heads, and a training objective based on a multi-scale spectral v-loss. We further validate the scalability of end-to-end TTS models by scaling the model size from 0.1B to 0.6B parameters. Using a single architecture, open-source data, and one-stage training, without external modules such as duration predictor or REPA, WaveGen demonstrates fully end-to-end waveform generation achieving state-of-the-art performance on the Emilia training dataset. In particular, WaveGen achieves a WER of 1.47 and a SIM of 0.75 on the Seed-en benchmark. Source code and checkpoints are available at https://github.com/sh-lee-prml/WaveGen

Workflow Status

Review status
pending
Role
unreviewed
Read priority
soon
Vote
Not set.
Saved
no
Collections
Not filed yet.
Next action
Not filled yet.

Reading Brief

No structured notes yet. Add `summary_sections`, `why_relevant`, `claim_impact`, or `next_action` in `papers.jsonl` to enrich this view.

Why It Surfaced

No ranking explanation is available yet.

Tags

No tags.

BibTeX

@misc{lee2026wavegen,
  title = {WaveGen: Truly End-to-End Waveform Generation via Internal Spectral Trajectory Learning},
  author = {Sang-Hoon Lee and Ha-Yeong Choi},
  year = {2026},
  abstract = {This work proposes WaveGen, a fully end-to-end diffusion-based text-to-waveform framework based on Spectral Forcing, which jointly learns an internal spectral trajectory and the target waveform trajectory within a single model, enabling direct end-to-end text-to-waveform generation. Unlike recent TTS pipelines, WaveGen does not rely on separately trained or externally decoded intermediate representations, such as neural audio codecs, Mel-spectrogram vocoders, or VAE-based latent models. In addit},
  url = {https://huggingface.co/papers/2611.12527},
  keywords = {diffusion-based text-to-waveform, Spectral Forcing, end-to-end text-to-waveform generation, TTS pipelines, neural audio codecs, Mel-spectrogram vocoders, VAE-based latent models, internal text-speech alignment, duration predictors, self-supervised representations, BERT-based text embeddings, Wav2Vec 2.0-based semantic representations, semantic distillation, disentangled diffusion heads, multi-scale spectral v-loss, REPA, code available, huggingface daily},
  eprint = {2611.12527},
  archiveprefix = {arXiv},
}

Metadata

{}