MIDI-Conditional Text-to-Audio Synthesis Using ControlNet on AudioLDM
Text-to-audio systems have gained attention in recent months, achieving impressive results in general audio synthesis. However, they often lack fine-grained control over the musical output, as note-level adjustments cannot be determined by text. In this work, we present MIDI-AudioLDM, which implemen...
| Autor: | |
|---|---|
| Tipo de recurso: | tesis de maestría |
| Fecha de publicación: | 2023 |
| País: | España |
| Institución: | Universidad Nacional de Educación a Distancia |
| Repositorio: | e-spacio. Repositorio Institucional de la UNED |
| Idioma: | inglés |
| OAI Identifier: | oai:e-spacio.uned.es:20.500.14468/23782 |
| Acceso en línea: | https://hdl.handle.net/20.500.14468/23782 |
| Access Level: | acceso abierto |
| Palabra clave: | 1203.17 Informática audio synthesis MIDI conditioning text-to-audio systems AudioLDM ControlNet |
| Sumario: | Text-to-audio systems have gained attention in recent months, achieving impressive results in general audio synthesis. However, they often lack fine-grained control over the musical output, as note-level adjustments cannot be determined by text. In this work, we present MIDI-AudioLDM, which implements MIDI conditioning into AudioLDM with the use of ControlNet. This enables MIDI-conditional text-to-audio synthesis, which adds up to AudioLDM’s previous capacities, including direct text-to-audio synthesis as well as audio style transfer and inpainting. Like AudioLDM, the model uses contrastive language-audio pretraining (CLAP) latents and is trained on audio embeddings, while using text embeddings for inference. In contrast to unconditional audio synthesis, MIDI-AudioLDM offers detailed control over various musical aspects such as notes, genre, mood, and timbre, which makes it a more valuable tool for the music production process. A demo is available at https://huggingface.co/spaces/lauraibnz/midi-audioldm. |
|---|