Jordi Pons

Estimating pitch in polyphonic music

By Jordi Pons in Code, Music August 12, 2024

Any python libraries to estimate the main melody’s pitch from polyphonic music?

Basic Pitch is for Automatic Music Transcription (AMT), using lightweight neural networks.
Link: https://github.com/spotify/basic-pitch
MELODIA in Essentia. It estimates the fundamental frequency of the predominant melody from polyphonic music signals using the MELODIA algorithm.
Link: https://essentia.upf.edu/reference/std_PredominantPitchMelodia.html
Omnizart is for automatic music transcription, given polyphonic music it is able to transcribe pitched instruments, vocal melody, chords, drum events, and beat.
Link: https://pypi.org/project/omnizart/
Sheet Sage transcribes your favorite pop song into a lead sheet containing the melody and chords.
Link: https://github.com/chrisdonahue/sheetsage
CREPE is a monophonic pitch tracker based on a deep convolutional neural network operating directly on the time-domain waveform input.
Link: https://github.com/marl/crepe
SPICE is Self-supervised PItCh Estimation.
Link: Github Python Notebook
ReconVAT is a semi-supervised automatic music transcription model.
Link: https://github.com/KinWaiCheuk/ReconVAT
AudioFlux is a deep learning tool library for audio and music analysis, feature extraction.
Link: https://github.com/libAudioFlux/audioFlux

Trends at ICML 2024

By Jordi Pons in Conferences, Deep learning, Personal July 26, 2024

I had the pleasure of attending ICML in Vienna. I fell in love with the city’s artistic and musical vibe. We presented our paper on Stable Audio 1.0, both as an oral presentation and a poster.

Check our paper: Fast Timing-Conditioned Latent Audio Diffusion.

Preprint: “Stable Audio Open”

By Jordi Pons in Deep learning, Ethics, Model Release, Paper is out July 24, 2024

Stable Audio Open paper is finally out. Our aim is to further improve current best practices for open model releases, with an emphasis on evaluation, data transparency, and accessibility for artists and scholars. Stable Audio Open allows for high-quality stereo sound synthesis at 44.1kHz, compatible with consumer-grade GPUs. The model was trained solely on Creative Commons licensed audio. Model weights, code, and data attributions are publicly available.

Link to the arXiv and to the HuggingFace.

Presenting at +RAIN Film Festival and Sonar+D

By Jordi Pons in Conferences, Deep learning, Music June 13, 2024

At this year Sonar’s week, we presented Stable Audio 2.0 and Stable Audio Open. At the +Rain Film Festival, we presented the potential of Stability AI tools in film and cinema, showing how they can enhance current sound design and audiovisual production tools. During Sonar+D, we engaged with musicians and the general audience, presenting our technology for music creation, from composition to sound design.

Stable Audio Open model release

By Jordi Pons in Code, Deep learning, Model Release June 5, 2024

Stable Audio Open WEIGHTS are out in the HuggingFace! What’s in there? Text-to-audio diffusion, T5-based text conditioning, diffusion transformer architecture (DiT), 47s long stereo generations at 44.1kHz, pre-trained on sfx/samples from FreeSound and FMA, good for making samples for your music and to fine-tune it.

The paper is on the way, but here’s a sneak peek at the results. Stable Audio Open achieves the best FD results among competitors in generating sound effects. FD is used by researchers to benchmark the realism of the generations.