Background

We believe machine learning affords a novel approach to music making and listening. The generative nature of these algorithms allows musicians to not only capture the output of their musical processes, but the process itself. This creates a whole new kind of intellectual property, “the model”. Models can be trained on a specific instrument or ensemble (such as saxophone or gamelan) but can also capture playing styles and articulation which is unique to an individual, place, time or culture.

Being interactive and generative, these models then enable other musicians to create new music and material from the dataset, collaborating in unconventional ways and producing new mashups and hybrids. Our approach to machine learning is not to create systems which supplant musicians, but put these models into musician’s hands and make the process creative and fully attributed.

Datasets

We care deeply about musicians and know that a lot of energy and musicianship goes into creating these recordings. We only use open-source datasets, recordings with permissive licenses, or make our own datasets.

It is important to us that we tell the story of our models. They came from a specific time, place and person.

The model resynthesizes any input audio using the textures and timbres learned from a dataset of instrumental or vocal recordings, all while retaining the pitches and rhythms of the original. The output is imperfect and idiosyncratic, and produces unexpected (and sometimes bizarre) combinations of the model’s training dataset and the input performance.

Environment

We also care deeply about the environmental impact of machine learning. A large model trained on cloud GPUs can emit as much CO2 as 300 flights from SF to NY [source]. Our models, on the other hand, are comparably tiny. They run on a single in-house GPU and consume about as much power as a toaster to train and run.

Our real-time hardware models are so small that they can run on mini, embedded GPUs.

Custom Models

If you are looking for a custom model and have a dataset, contact us! We are always looking to train personal and unique musical models.