Variational Autoencoders for Polyphonic Music Interpolation

The Problem

In traditional Machine Learning, the generation of music is conditioned on past events. But what if we could condition the music generation on both past and future events? The problem of music interpolation composition is constructing new data points within the range of a discrete set of known data points.

If we input a begin track and an end track of 10 seconds each to our model, the goal is to obtain a middle (or interpolation) track of also 10 seconds as output, whose pitches and dynamics smoothly match both given tracks.

The Approach & Architecture

This thesis aims to use Machine Learning techniques to solve this novel problem using generative models. Rather than relying on standard linear sampling of the latent space (which often yields poor musical results), a novel architecture is proposed: Variational Autoencoder + Neural Network (VAE+NN).

• Dataset Creation: The Hsinchu Interpolation MIDI Dataset was created to make the model more efficient than previous approaches in the literature in terms of computational and time requirements during training.
• Latent Space Estimation: The core of the novel architecture performs interpolation via direct estimation of the encoded vector.

The process follows these steps:

1. Encode the begin track and end track with the VAE to obtain z_begin and z_end, respectively.
2. Use a secondary neural network (NN) to directly estimate the interpolation encoded vector z_interpolation based on z_begin and z_end. This predicts a realistic, curved trajectory through the latent space instead of a naive straight line.
3. Decode the interpolation encoded vector z_interpolation to obtain the final polyphonic interpolation track.

Outcome & What I Learned

The novel VAE+NN model significantly outperformed standard VAEs that rely on linear sampling of the latent space (an approach comparable to the state-of-the-art, such as Google Magenta’s MusicVAE). By predicting a realistic path instead of a naive mathematical average, the VAE+NN architecture achieved a much lower reconstruction loss.

To ensure the validity of the results, a quantitative user study was conducted with 32 users from 8 different countries. The subjective evaluation confirmed that human listeners strongly preferred the interpolations generated by the VAE+NN model over the linear sampling baseline.

This research resulted in two publications to international journals and conferences (JISE 2022 and IEEE TAAI 2020).