Variational Autoencoders for Polyphonic Music Interpolation

What is music interpolation?

In traditional music generation, the model generates music based on past events. But what if we could use both past and future events? The goal is to generate a middle track that connects a given beginning and ending.

We input a begin track and an end track of 10 seconds each, and the model generates a middle track of 10 seconds. We can then compare the generated track with the original middle track from the human composition.

The tracks are polyphonic, meaning that several notes can play at the same time. They are represented as MIDI, which describes notes symbolically instead of using a recorded audio waveform.

Each timestep is represented as a vector of 64 binary elements, one for each pitch: 1 means the note is on, and 0 means it is off.

Dataset and approach

I created the Hsinchu Interpolation MIDI Dataset, based on the Lakh MIDI Dataset. It contains 30,830 segments of 30 seconds each, selected to include segments where the beginning and ending tracks are different.

I tested four approaches: random data, a bidirectional LSTM, a variational autoencoder (VAE), and a VAE combined with a neural network (VAE+NN). The VAE approach encodes the beginning and ending tracks into vectors, averages those vectors, and decodes the result to produce the interpolation track.

The VAE+NN approach replaces the averaging step with a neural network that estimates the interpolation vector:

1. Encode the begin track and end track with the VAE to obtain z_begin and z_end.
2. Use a neural network to estimate z_interpolation from those two vectors.
3. Decode z_interpolation to obtain the polyphonic interpolation track.

Results

I evaluated the models using mean squared error (MSE) against the original human-composed middle tracks. In these experiments, VAE+NN achieved lower reconstruction error than the VAE with linear interpolation. This measures similarity to the reference track; it does not by itself tell us how the music sounds.

More objective results

The table below compares track MSE at different binarization thresholds. The threshold determines whether a predicted note is treated as on or off.

The next table compares MSE in the latent space for the two VAE approaches.

Listening study

I also conducted a listening study with 32 people from eight countries. In the total votes shown below, VAE+NN received 52 votes, compared with 44 for VAE; human compositions received 66.

This research resulted in a conference paper at IEEE TAAI 2020 and a journal paper in JISE in 2022. The papers, thesis, code, and presentation slides are linked above.