The Voice Memo Graveyard: Building a Second Brain for Musicians with Python

Over the past two years, I recorded 54 voice and guitar memos but finished only one song. I wanted to go back to those ideas (but I rarely did), so I built keep-audio.xyz to help me turn those amateur songwriter ideas into full songs.

Last week I talked about this experience at PyCon Taiwan, and oh boy, sharing it was fun!

What was not that fun was realizing that this SaaS was dead from the very beginning, and even my friends who make music weren’t interested in using it. The target audience is too small and too niche, or the features aren’t compelling enough. Unlike Spotify, where anybody could listen to a song, how many people are actually making them?

Still, I wanted to see if it could help me with my own songwriting.

1. The voice memo graveyard

The chord progressions, riffs, and melodies that I had recorded on my phone were mostly dead. So I thought: “Can I use Python to organize my ideas better and to move forward in the creative workflow?”

I didn’t know exactly what to build, so I defined my anti-goals first:

2. What to build

Cool, but how are the pros making good songs? I defined a promising combination of ideas as a Song Seed, for example:

1. a riff (think the Seven Nation Army intro)

2. a melody (either verse or chorus, think Shape of You)

3. lyrics (think The Smiths)

   So we go inside and we gravely read the stones
   All those people, all those lives
   Where are they now?
   With loves and hates
   And passions just like mine
   They were born
   And then they lived
   And then they died
   Seems so unfair
   I want to cry
   — Morrissey, “Cemetry Gates”

Each of those components is an idea that users record on their phones after having a burst of creativity, then they just:

1. Upload each idea (recording) to the website.
2. Add tags to the idea (guitar/piano/voice, high/low energy, quick/slow, etc).
3. Click “find matches” and the system groups their ideas together!

3. Python rules to match my ideas to each other

What kind of magic makes 2 ideas “go together”? Can I just set a reward system for certain behaviors among ideas? Kinda:

I wanted to match more than 2 ideas, but checking every possible combination quickly becomes expensive (combinatorial explosion problem). Like in classic recommendation systems, I used two stages:

1. Stage 1 is fast candidate pairing: the system looks for ideas to complement the recording I picked, rewarding different roles and instruments, or a shared mood.
2. Stage 2 evaluates that entire group as a potential song, considering things like energy, key, and language.

4. You can’t argue with black boxes

A core idea was rejecting a black box telling me which ideas go together. Python rules give me deterministic behavior and explainability for free, and that’s what I wanted as a user:

5. Architecture and local ML inference for speech-to-text

How to run this without keeping my laptop on 24/7? I hosted Keep-audio on a free 4 vCPU, 24GB memory Oracle Cloud VM running Caddy, FastAPI, SQLite, and a local ML model: Whisper (a ~140MB open-weights model developed by OpenAI that can transcribe a song in just a few seconds).

6. What to take back to your own project

Presenting at PyCon for the first time was exciting, and I wanted to share my opinionated views on building software products (especially for solo devs like myself):

1. [Domain] Try rules before complex ML models. You get deterministic behavior and explainability for free (good for users).
2. [Engineering] Stick to a simple architecture until it breaks. A single VM with FastAPI and SQLite can take you shockingly far.
3. [Mindset] Define your anti-goals early. Decide what you won’t build, and leave the enjoyable parts of the craft to humans.

To finalize, I’ll answer the million-dollar question: has Keep-audio helped me finish more songs? Not really (yet), as I only just built it. For now, allow me some more time to play around with it and make one or two hits.