I wonder if anybody has tried using AI/LLMs to try to arrange music. I hadn't really seen any, and musicians don't tend to be on the bleeding edge of tech, so I just might be on the forefront of this.
Anyhow, I've long wanted to try Lilypond... for years. It's a music engraver software that's code-based, and as somebody whose day job is a software developer, it immediately resonated with me. Now with AI being surprisingly good at writing code, maybe there's something here...
So I gave it a try...
tl;dr — it kind of sucks at it.
Lilypond has been on my radar for years, but whenever I tried to get going with it, I found the tooling to be very rough, and the documentation rather sparse. It seemed I spent more time looking up documentation for syntax (e.g. how do I write a tenuto? How do I add a tie to this note?) than I was actually writing.
Compared to using a GUI like Sibelius, this was a significant impediment that I had to work through.
I never did learn the syntax well enough, and after multiple false starts, I ended up giving up years ago.
In the years since, the tooling has gotten a little better, and there's native integration with my IDE (Visual Studio Code), so things are looking up.
I cobbled together a quick setup with the following:
Qwen 3.6 35B-A3B running locally on my laptop via llama.cpp
Pi coding harness
Visual Studio Code with the LilyPond Syntax plugin
The key here is I'm not going to rely on AI to write the actual music, but could it autocomplete melodies? If I gave it something like the start of a phrase, and asked it to fix the rhythm, or continue the melody, could it?
The short answer is no — it was pretty awful at it.
I couldn't rely on it to do anything of substance. A couple times it even looked at my notation and decided that my bars were missing beats, even though they weren't. Here's a funny one — it saw a dotted half note and thought it was worth 2.5 beats... which if you think about it literally (and not at all musically), makes sense... it's a half note, worth two beats, and a "half", so half a beat.
So yeah, grade 2 rudiments it has not.
Other times it tried to look at my notation, and try really really hard to put together the melody, but would hallucinate lyrics and make up melodies that were nowhere coherent.
But here's the kicker... it was decent at Lilypond syntax! I was able to get rid of my biggest hurdle (having to figure out obscure syntax) and just ask for what I wanted:
Q: How do you write a triplet?
Use \tuplet with a ratio in parentheses, then the notes in parentheses:
\relative c'' {
(c8 d e) % regular eighth notes
\tuplet 2/3 { (c8 d e) } % triplet: 3 notes in the space of 2
}
The ratio is numerator/denominator — "3 notes in the space of 2" = 2/3.
Common examples:
\relative c'' {
% 3 eighth notes in the space of 2 (normal triplet)
\tuplet 2/3 { (c8 d e) } f4
% 3 quarter notes in the space of 2
\tuplet 2/3 { (c4 d e) }
% 5 notes in the space of 4
\tuplet 4/5 { (c8 d e f g) }
% 7 notes in the space of 4
\tuplet 4/7 { (c8 d e f g a b) }
}
The parenthesized notes (c d e) inside the tuplet are required — they tell LilyPond to draw the bracket and number.
Not bad at all!
Note that I'm using a very small LLM relative to the ones you can use online (ChatGPT, Gemini, etc.) so there's definitely a gap in intelligence between mine and the frontier, so I'm interested in seeing how far I can take this.
