Generating sequences of musical notes from lyrics might sound like the stuff of science fiction, but thanks to AI, it might someday become as commonplace as internet radio. In a paper published on the preprint server Arxiv. (“Conditional LSTM-GAN for Melody Generation from Lyrics“), researchers from the National Institute of Informatics in Tokyo describe a machine learning system that’s able to generate “lyrics-conditioned” melodies from learned relationships between syllables and notes.
“Melody generation from lyrics has been a challenging research issue in the field of artificial intelligence and music, which enables to learn and discover latent relationship between interesting lyrics and accompanying melody,” wrote the paper’s coauthors. “With the development of available lyrics and melody dataset and [AI], musical knowledge mining between lyrics and melody has gradually become possible.”
As the researchers explain, notes have two musical attributes: pitch and duration. Pitches are perceptual properties of sounds that organize music by highness or lowness on a frequency-related scale, while duration represents the length of time that a pitch or tone is sounded. Syllables align with melodies in the MIDI files of music tracks; the columns within said files represent one syllable with its corresponding note, note duration, and rest.
The researchers’ AI system made use of the alignment data with a long-short-term memory (LSTM) network, a type of recurrent neural network capable of learning long-term dependencies, with a generative adversarial network (GAN), a two-part neural topplay network consisting of generators that produce samples and discriminators that attempt to distinguish between the generated samples and real-world samples. The LSTM was trained to learn a joint embedding (mathematical representation) at the syllable and word levels to capture the synaptic structures of lyrics, while the GAN learned over time to predict melody when given lyrics while accounting for the relationship between lyrics and melody.
To train it, the team compiled a data set consisting of 12,197 MIDI files, each paired with lyrics and melody alignment — 7,998 files from the open source LMD-full MIDI Dataset and 4,199 from a Reddit MIDI dataset — which they cut down to 20-note sequences. They took 20,934 unique syllables and 20,268 unique words from the LMD-full MIDI, and extracted the beats-per-minute (BPM) value for each MIDI file, after which they calculated note durations and rest durations.
