From skit to symphony: how AI and a team of experts completed Beethoven’s 10th

THEudwig van Beethoven died in 1827, three years after completing his Ninth Symphony, which many consider his masterpiece. He had started to work on a 10th Symphony, but the deterioration of his state of health did not allow him to carry out this project, which remained in the draft stage.

Since then, musicologists and Beethoven enthusiasts have wondered – and lamented not knowing – what could have become of this symphony. Today, thanks to the work of a team of music historians, musicologists, composers and computer engineers, Beethoven’s vision has come to life.

Responsible for the artificial intelligence (AI) component of the project, I led a scientific team at the AI ​​start-up Playform who taught a machine all of Beethoven’s work and his creative process.

A full recording of his 10th Symphony was released on October 9, 2021 and was performed for the first time on stage in Bonn, Germany, capping more than two years of effort.

Past attempts hit a wall

Around 1817, the Royal Philharmonic Society in London commissioned Beethoven to write his ninth and tenth symphonies. Written for an orchestra, the symphonies consist of four movements: the first is played at a fast tempo, the second at a slower tempo, the third at a moderate or fast tempo, and the last at a fast tempo.

In 1824, Beethoven completed his Ninth Symphony, which ended with the timeless “Ode to Joy”.

When it comes to the 10th Symphony, however, the composer left little behind except a few musical notes and a handful of ideas that he wrote down.

There have been some past attempts to reconstruct parts of Beethoven’s 10th Symphony. Most famous, in 1988, musicologist Barry Cooper ventured to complete the first and second movements. He woven together 250 bars of music from the sketches to create what was, in his eyes, a production of the first movement that was true to Beethoven’s vision.

However, the rarity of Beethoven’s sketches prevented symphony experts from going beyond this first movement.

Assemble the team

In early 2019, I was contacted by Dr Matthias Roeder, director of the Karajan Institute, an organization in Salzburg, Austria, which aims to promote the links between music and technology.

He explained that he was forming a team to complete Beethoven’s 10th Symphony on the occasion of the 250th anniversary of the composer’s birth. Aware of my work on AI-generated art, he wanted to know if AI could help fill in the blanks left by Beethoven.

The challenge seemed sizeable. To achieve this, the AI ​​would have to do something it had never done before. But I replied that I was ready to give it a try.

Roeder then formed a team which included Austrian composer Walter Werzowa. Famous for writing Intel’s signature jingle, Werzowa was tasked with implementing a new type of composition that would integrate what Beethoven left behind with what AI would generate.

Mark Gotham, an expert in computer music, led the effort to transcribe Beethoven’s sketches and process all of his work so that AI could be properly trained.

The team also included Robert Levin, a musicologist at Harvard University who also happens to be an incredible pianist. Levin had already completed a number of incomplete 18th century works by Mozart and Johann Sebastian Bach.

The project takes shape

In June 2019, the group reunited for a two-day workshop at the Harvard Music Library. In a large room with a piano, blackboard and a stack of Beethoven sketchbooks covering most of his known works, we explained how fragments can be turned into a complete piece of music and how AI can help. to solve this puzzle, while remaining faithful. to Beethoven’s process and vision.

Music experts in the room were eager to learn more about the kind of music AI has created in the past. I explained to them how the AI ​​had succeeded in generating music in the style of Bach. However, it was only a harmonization of an input melody that sounded like Bach. It didn’t come close to what we needed to do: build an entire symphony from a handful of phrases.

Meanwhile, the scientists in the room – myself included – wanted to know more about what kind of material is available and how the experts plan to use them to end the symphony.

At every moment, the genius of Beethoven presented itself, challenging us to do better. As the project evolved, the AI ​​did as well

The task at hand finally crystallized. We would need to use full notes and compositions from all of Beethoven’s work – as well as the available sketches of the 10th Symphony – to create something that Beethoven himself could have written.

It was a huge challenge. We didn’t have a machine to which we could send sketches, push a button, and spit out a symphony. Most AIs available at the time couldn’t continue an unfinished piece of music beyond a few extra seconds.

We would need to push the boundaries of what creative AI could do by machine-teaching Beethoven’s creative process – how he would take a few bars of music and painstakingly develop them into soulful symphonies, quartets, and sonatas.

Reconstructing Beethoven’s Creative Process

As the project progressed, the human side and the machine side of the collaboration evolved. Werzowa, Gotham, Levin and Roeder deciphered and transcribed the sketches of the 10th Symphony, trying to understand Beethoven’s intentions. Using his completed symphonies as a model, they attempted to piece together the puzzle of where the sketch fragments should go – what movement, what part of the movement.

They had to make decisions, like determining whether a sketch indicated the starting point of a scherzo, which is a very lively part of the symphony, usually in the third movement. Or they could determine that a line of music was probably the basis of a fugue, which is a melody created by interweaving parts that all echo a central theme.

The AI ​​side of the project – my side – found itself struggling with a series of difficult tasks.

First, and more fundamentally, we had to figure out how to take a short phrase, or even just a pattern, and use it to develop a longer, more complicated musical structure, just like Beethoven would have done. For example, the machine must have learned how Beethoven constructed the Fifth Symphony from a basic four-note motif.

One of the music experts said the AI ​​reminded him of a passionate music student who practices everyday, learns, and gets better and better.

Then, because the continuation of a phrase must also follow some musical form, whether it is a scherzo, a trio or a fugue, the AI ​​had to learn Beethoven’s process to develop these shapes.

The to-do list was growing: we had to teach the AI ​​to take a melodic line and harmonize it. The AI ​​had to learn to connect two sections of music. And we realized that the AI ​​had to be able to compose a coda, which is a segment that ends part of a piece of music.

Finally, once we had a full composition, the AI ​​was going to have to figure out how to orchestrate it, which involves assigning different instruments for different parts.

And he had to accomplish these tasks as Beethoven could have done.

Take the first big test

In November 2019, the team met in person again – this time at the Beethoven House Museum in Bonn, where the composer was born and raised.

This meeting was the litmus test to see if AI could complete this project. We printed musical scores that had been developed by AI and built from sketches of Beethoven’s 10th. A pianist performed in a small concert hall in the museum in front of a group of journalists, musicologists and Beethoven experts.

We challenged audiences to figure out where Beethoven’s sentences end and where the AI ​​extrapolation begins. They could not.

A few days later, one of those AI-generated scores was played by a string quartet at a press conference. Only those who were intimately familiar with Beethoven’s sketches for the 10th Symphony could determine when the AI-generated parts arrived.

The success of these tests told us that we were on the right track. But it was only a few minutes of music. There was still a lot of work to do.

Ready for the world

At every moment, the genius of Beethoven presented itself, challenging us to do better. As the project evolved, the AI ​​did the same. Over the next 18 months, we built and orchestrated two entire movements of over 20 minutes each.

We anticipate some setback from this work – those who will argue that the arts should be off limits to AI and that AI does not have to try to replicate the human creative process. Yet when it comes to the arts, I see AI not as a replacement, but as a tool – a tool that opens doors for artists to express themselves in new ways.

This project would not have been possible without the expertise of human historians and musicians. It took an immense amount of work – and, yes, creative thinking – to achieve this goal.

At one point, one of the team’s music experts said the AI ​​reminded him of a passionate music student who practices every day, learns, and gets better and better.

Now this student, having taken over from Beethoven, has presented the 10th Symphony to the world.

Ahmed Elgammal is Professor and Director of the Art & AI Lab at Rutgers University. This article first appeared on The conversation.