GENIUS! #23: Peter Neubäcker and Melodyne Direct Note Access23 March 2015
Munich-based Celemony was founded in late 2000 and soon became a familiar name in the DAW-based studio environment through its popular pitch correction software, Melodyne. Then the company’s founder Peter Neubäcker introduced DNA (Direct Note Access) to the package, and suddenly users could grab individual notes of audio in polyphonic arrangements and shift them around as if using a MIDI editor. And it’s all down to some wrongly recorded percussion…
How did you get started?
I have always been interested in studying the mathematical relationships in music. At first, I used paper, a pencil and a pocket calculator. Then, in the mid ’80s, I realised that a computer would be the ideal tool to help me with my research, so I taught myself to program. At the beginning of the ’90s, I switched to a NeXT computer with the integrated DSP, audio processing also then became possible. [NeXT was a shortlived business created by Steve Jobs when he first left Apple.] The question that interested me above all at the time was what a “sound” actually is and how one can shape it. I developed a process for handling sound independently of its pitch and evolution over time.
Even before Direct Note Access, Melodyne was gaining traction as one of the preferred pitch correction tools. Was polyphony always a goal?
At the start, Melodyne was not conceived as a correction tool at all; my idea was rather that you could use existing audio material to compose freely with; that in the process of composition you could position sounds freely in terms of time, duration and pitch. But Melodyne then came to be perceived by users as above all a correction tool.
The editing of polyphonic material was, of course, always an implicit objective. There were no rational grounds for anyone saying “I only want to edit monophonic material”, but based on what I knew at the time I did not at first even consider it possible. Having said that, I had long reasoned that if we, as listeners, are capable of identifying the individual notes in polyphonic music, there must be indicators in there somewhere.
What led to the breakthrough that resulted in DNA?
One day a friend sent me a recording of a track with a marimba; the problem he had was that in the performer’s part one of the notes had been miscopied. Unfortunately the marimba had been hired for the session and was no longer available, so they were stuck with a wrong note in the recording. And because successive notes played on the marimba run into each other, it could not be rescued with the existing monophonic Melodyne. By this time, my thoughts on the subject of audio separation had advanced at least to the point where I could conceive of having a go at solving the problem. It often happens that I have ideas that go on fermenting for years without my writing anything down or doing any actual programming. Even though I had not yet written a line of code, once I began it all went very quickly: only a week later, I was able to send the user the marimba notes as individual audio files.
What are the next challenges you hope to tackle?
I spent the best part of last year developing a reliable tempo detection algorithm for freely recorded music. I think that is another import step forward in the way music is produced. Many musicians in the studio nowadays record to a click, but this is not something anyone would want to do on musical grounds. With our new tempo detection, a radically different workflow is possible: a musician or band can play the music freely in their own tempo when recording, and the tempo track with its beats and bars is later extracted automatically from the recorded material. This then makes it possible to copy notes from any one passage into another or synchronise a drum track to the recording later. This technology will be integrated into our next version of Melodyne.
Hail to the boffins! Genius! is all about celebrating those clever people whose inventions have transformed the world of professional audio. Mailed out with the February print edition of PSNEurope, the 36-page supplement is also available to read in handy digital-edition form. Read it online, or download as a PDF, at www.psneurope.com/introducing-genius.