By Alex Li
In the early 90s, an audio technology, with its incredible compression capability, rapidly spread all over the world and completely changed the way people were listening to music.
This technology is called MP3, which stands for MPEG Audio Layer III, an encoding format for digital audio and music. It could commonly achieve a 75% to 95% reduction in file size while still preserving the best sound quality.
Using MP3, people could easily access to a huge range of music worldwide, which led to its instant and huge success. However, this was never good news to music industry.
What happened at the time? What is the whole story behind MP3?
To take a closer look at this great invention, we recently interviewed Professor Karlheinz Brandenburg, who is known as “The Father of MP3”. Though he doesn't like this title, he is no doubt one of the most important scientists involved in the development of this popular format.
In this email interview, Professor Brandenburg talks about how he started his work on MP3, the challenges he and his team encountered while creating MP3, the main reasons for MP3's huge success and the catastrophe brought to the music industry.
Professor Brandenburg also shares his new goal of building a leading audio technology company and his current focus.
Karlheinz Brandenburg's new company
Looking back, Professor Brandenburg told us he was deeply influenced by his parents and cultivated his interests in mathematics and engineering at an early age, which naturally laid the foundation for his future fulfilling career.
The following is our conversation with Professor Karlheinz Brandenburg.
LiveVideoStack: Professor Brandenburg, we are thrilled to have you here. Before the interview starts, could you introduce yourself?
Karlheinz Brandenburg: My name is Karlheinz Brandenburg. I am currently a senior professor at Technische Universität Ilmenau. Among other ventures, I am the CEO of Brandenburg Labs GmbH, a startup company working towards a perfectly realistic immersive audio experience via headphones. If you have heard my name before, it was probably in connection with the MP3 technology. My team and I were significantly involved in the development of the MP3 format, starting from basic research and standardisation and later on the marketing of the technology.
LiveVideoStack: Professor Brandenburg, how did you get interested in mathematics and engineering?
Karlheinz Brandenburg: As a child, I used to play with children's engineering play sets, and not only did I redo them, but I tried to build everything from scratch. I liked playing with electronics and even made my own circuits at an early age. Moreover, my parents were very good at mathematics, even though they followed different career paths later. As a result, mathematics was easy for me in school. Since I enjoyed the subject, I did it very well.
LiveVideoStack: Was there a particular person who influenced you when you were young?
Karlheinz Brandenburg: I look up to my parents. Growing up, I voraciously read books to broaden my knowledge, played musical instruments, and loved being in nature. I remember reading books about famous inventors and thinking about their significant accomplishments. I never thought I could be an inventor one day.
LiveVideoStack: Looking back on the early days, what advice would you give to your younger self?
Karlheinz Brandenburg: My advice for my younger self would be to keep going. Earlier in my school days, I was always the best student and had responsibilities such as a group leader or team organiser. When I joined the university, I decided to study electrical engineering and mathematics as my majors. Therefore, I had to study hard and dig deep into concepts to understand them better. I engaged myself with several tasks and responsibilities at once; looking back, I would advise myself that there are limits and I should not try to do too much.
LiveVideoStack: Could you tell us how you started your work on MP3?
Karlheinz Brandenburg: In fact, there was no major plan behind that. I did my master's in Electrical Engineering with a thesis focusing on digital signal processing. At that time, the work required a fast processor, which was very expensive, and there were no special computers available at the university.
Thereafter, my Professor and Ph.D. advisor Dieter Seitzer dreamed of transmitting high-quality audio, especially music, via an Integrated Services Digital Network (ISDN). This former digital phone network had a speed of 128 kilobits per second. Today, it is no longer available. He applied for a patent for a system that would distribute music using such a system over wires. The patent examiner told him this would not be possible, as he needed much higher bit rates.
Consequently, the Professor decided to find a Ph.D. candidate who would look into this and asked me whether I would take on the topic. This was good enough as a Ph.D. project; therefore, I agreed to examine the topic. Nevertheless, I thought the patent examiner was correct, and it could not work. Then, of course, it worked much better than anticipated. We had the first system ready in the early 1980s. This was basically a development of something proposed for speech coding, and it ran on a specially built computer at a sampling frequency of 30 kHz. Anything more would have made our computer in Erlangen crash.
We presented our findings at conferences, and then the real work started. We looked into why the system, at least for some music, delivered bad audio quality to the end user. At that point, someone proposed to us to look into the human perception of sound (psychoacoustics). I did some tests using the mini-computers, which were still very new and slow. Moreover, it took many hours to process just 20 seconds of music. Unfortunately, there was still no progress, and following the scientific approach, we had to think of other ideas. This process went on for several years.
In 1986, I had an idea that proved to be a breakthrough. I read some papers on what people had done in speech coding and combined that with our early ideas of music compression. This gave us much more flexibility, especially enabling better sound quality for music that seemed hard to code before. Using this technique, it worked much better than before.
Later that year, I wrote a note to my thesis supervisor that I supposed there was a new algorithm that could become the best music coding system. We got public funding because we were part of a big contract to plan and develop a system for digital radio called Digital Audio Broadcasting (DAB). I was still just a Ph.D. candidate at the university. At that time, Professor Seitzer started a new Fraunhofer group in Erlangen. Dr. Gerhaeuser, my former master's thesis advisor, was the head of one of the departments that looked into these topics. Shortly, we had a few people and resources to develop and standardise what would become MP3.
While we and other research groups in Europe were active in this project, Dr. Leonardo Chiariglione of Telecom Italia started efforts to develop a generic standard for coding audiovisual signals. The original aim was to use digital low-bitrate coding of audio and video to get movies on CD-ROM. We had a total bit rate of 1.5 megabits per second, most of which was taken up by video. I got to attend the first meeting of the audio expert subgroup within the Moving Pictures Experts Group (MPEG) in Hannover, Germany.
These were the early days of the audio subgroup within the Moving Pictures Experts Group (MPEG). Basically, all research groups known to be active in the field attended this first meeting. At this meeting in Hannover, I got the first connection to AT&T Bell Laboratories, where James D. Johnston did very similar work as I did. Later we joined forces and worked together. Finally, 14 research groups submitted proposals. Given the pressure that companies with similar ideas should work together, we, together with AT&T and others, submitted a proposal called ASPEC. This later became a basis for the so-called Layer III of MPEG Audio. This mode had the best audio quality at low bit rates but, at the same time, was the most complex mode of the original MPEG-1 Audio standard.
LiveVideoStack: What was the most challenging part of creating MP3?
Karlheinz Brandenburg: Primarily, when I finished my Ph.D. work in 1988, there was still one signal which didn't work as anticipated, and that was Suzanne Vega's acapella voice in the song “Tom's Diner” on her Solitude Standing album, which sounded terrible. Thus, it was a technical obstacle to finding out the problem and how to solve it.
Another subsequent hardship was politics in the standards committee. Our group consisted of relative newbies in the field, while other competitors were either supported by extensive industry or connected to major public broadcasters. Professor Seitzer was even told “no interest” when he proposed to public broadcasters in Germany that they could have a look at our demo technology.
Furthermore, the standards committee was split with various individuals with power and voice in the group opposing everything we proposed.
LiveVideoStack: MP3 faced many strong competitors as an audio compression scheme at the time. How did it stand out among others and become successful?
Karlheinz Brandenburg: Regarding licensing, at first, we referred to how other technologies are licensed. We were looking for a licensing scheme that could benefit us as well as the end user at no cost. Software decoder companies had to pay a relatively low one-time fee. For everything which was not a PC (including mobile phones), licensees had to pay a small amount for each piece of hardware sold. This added up nicely, and today there are billions of mobiles and computers with MP3 technology.
An important manager in the consumer electronics industry later told me that our group in Erlangen was one of the few in this field who understood how to do business on the internet.
A major reason behind the success of MP3 was that, at some point in about 1997, without authorization from the rights owners, people got CDs and distributed new music over the internet using MP3. For me, this was like stealing; we never supported the idea, and we were not responsible for these actions, but of course, it helped the proliferation of MP3.
LiveVideoStack: You are best known for co-inventing MP3, but what you’ve done for the audio world is much more than that. Could you talk a little bit about it?
Karlheinz Brandenburg: Initially in Erlangen and later in Ilmenau, we continued to work on how to do the best music compression for different purposes. I was heavily involved in developing Advanced Audio Coding (AAC), which Apple selected as their primary codec. Later, in Ilmenau, our work aimed to produce better audio reproduction for surround sound situations. We wanted to create the perfect illusion for loudspeakers, which worked very well using ideas initially developed by Delft Technical University in the Netherlands. We worked with them, further developed the ideas, and finally pushed that to the world. Of course, our technology (now called SpatialSound Wave) is still a niche product, but some basic concepts from that time are now used everywhere.
LiveVideoStack: After MP3, were there any innovations in the audio world that impressed you? What do you think could be the next milestone for audio compression?
Karlheinz Brandenburg: I believe the problem of audio compression is solved. Although, some people are still trying to improve audio quality. Another old dream that came true with a lot of work from the early group is to have high-quality audio over phone lines just for communications. Today, most recent mobile phones understand the Enhanced Voice System (EVS). The group in Erlangen played a big part in developing this standard. Our current work is to make another old dream come true. While you are wearing headphones, the real immersive audio, as we call it, makes you feel immersed in an acoustic world around you and sounds very realistic.
LiveVideoStack: Why did MP3 spread so quickly over the Internet? What happened at the time?
Karlheinz Brandenburg： The idea of sharing music over the internet was there already. The problem was that big labels and music companies did not want their business of selling vinyl records and CDs to be disturbed. With MP3, people spread music like wildfire without bothering about the legality of the distribution. Steve Jobs, the founder of Apple, told the big labels that they did not have much choice, and the idea for legal music distribution took off.
LiveVideoStack：When did you realize the catastrophe MP3 had brought to the music industry?
Karlheinz Brandenburg： In late 1994, an entrepreneur who had the idea of selling music from artists' back catalogues via the internet visited us to ask for a license of our technology. He already predicted that low bit-rate coding would destroy the music industry. We considered helping with the legal distribution of music because it was very clear that it is impossible to undo new technologies; they would always be around.
As a result, we worked on the encryption of music with a plan to distribute decoders that would, without additional cost, decrypt the legal files. Distributing music in an encrypted form would mean it couldn't be copied to somebody else. At that time, it was known to people in the music industry that we were working on this, so they never directly showed any bitterness toward us. In fact, the former president of the RIAA (Recording Industry Association of America) once called me a friend in an interview. He would not have said that if he thought we had caused problems.
In 1998 and later, several times we were asked if we thought we could get sued for enabling the unauthorized distribution of music. My answer was that MP3 is a standard issued by the International Standards Organisation (ISO) in Geneva, and we are not legally responsible for how the standard has been used.
LiveVideoStack: In your view, what are the main reasons for MP3's huge success?
Karlheinz Brandenburg： I think the main reason is that it is a very good technical solution to the problem of compressing music. We had an excellent licensing business model to be accessible to everybody. It was developed at the right time, just when PCs got fast enough and widely available. I believe it was just the right technology at the right time, and we were lucky.
LiveVideoStack: For MP3, you are not only a researcher but also a marketer. As you can see, which job is more challenging?
Karlheinz Brandenburg： I am an engineer, but I believe one should also take on the challenge of getting their own inventions to market. One place where I met a lot of other companies was SDMI, the Secure Digital Music Initiative. This was an international group of companies and people who tried to find solutions for the music industry. The meetings of this group involved both the content and the technology sides. I remember having the first meeting for one day at an airport hotel in Los Angeles. SDMI went nowhere, but our technology succeeded. Everyone knows how well MP3 has taken off, allowing easy access to a huge range of music worldwide.
LiveVideoStack: In 2019, you left Fraunhofer IDMT as director and founded Brandenburg Labs GmbH. Why did you found a new company? What goal do you want to achieve?
Karlheinz Brandenburg: When I no longer worked with Fraunhofer, I realized this could be an excellent opportunity to work on ideas I had for a long time, but there were just not enough resources to work on these. Especially the idea of a perfect audio illusion via headphones had been around for decades, but all earlier attempts by other companies did not sound as good as they should. We started to look into this topic at the university, with several Ph.D. candidates conducting basic research. This finally showed what had been overlooked by other researchers earlier on.
Of course, as a scientist, that is the way I always liked technology to proceed. The work that my Ph.D. candidates and I have done meant that the textbooks need to be rewritten. The question now is, can we put these ideas into the next generation of intelligent headphones?
After my time at Fraunhofer ended, I used part of the money left from my share of the MP3 patent royalties as seed funding for the new company. We have a proof of concept, but we still have way to go and a lot of other things to develop. Therefore, the goal I want to achieve is to build a leading audio technology company.
LiveVideoStack: What projects are you focusing on now? Could you give us an introduction here?
Karlheinz Brandenburg: The basic idea is to use the information on how our brain processes sound signals to improve listening. Just like people with glasses, who sometimes search for them while they are on the nose: they just improve our visual sense. In the same way with these headphones, if there is too much noise in the room, you can do intelligent noise cancelling but different from noise cancelling as it is done today. For example, you can filter out a group of people yelling at each other while being able to talk to other people with their voices elevated. This will give users a personalised, auditory reality.
LiveVideoStack: With its ultra-low bitrate compression capability, the AI audio codec is capturing people's attention. How do you see the advantages and limitations of an AI audio codec? What would be the best use for it?
Karlheinz Brandenburg: I'm not a believer in this kind of codec. It always depends on what it has learned. What we found the most crucial thing in audio compression or the original MP3 is that it should work well for everything. I think the basic problem is that these might work very well for a certain class of instruments or voices, but not for every signal you throw at it.
LiveVideoStack: Lastly, if you were given a chance to have a conversation with a mathematician or a musician, who would you want to talk to most? What would you like to talk about?
Karlheinz Brandenburg： Very interesting question! Music and mathematics often go together, and I would like to talk to people in both fields. In mathematics, there are still so many unsolved problems. The beauty of mathematics is that it helps us to understand things better. So I always think I can learn from other mathematicians. You may find this hard to believe, but I would never call myself a mathematician. There is always so much more to know.
On the other hand, I like music, so I have always liked to talk to musicians.
At last, we'd like to thank 王晶, 高泽华, Zhe Wang and SoundStudio for their contribution to this interview. We also appreciate the assistance provided by Franciska Wollwert at Brandenburg Labs GmbH during the interview.
All photos in the interview are provided by Brandenburg Labs GmbH.