• Classical:NEXT Encore
  • 30 NOV - 11 DEC
  • online
Behind the scenes

The Virtual Konzerthaus

Opportunities for digital cultural mediation through immersive technology

The Virtual Quartet in augmented reality. Photo: Pablo Castagnola.

Digitalisation has literally flooded all areas of our lives in recent years. In education, at universities, in industry or in the tourism sector, digital media has become indispensable in all its forms and facets. The topic is also increasingly relevant in the cultural sector. For several years now, the Konzerthaus Berlin has been working intensively on new ways of conveying music digitally, and thanks to the use of modern technologies such as augmented and virtual reality, it is spearheading the field in Germany. In this blog post, Elena Kountidou and Annette Thoma from the Konzerthaus Berlin offer insights into their work, their approach and their convictions on how and why digital technologies must also be increasingly anchored in culture.

Immersive technologies at the Konzerthaus Berlin


The Konzerthaus Berlin has been placing an emphasis on innovative formats for years. It all began in 2015. Together with the HTW Berlin University of Applied Sciences, an initial 360° video was created with the Konzerthausorchester Berlin under its honorary conductor Iván Fischer. This production was shown in the vestibule of the Konzerthaus Berlin and was so well received that it led to a long-term cooperation with the HTW Berlin. The “Apollo” project, supported by the EU through the ERDF Structural Fund, got underway in 2016 and is still running today.

The focus of the project is enabling classical music to be experienced in an entirely new way – and not only by the general public, but also by classical music experts. The members of the interdisciplinary project chose VR and AR for this purpose to ensure a modern-day presentation using the latest technologies. The team, coming from a diverse range of fields – programmers, developers, designers, musicologists, dramaturges and Communication Department staff at the Konzerthaus Berlin – found a common language. This was not always a matter of course in what was then still quite a new interface between classical music and technology.

Programmers have to understand how the world of classical music works, musicologists have to learn what kinds of things are possible in AR or VR and what the limits of these technologies are. The mutual exchange, perspective shift and open communication have been basic requirements for the project having worked so successfully over the years.

In the following section, some of the projects introduced by the Apollo team will be described as examples of collaborative work:

Projects using augmented reality:
About a year after the project started, the Konzerthaus Berlin published the augmented reality app “Konzerthaus Plus” in 2017. The first augmentations could be found in the 2017/18 season brochure with an interactive sound map of the Baltic States, photo galleries, musical excerpts and 3D models. Digitally enhanced postcards, an orchestra puzzle and a virtual timeline followed. Probably the most extensive project, in addition to a high-quality 3D model of the Konzerthaus and five of its halls, is the “Virtual Quartet”.

 

 

Here, members of the Konzerthaus Quartet are transformed into virtual musicians. By scanning several playing cards with the app, the respective musicians appear on the smartphone screen. Users can then decide whether they want to listen to the musicians together or concentrate on a single instrument – once a playing card is moved out of the camera’s focus, not only the virtual musician but also his or her individual instrument disappears. The Virtual Quartet is particularly suitable for music lessons in schools, as well as music education in general. Users playfully learn how a string quartet is structured and what the individual instruments are called and sound like – and can decide for themselves how they would like to explore the work being played.

Projects using virtual reality:
To date, three major VR projects have been implemented as part of the Apollo project. The first was a multi-perspective 360° video with the Konzerthausorchester Berlin. The user takes a virtual seat on stage between the musicians and its honorary conductor Iván Fischer and can follow the interaction and non-verbal communication between the orchestra members from several perspectives. Each 360° sphere also has 3D sound, giving an authentic impression of the symphony being played.

The second project was a virtual tour through the Konzerthaus Berlin. By putting VR glasses on, the user learns more about the history and architecture of the building with the help of a narrator, pictures, videos and animations. The tour is based on 3D models of the exterior facade and the halls, which are already part of the AR app.

The third and most recent VR project is an interactive composition that was created with the support of #experimentedigital, a cultural initiative of the Aventis Foundation. Together with the American composer Mark Barden and the Mexican visual artist Julian Bonequi, Apollo members have created “Environments”, a musical score come to life that can be walked through and explored. The complex and highly interactive project encourages users to become composers themselves by searching out and activating individual sounds in a surreal visual world.

A glimpse into the interactive composition “Environments”. Screenshot: Julian Bonequi.

The digital exhibition

Since the summer of 2018, the projects have been showcased in a digital exhibition that can be found in the vestibule of the Konzerthaus Berlin. The exhibition is open almost daily free of charge, offering direct, low-threshold access to everyone. Each month, volunteers overseeing the exhibition count about 10,000 visitors from all over the world – not including evening concert audiences. The applications are all communicated in German and English for international visitors. The scope of the exhibition is continually being expanded, and its content is also regularly updated. Feedback from colleagues and visitors is decisive here.

The digital exhibition in the vestibule of the Konzerthaus Berlin. Photo: Pablo Castagnola.

Outreach

Even though the AR app “Konzerthaus Plus” is available to everyone globally and free of charge, it is an important concern of the Konzerthaus Berlin to make the digital projects available without a high level of technological knowledge in the urban realm as well. In so doing, even more people of all ages and technology levels can be reached. Thanks to the technology’s portability, Apollo members have been guests in several primary schools, using VR glasses and tablets to teach students how a symphony orchestra is organised or the sound of a string quartet. The projects are also presented at the “Kiezkonzerte” (neighbourhood concerts), which take place once a year in Berlin.

A school class explores the virtual Konzerthausorchester Berlin in the digital exhibition. Photo: Annette Thoma.

During the winter months of 2019/20, the exhibition went on its first major tour through three Berlin district libraries. For several weeks at a time, the pillars could be found in Spandau, Friedrichshain and Neukölln libraries, where they reached a completely new but very interested target group. They were accompanied by activity days and small concerts. However, the Konzerthaus Berlin also regularly opens its doors to school classes who are very interested in this modern way of learning about music. Incidentally, requests do not only come from teachers. Tourist groups and senior citizens’ associations often find their way to the vestibule and, in addition to a tour of the building, are given a guided explanation of the exhibition followed by a discussion.

Concert halls should not be afraid of new reception habits. Visitors and decision-makers have to lose the idea that digital offerings are replacing the live experience.

The authors about new reception habits.

The Konzerthaus Berlin embeds digital music education in its overall programme, conceptualising and communicating it in cross-media form. The projects are disseminated via social networks and thus reach a broad audience.

The blog virtuelles-konzerthaus.de is primarily aimed at people who are interested in technology. Particularly the “Konzerthaus Plus” has extended the range of educational offerings into the digital realm. Print publications such as season brochures, festival flyers and programme booklets are also augmented to convey content in an innovative way. Konzerthaus 24/7: the logical consequence.

Hurdles and obstacles in digital productions


It is clear that despite many advantages, there are always obstacles when it comes to producing digital projects. Three cornerstones are listed here as examples (the list certainly is not exhaustive). Each project is different; sometimes more complex, sometimes more straightforward in terms of planning and implementation. The hurdles and issues that arise depend on previous experience, institutional requirements, project complexity, target group expectations and budget – just to name a few factors.

Human and financial resources


Every project costs money. This fact is both simple and understandable. However, especially when it comes to producing digital projects, many institutions find it difficult to understand how such a large sum of money could possibly turn up at the end of the cost estimate. The problem: many are familiar with top-quality animations from Hollywood films or graphics from Triple A games. But they often aren’t aware of the amount of effort involved. The request: this quality is omnipresent nowadays, so my product also needs to be high-end! The answer: of course – but only if the budget, personnel resources and timetable allow.

When a cultural institution seeks out an agency or a production company, it is of great advantage if the employees are already familiar with the client’s working environment and also the expectations of the respective target group. If not, communication is the key. After all, the project can also be implemented as a mere tech demo – whether and how it ultimately helps the institution to communicate its content and reach visitors is another question to explore.

Technological conditions vs. musical excellence


For a larger production in VR, e.g. with a symphony orchestra, the fact that classical music and technology are frequently at odds with one another should never be overlooked. Technology imposes strict requirements – the lighting on site, the distance separating the 360° cameras from the musicians and the exact positions of the microphones are just some of the aspects that need to be considered by all those involved. Then there are also the performers, whose most important goal is to make music together. Under certain circumstances, the technical conditions can lead to restrictions or altered situations in the orchestral structure. Here, too, communication between all those involved – the production company, musicians and conductor – is a decisive factor for the project’s success.
When a cultural institution seeks out an agency or a production company, it is of great advantage if the employees are already familiar with the client’s working environment and also the expectations of the respective target group. If not, communication is the key. After all, the project can also be implemented as a mere tech demo – whether and how it ultimately helps the institution to communicate its content and reach visitors is another question to explore.

One of the four 360° cameras with 3D microphones in the midst of the musicians of the Konzerthausorchester Berlin. Photo: Albrecht Sensch.

What comes next?


Even when a project is completed, it hasn’t ended yet. Depending on how an institution presents its project, both the software and the hardware must be maintained and serviced. In the Konzerthaus Berlin, several tablets and VR glasses are available for visitors in the digital exhibition, but the exhibition does not run itself. The hardware has to be cleaned, the app receives regular updates, questions from visitors who have never worn VR glasses and have no previous technical experience have to be answered. In the Konzerthaus Berlin, this is carried out with the help of specially trained volunteers. They support visitors at the AR pillars and encourage them to put on the VR glasses, as VR and AR are still largely unfamiliar fields in the classical field. The challenge is to sensitize visitors to digital content instead of asking them to hastily change their views.

 

 

Why are digital media so important for cultural institutions?


Like many areas of life, music and listening habits are undergoing rapid technological changes. The listening experience, initially limited to the concert hall, has increasingly emancipated itself from this performance venue. Music has been recorded on vinyl records or CDs, transmitted via radio and is now available in completely digital form. Music has become omnipresent and independent; it is more immediately available than ever before.

The classical music industry should see this as a challenge. Concert halls should not be afraid of new reception habits. Visitors and decision-makers have to lose the idea that digital offerings are replacing the live experience. The shared experience of a real concert cannot be replaced by VR glasses, smartphones or live streams. It is much more likely that technologies such as AR and VR will form another building block in the evolution of music communication.

Linear mediation approaches are complemented by these new media possibilities. Another big plus point: AR, VR, but also apps and games in general, ensure enormously low-threshold access. Nearly every teenager has a smartphone that he or she can use to access mediated content; it is an everyday part of their world. A smartphone is also portable, so it can be used anywhere and at any time. Cultural participation has never been so omnipresent, direct and simple as it is today – if cultural institutions understand this as an opportunity, both for themselves and the upcoming generation of classical music lovers, and use it in a meaningful and lasting way. The potential of digital cultural mediation, which is still in its infancy in many places, is palpable. It creates new media-based access to knowledge and education.