Week 9: 7/17-7/23 Special Topics: Museum Informatics
Passing the Baton: The Mendelssohn Effekotorium
A unique partnership of the Mendelssohn Foundation, WHITEvoid, Aconica, and Bertron Schwarz Frey, has created an interactive museum installation that allows anyone to become a virtual conductor of the music of the romantic composer Felix Mendelssohn Bartholdy (1809-1847) using an innovative 3-D motion capture system. Housed in a room in the Mendelssohn House Museum in Leipzig, Germany, the Mendelssohn Effecktorium was opened in February of 2014 and quickly achieved worldwide notoriety for the excellence of the design and the high-quality of the virtual experience. I will be analyzing the installation in the context of museum informatics and provide a brief history of the project, a description of the technology used, and analysis of the project. Before reading further, I suggest you watch the two linked video presentations to get a sense of the installation:
(1) https://www.youtube.com/watch?time_continue=8&v=LlPlmRzThW8 (Links to an external site.)Links to an external site.
(2) https://www.youtube.com/watch?v=1EdRFE3JS30 (Links to an external site.)Links to an external site.
Background
The Mendelssohn Foundation (MF) is the not-for-profit independent foundation that oversees the legacy of the composer Felix Mendelssohn including his original home and carriage house in Leipzig which have been converted into a museum overseen by the MF. The MF was chartered in 2012 and since that time it has invested €1.5 million – “dramatically raising the standard of the displays and exhibits in the Mendelssohn Museum” (Wikipedia, 2017). As part of its mission, two main objectives of the MF are:
“international research and nurture concerning the artistic and societal inheritance bequeathed by the Composer and Director of the Gewandhaus Orchestra, Felix Mendelssohn” “encourage musical training and education in a manner consistent with the priorities of the composer who founded in Leipzig Germany's first music conservatory” (Wikipedia, 2017) One of the ways to fulfill that mission was to create the Mendelssohn Effektorium inside the Museum to allow visitors to experience first-hand what is it like to conduct an orchestra while simultaneously experiencing the music in a fun exploration of the role of the composer in the expression of the music. The Museum worked with WHITEvoid (creative direction, GUI, and visuals), Bertron Schwarz Frey (interior design), and aconica (creative direction, sound, supervision recording, production, programming) to create the installation. I was able to track down an interview with Martin Backes of aconica discussing the project and I have based much of my account upon his account with supplementation from other web resources.
The Installation
The Mendelssohn Effektorium consists of a single room housing a conducting stand with an embedded touch-screen interface equipped with a 3-D motion detection camera and conducting baton facing an “orchestrally-positioned” group of thirteen sculptural-quality LED/loudspeakers with floor-to-ceiling length white curtains on three sides of the room which hide an elaborate LED lighting array which creates atmospheric lighting effects (hue, intensity) linked to the performance of the music – all under the control of the “virtual” conductor.
Controlling the installation are two powerful ethernet-linked computers which control the touchscreen, a 3-D camera, the LED lighting array, and the LED/loudspeakers using Open Sound Control (OSC) protocol-based messaging to synchronize the system components to the real-time performance of the virtual conductor and the touchscreen controls.
The heart of the virtual conducting experience is the technology housed in the conducting podium.
A LEAP motion tracking system captures the movement of the conducting wand (which Mendelssohn helped popularize) and translates the up/down motion of the wand into information to control the tempo (fast|slow) of the performance of the score. The effective range of the system is around 60cm so virtual conductors are constrained by the technological limitation to perform close to the conducting stand. Actual composers use a full-range of body motions but this currently cannot be captured by the system. The system allows a visitor to select the options of playback audio without conducting or with conducting. If conducting is chosen, the visitor must conduct continuously or the system will stop. A visualization of the visitor's performance to see if they are going to slow or too fast helps guide users to conducting the score at the right tempo. The touchscreen interface was built using Java and OpenGL and allows the virtual conductor to control the following elements of the performance: Selection of a particular score and position within the score Volume for single orchestral or choral groups Selective muting to hear individual instruments or voices Visualization of the score and notes Ability to compare a music piece as directed by five different conductors Alter the room acoustics: dry, music salon, concert hall, church Adjust the tuning: historical instruments (pitch 430 Hz) and modern instrument tuning (pitch 443 Hz) Visualization of timbre and volume through the LED/loudspeaker displays General lighting effects
The 13 LED/loudspeaker sculptures are positioned to mimic the structure of actual instrument groups in a physical orchestral arrangement and allow visitors to walk through and listen to individual speakers to isolate particular instrument groups. This allows for a unique experience virtually “walking through” an orchestra pit. The sculptural quality of the speakers gives them an anthropomorphic aspect which is even more pronounced by the use of circular speakers to “crown” the columns. The displays are used to display the names of the instrument group playing through each speaker as well as visualizing timbre and volume. The volume display gives the 13 speakers the look of a giant equalizer while the timbre display option is more representational of the “tonal” quality of the instrument group (aconica, 2017).
The second computer hosts the digital audio files of the music and the music software that synchronizes all the various inputs to create a seamless, zero-latency experience for the virtual conductor. The software used for the project is Ableton Live and Max for Live along with additional plugins including Convolution Reverb Pro and Max Patch/Max for Live Patch. The Live Arrange view is used a the synchronization hub for the entire system and it integrates the incoming and outgoing OSC messages between the computers and the lighting and sound arrays. As Martin Backes summarizes it, “We basically read out the time of the audio files – we’re basically tracking the time and the position within Ableton’s timeline” (Create Digital Media, 2014).
Martin Backes also oversaw the creation of the audio files created for the installation. He worked with MDR studio engineers, conductor David Timm, a choir, and orchestra. He remarks: “We had specific needs for that installation, for both the choir and orchestra sounds. So everything had to be very isolated and very dry for the installation, which was very unusual for the MDR Studio and their engineers and conductor” (Create Digital Media, 2014). This in effect, allows the individual voices to be isolated or muted by the virtual conductor and clearly the recording parameters had to be thought out far in advance of the actual recording.
The Live software and plugins handle the various transformations of the original audio performance data and allow the music to keep the right pitch when the virtual conductor changes the instrument tuning, room acoustics or tempo.
The result of all this planning and technology integration is a seamless experience for the visitor with no noticeable latency in performance. The virtual conductor can stay immersed in the experience and be emotionally captivated by a rich aural and visual palette under their control. Martin explains, “It has to be easy to understand and more or less self-explanatory. So by gesturing up and down, one will have influence on the tempo of Mendelssohn’s music only – that’s basically it. All the other features were given to the touchscreen – functional wise. So we have basically two interactive components for the installation setting” (Create Digital Media, 2014).
Two interactive control components, immersive sound, and lighting displays synchronized to the performance. The result?
According to Martin, “The audience during the reopening was pretty mixed – from young to old. Most people were very surprised what they are able to do with the interactive installation. I mean, everything works in real time, so you would have direct feedback on whatever you’re doing. The audience could be an active part – I think that’s what people liked the most about it. … I saw a lot of happy faces as people played around with the interactive system and as they walked around within the installation room” (Create Digital Media, 2014).
Interpreting the Mendelssohn Effektorium with MacDonald’s UX Rubric
While MacDonald’s UX rubric is primarily structured with visual materials as a primary focus (MacDonald, 2015), the general format can be fruitfully applied to the Mendelssohn Effektorium to offer an “expert commentary” on the installation.
MacDonald breaks his rubric into three main headings: (1) Visceral – strength of visual content, visual aesthetics, (2) Behavioral-system reliability and performance, usefulness of metadata, interface usability and support for casual and expert users, and (3) Reflective-uniqueness of virtual experience, openness, integration of social features, personalization of features.
I am going to quote his definitions of “Emerged” (i.e. the highest rating category of each dimension) verbatim and comment on the applicability to the Effektorium.
Visceral
Strength of visual content: Artwork is presented as the primary focus of the collection, with images as the dominant visual element. All images are large and high quality. Text is used purposefully but sparingly to enhance the visual content. Visual aesthetics: Color, graphics, typography, and other non-interactive interface elements are harmonious and used consistently. Elicits affective reactions that are universally positive. MacDonald is clearly privileging the visual channels of experience in his rubric, but the category of visceral could be extending to cover the aural component of virtual conducting. The Effecktorium interface and installation meets a high standard for the strength of its visual content and aesthetics. The anthropomorphic aspect of the LED/loudspeaker sculptures is evidence in favor of this view along with the attractive presentation of the musical score with the touchscreen.
Behavioral
System reliability and performance: The interface is fully functional and completely free of technical errors. The pages consistently load quickly, and all aspects of the interface respond immediately to user actions. Usefulness of metadata: Metadata structure is purposefully designed to enhance users’ ability to find and learn about artworks. Includes novel metadata facets that offer innovative ways to browse, search, and filter artworks. Interface usability: Interface is intuitive and accessible. Interface elements are easy to locate and easy to use, creating a seamless and immersive interaction between the user and the collection. Support for casual and expert users: Primarily provides basic content and functionality for casual users. Advanced features are visible but unobtrusive, which effectively supports expert users and encourages learnability for casual users. Allows for a seamless transition between casual browsing and advanced research. The Effektorium scores would score high marks in all the relevant dimensions – it is reliable and exhibits no latency effects, the interface is intuitive (children as able to navigate easily), and would rate a slightly lower score for support for casual and expert users because of the range limitations of the current Leap motion software. Metadata in this context might be interpreted in this context to offer ways to experience the score along a range of parameters like tempo, tuning, instrumentation, and room acoustics that are analogous to extra-information about the score.
Reflective
Uniqueness of virtual experience: Virtual museum experience is entirely different from the physical museum experience. Finding and viewing virtual artworks allows for new and insightful perspectives that would not be possible in the physical museum. Openness: Users are given complete control over the content, with clearly marked options to download, print, and/or save high-quality images. Integration of social features: Encourages varying levels of participation within a virtual community, of which the museum is an active participant. Social tools are prominently integrated into the collection. Provides multiple options for sharing and communicating with others. Personalization of features: Allows users to craft dynamic personal experiences with few, if any, limitations. Integrates robust customization, gaming, and/or other innovative personalization features. Inspires users to be active co-creators of their virtual museum experience. In the reflective dimensions, the Effektorium would again rate high marks for the uniqueness of experience (virtual conducting); openness in terms of being able to alter many of the parameters of performance via the touchscreen; participation is high even among non-conductors who can enjoy the music, lights, and virtual conductors performance; and the range of performance parameters and control of lighting makes the conducting experience highly personalized.
As MacDonald continues to refine his useful UX rubric, he will need to account for other sensory modalities beyond vision to create a more general and more useful standard to use when evaluating UX standards. Even an online museum collection could have audio components.
Potential impacts
At a local experience level, the Effektorium adds an interactive experience to the Museum that both complements and amplifies the experience of the other elements the house museum experience. It provides a unique and dynamic way to listen and participate in the music of Mendelssohn as well as offering a vicarious introduction to the role of the conductor in concert music with the opportunity to experience a range of performance experiences in real-time under patron control. Taking the experience beyond objects and into the music in a way that is a different experience than would be had by passively listening to a recorded or even live ensemble performance of the Mendelssohn’s music.
It also helps fulfill the mandate of the Mendelssohn Foundation to provide music education to its audience as part of its larger institutional goal set. A because it is a dynamic experience much of its flavor can be conveyed through social media channels with a video presentation which creates a desire to experience the Effektorium even if some patrons aren’t particularly motivated to normally visit the Museum. This amplification creates greater visibility and hopefully greater numbers of visitors who can be exposed to the life and works of Mendelssohn and potentially be influenced by the broader mission objectives of the Foundation. By creating a project of excellence, the global stature of the Mendelssohn Foundation has been raised as well.
And potentially the installation could be duplicated in other music museums around the world and perhaps licensed (earning revenue for the Foundation) or presented in a traveling exhibit form to extend the mission reach of the Foundation to a global audience.
Challenges
The technical challenges while significant were largely overcome and able to be realized in the scope of much of the existing technology. No new radical informatic solutions were necessary for the project to be created. Certainly, the installation is generating accolades and positive publicity. But as with most museum informatic experiences, it can run the risk of becoming dated because of advances in technology or saturation of experience after being a conductor. It may overshadow other aspects of the museum and risk becoming the reason for the existence of the museum more than the life and music of the composer. There is also the risk of patrons leaving the exhibit with a confidence level that they understand the role of the conductor and perhaps don’t have a realistic appreciation of the actual work and role of the conductor during rehearsals, music selection, “fine tuning based on the skills of individual orchestra players”, and the demands of real-time live performances. A false sense of knowledge may especially be created in children that the baton is a like a magic wand that “makes the music happen”. As with any experience that is a representation of a real world activity or performance event, there is the risk that the wrong conclusions will be drawn.
Although I couldn’t find any monetary measures of the installation, a potential challenge for the Foundation may be the initial cost of the Effektorium installation and on-going maintenance and operating expenses. There may also be pressure to divert resources to expanding and amplifying it which might conflict with operating expenses and fulfillment of other Foundation goals.
Improvements
One immediate area where improvement could be made was in the range of motion and need for close proximity to the sensor array when using the baton. Due to the current state of development of the LEAP motion solution, a compromise in the installation was made – sacrificing the ability of the “conductor” to be less constrained in the body motions and to have greater freedom to move the baton around and away from the constricted range of the sensor array. This an area that according to the interview article will be revisited when the next major version of the software is released.
As an abstraction of the concert orchestra, the Effektorium has created an exceptional experience that seamlessly allows a patron to immediately control an “orchestra” and experiment with choices that affect the experience of the musical performance and emotional contours of the music itself in a way that is direct, largely unmediated, immersive, and fun. Beyond the improvement of the motion capture technology, the installation might be improved by adding more music from the Mendelssohn canon or incorporating augmented reality that would complement the conducting experience with a unique visualization of the music and musician representations in exciting and creative ways. Switching to period instrumentation and tuning might create an augmented reality display of performers in period costumes as a hypothetical. Additional textual or graphical information might be conveyed with AR or the lighting of the existing installation might be enhanced with a different type of display graph representing the conductor's performance.
Since the installation is capturing and performing in real time, it might be possible for patrons to login and have a digital audio file of their conducting session sent to a home email address as reminder of the experience and a marketing channel to provided Foundation messaging as well as encouraging motivated patrons to perhaps enroll in classes or educational or institutional programs. It might be possible as well to capture a video stream of the patron conducting and send that as part of their reminder email.
Evaluations
If I had access to the necessary data, I would perform an analysis of the Effektorium’s contribution to increasing traffic to the museum and website & calculate a return on investment based on the methods of throughput accounting where ROI = (Throughput - operating expenses) / investment to judge the contribution of the installation to the mission of the Foundation. I would also judge the Effektorium on its contribution to the main goals of the Foundation and how well it helps the institution reach and fulfill those goals.
A superficial SWOT analysis would identify the quality of the implementation as strong internal strength with the potential internal weakness being that it might overshadow less interactive parts of the Museum. But it seems to fulfill the mission of the Foundation to create an experience that goes beyond objects and elevates the music. One could definitely make a case for the ability of this installation to fulfill that mission. Opportunities would include creating a traveling exhibition based on the installation technology to reach more potential patrons, to increase interest in Mendelssohn in particular and concert music in general. Potential threats could arise from new technologies superseding its wow factor.
If I was in charge of operations of the Mendelssohn Foundation, I would conduct interviews with patrons after experiencing the Effektorium to measure the actual effects of interacting with the music of Mendelssohn in the role of a “conductor”. How much has it increased the targeted metrics the Foundation created before it even began the planning, construction, and implementation of the installation? Has the publicity and accolades from the project added value to the Foundation and Museum by increased reputation attracting better employees, more visitors, or increased funding from donations
References
Aconica. Effecktorium – interactive museography / interactive sound & light installation. [website]. Retrieved July 21, 2017 from http://www.aconica.de/portfolio_page/effektorium-interactive-museography/
Create Digital Media. (August 15, 2014). How gestures and Ableton Live can make anyone a conductor of Mendelssohn [behind the scenes]. Retrieved from http://cdm.link/2014/08/gestures-ableton-live-can-make-anyone-conductor-mendelssohn-behind-scenes/ (Links to an external site.)Links to an external site.
MacDonald, C. (February 1, 2015). Assessing the user experience (UX) of online museum collections: Perspectives from design and museum professionals. MW2015: Museums and the Web 2015. Retrieved from http://mw2015.museumsandtheweb.com/paper/assessing-the-user-experience-ux-of-online-museum-collections-perspectives-from-design-and-museum-professionals/ (Links to an external site.)Links to an external site.
Mendelssohn Foundation. (n.d.) Retrieved July 21, 2017 from https://en.wikipedia.org/wiki/Mendelssohn_Foundation (Links to an external site.)Links to an external site.