While rehearsing at SXSW last month, Chagall van den Berg had an unusual problem: his digital knee shot out in the wrong direction. "My friend laughed and said:" Wow, that's a problem that no SXSW artist has ever had ", says van den Berg The Verge .
van den Berg is a musician who tracks the movement. gloves and a full body suit covered by sensors that, during this SXSW performance, not only control the projection of a digital avatar that appears behind it, but also control almost all the instruments and effects of the music and its voice. As you move around the stage, your avatar, floating in space, moves in synchrony. When she stretches her arms over her head, the audio grains become slow and she stutters. Every movement of hands and body has a cause and an effect, creating an oniric landscape with a pop infusion that is fascinating to watch.
With these sensors, van den Berg can bring chords and melodies with a wave of his hand, or distort the video of himself in extravagant ways by raising an arm. Because each movement can create audio or visual changes, their performances are very physical, but in an elegant and deliberate way. "I can do all the movements and look like an air traffic controller," says van den Berg. "That would work, but that's not very performative, all the songs I play have movements that are functional and also significant."
van den Berg performs with movement tracking gloves in 2017. Performance begins at 3:50.
Born in Amsterdam, Netherlands, van den Berg was obsessed with music and computers from a young age, but he had always seen them as separate things. She played instruments in bands, was a singer / songwriter and traditionally worked with other producers to create rhythms for her songs. Then, in 2011, he entered a "real" studio for the first time. Seeing the producers make noodles with their songs to make remixes, he clicked on "I can do this." She taught herself in music production and, shortly after, in 2012, she released her first EP. "I immediately realized how much freedom and independence they gave me," says van den Berg. "I no longer needed other people to produce my songs, my musical expression became much more direct because I could only make the sounds I had in my head instead of explaining them to another human being."
van den Berg had solved a problem by itself, but releasing music created another: how would he interpret the songs he had made? The nature of most electronic tracks meant that you had two options: stand behind a table with a set of gears, buttons and faders, or play an accompaniment track and sing at the top. None was acceptable to her. "I had this option," says van den Berg. "Or it was going to be real and live, but boring to see and distant from the audience, or I would play a recording and I could dance on stage." Dancing and being one with the audience was much more attractive, but the musician who was in me really did not like the idea of singing on a track. So I had a dilemma. "
This dilemma faced by Van den Berg is a problem faced by many DIY and electronics artists: how do you incorporate movement and expressivity when you essentially stop at a desk, using an interface that the public probably never see, and then make it interesting? Acts like The Glitch Mob use electronic drum kits and Microsoft Surfaces pirated to the audience, while several startups, such as Enhancia and Genki Instruments, bet on MIDI controller rings The initial van den Berg solution was Mi.Mu, a pair of movement tracking gloves created by the musician Imogen Heap.Each glove has nine sensors and triggers that are fully customizable.An almost any movement can be assigned any parameter musical, so you could, for example, lower your arm to add reverb or pinch the air to add chorus to your voice. Den Berg was so inspired by the Mi.Mu gloves that she wrote "Sappho Song" the day she received them.
That was in 2014, and he launched a search for van den Berg to fluidly bridge the gap between his body. Movements on the stage, and control over audio and visual effects. While the gloves allowed Van den Berg to control the audio and video with his hands, she wanted to do more: how could she use her whole body instead of just her hands? The iteration he's working on now combines gloves with a motion capture suit, the kind of thing that is usually used to record the movements of people for video games or movies. For now, he has a song programmed to use all the technology he's using, but the goal is to expand to a full one-hour show. That is not an easy task. Along the way, he had to learn C ++, find a company to lend his sensors (he does not own the system, estimate the total cost at around $ 12,000) and continuously experiment with new combinations of existing technologies on all platforms to link everything . together.
the van den Berg configuration looks elegant and minimal on stage at SXSW, but it takes a complex network of hardware and software to make it work. Take the Mi.Mu gloves in your hands and then put on a custom suit with 15 of Xsens 3D motion tracking sensors that span your arms, shoulders, head, pelvis, legs and feet. From side to side there are three computers: a Mac laptop with three applications to control audio playback and effects, a Mac mini that uses the "creative coding" tool openFrameworks to manage all the images controlled by Mi.Mu gloves and a Windows-based laptop that runs the suit's sensors and connects them in real time to the Unity game engine via the wireless full-body VR platform. There is also additional hardware, such as microphones and an audio interface. To make sure there are no problems, the program runs on its own Wi-Fi network, which requires places to turn off Wi-Fi in your home or move it to a different frequency to avoid interference. Its alot. But the result, an accomplishment of five years of work to connect body, voice, music and video into a single reactive thing, is incredibly exciting.
The whole process should become much simpler if other electronic musicians are going to adopt it, says van den Berg. "[My setup] is not the way we're going to change the world of electronic music performance because nobody is going to spend all this money and then they will also feel very concerned about IP addresses and networks during the sound test", He says. "Who wants to tour with three expensive computers and set up all that during a 20-minute change in a festival? That's crazy. "
van den Berg is not the only electronic artist to think about the connection between his physical being and technology: the musician Laura Escudé also uses controllers based on movement and reactive visuals, for example, but it is one of the most ambitious, and even though each trick is self-sufficient, since it brings it closer to the program than has in mind, ultimately, has a greater goal: van den Berg hopes that his experimentation will encourage others to think how their performances can be more immersive, and that it can make it easier for artists who want to do similar things.
"My dream is that I'm not the only one in the world who does things like this," says van den Berg. "The interpretation of electronic music becomes more human and more expressive." That means being the one to deal with with all l technical headaches, so that others do not have to.
"I'm a kind of guinea pig," he laughs. . "But that's fine with me."
van den Berg performs at SXSW. Video credit: New Dutch wave.