GloveSynth
From Music Hack Day
Contents |
[edit] Name of your Hack
GloveSynth - controlling music and audio on laptops, phones, and iPads with one pair of gloves
[edit] Creators
Alessandro Contini, Johannes Marx, Kelly Snook (@kellysnook), Brio Taliaferro (@briomusic), Rachel Uwa (@onna_no_hito)
[Contributors to technology used in this hack: Imogen Heap (creator of the Gloves) and her Glove Team (see http://www.imogenheap.com/thegloves); Tarik Barri (assistance with OSC controlled visuals): Daniele Guevara (co-creator of CloudSynth app)]
[edit] About the Hack
We have created a system (by modifying several existing systems and hacking some new ones) that allows control of a roomful of computer musicians, VJs, and CloudSynth players with one pair of sensor-equipped gloves (in this case, Imogen Heap's recently debuted music gloves). The ultimate goal of this is to create an easy way for a performer to interact with a participatory audience, or a large group of people to play together, as in an orchestra.
This hack required a slight modification of the master glove code to distribute streaming sensor data to multiple devices, which have each been hacked in a different way to use the data to manipulate sound and visuals.
CloudSynth has been modified to use the external glove pitch, yaw, and roll to control delays, reverbs, and filters on SoundCloud-sourced audio clips instead of using built-in device sensors. Each user still chooses the audio clips that are being played and plays them normally.
Meanwhile on one laptop, Alessandro has written a PD patch that receives incoming OSC data from the gloves and maps controls to various samplers and effects inside PD.
On another laptop, Johannes has used the existing Max controls to map to pre-recorded clips and effects inside Ableton Live. Pointing with the hand for example plays 'You'. With 'Thumbs up' you can switch to major chords played on the Midi Rod, 'Thumbs down' switch to minor chords.
On yet another laptop, Rachel has mapped glove motions to visual elements.
[edit] Technology used
Gloves (created previously by Imogen Heap, Tom Mitchell, Seb Madgwick, Kelly Snook, Adam Stark, Hannah Perner-Wilson, Tim Exile, Rachel Friere, Becky Stewart, and Thomas Ermacora): - Bend sensors on all fingers - Integrated sensor chips on wrists (accelerometers, magnetometers, gyros) - Serial data via serial->USB into Glove Master laptop
Glove Master computer: - C++ code with neural networks to learn hand postures and convert raw data into gestural data (OSC) (Tom Mitchell) - Max MSP code to distribute OSC data to other devices (via UDPsends to specific IPs) (Imogen Heap, Adam Stark, Kelly Snook, Tom Mitchell) modified for this hack by Kelly Snook
CloudSynth app on iDevices (modified from previous hack by Brio Taliaferro, Alessandro Contini, and Daniele Guevara) : - Pure Data and Objective C to generate app code - CloudSynth App (starting with previous hack) allows user to pull audio from SoundCloud (by searching on keywords), then to define beginning and end points of sample used, then to play (pitch shift) that sample on a keyboard - CloudSynth app now hacked so that user can choose between local control of the sound or glove control (left or right hand). Filter control has been added. - Glove controls of the CloudSynth app are:
- Glove roll (around axis of the arm) = filter cutoff frequency - Yaw (left and right) = Delay send amount to the right, reverb send amount to the left - Pitch (up and down) = Effect amounts (reverb time and delay repeats)
- User can also choose whether to be controlled by Left or Right hand of the gloves, which allows splitting of participatory audience into two groups.
Audio on laptops: - Pure Data (custom patches by Brio and Alessandro), MaxMSP and Ableton Live (custom patches by Johannes)
Video on laptops: - Max/MSP/Jitter (custom patches by Rachel)
[edit] Media
The Team: http://www.youtube.com/watch?v=dE9yItWgoic&feature=youtu.be
