Design Iteration
I’ve created hundreds of “instrument” variations over the past several years. Each one represents a milestone in my creative journey. They are prototype interfaces for performance and improvisation with varying degrees of complexity
User Interface Design
Music and video controllers should help the performer improvise with ease. When I design a performance interface, I create layouts that represent the structure of musical arrangements. Musical layers, like visual layers, blend to compose a song or piece of visual artwork. This similarity is easier to understand when the instrument (iPad interface, in this case) models the shared properties of sounds and images.
Studying Gesture
Combined with software tools (Max/MSP), I use touch screens to test and catalog different types of gestures (taps, flicks, swipes, drags) and their relationships to sound and visual motion. As I do so, I ask questions like, what sound does this flick gesture make? How does a swipe change the path of an animation? Asking these questions allows me to bridge the gap between sound and motion in a visceral and tactile way. The touchscreen becomes a unified interface (language) for time-based media.
Algorithms: Mapping Interactivity
Performing synchronized audio/visual compositions requires mapping sound to video in a logical way. To keep the visuals in time with the music, I program musical notes to trigger video clips. As my compositions get more complex, I build sophisticated algorithms for controlling inputs and outputs between music controllers and audio/video software.
Interface as Art
Digital interfaces are usually a series of boxes, buttons, and text. While this style of interface is useful for completing tasks in a rational and linear way, artistic expression often deviates from linearity. For this reason, I occasionally take a painterly approach to UI design allowing the interface itself to become more emotive and abstract.
Hardware & Sensor Design
Sometimes the right interface or control surface isn't available for purchase. In this example (see image), I created force-sensing touch interface predating Apple's ForceTouch. Using a Nintendo DS touch screen, magnetoresistive sensors, and software built in Processing, I created a prototype surface for a simple force-detecting digital painting application.