urMus - a meta-environment for mobile interaction design
urMus is a meta-environment for live and interactive application design and programming on and for multi-touch mobile devices such as the iPhone. It's design incorporates a range of new concepts to support rapid prototyping, on-the-fly patching and natural integration of sensors and actuators of mobile devices into interactions and artistic output. It also offers a detailed event-driven engine for multi-touch is a meta-environment. It is designed according to many contemporary principles for universal human-computer interaction such as multi-layered design, design by molding and multiple representations. Check out the urMus web page for more. Selected publications:
mobile coding & mobile live coding
mobile devices are fully functional computers. However, programming them directly is as yet poorly supported or understood. This research is interested in investigating programming on mobile devices that takes into account the limited screen space and the capabilities of the devices input capabilities such as multi-touch. In order to allow programming as an artistic practice we are also interested in live-coding mobile devices. Here we take two approaches: On-the-fly principles on the device itself and live-coding mobile musical instruments remotely. The realm of live-coding is still in its infancy and there are many important open problems to be resolved in the area. Check out Sang Won's blog for more on his work on live-coding mobile musical instruments. Selected publications:
two-sided force interactions on mobile devices
How should one act on devices that are hand-sized? One possibility is two-sided interactions. We naturally grab objects with opposing thumb and finger. In addition we actually have quite good control over applying forces in that posture. Our research explores these type of interactions and hardware that allows us to prototype these interactions. The first prototype is called sandwich and uses two iPhones and external force sensing.
mobile phone orchestras and ensembles
I have been interested in large group performance of mobile devices since this has become possible. I am currently direting the Michigan Mobile Phone Ensemble. And in fact I co-founded and co-directed the first repertoire-based mobile phone orchestra at CCRMA Stanford University, joint with Ge Wang (Stanford) and Henri Penttinen (HUT). We had a bunch of other concerts since, in Genova, Italy, in Belfast, UK and in San Francisco. In 2009 I formed the Berlin Mobile Phone Orhcestra which played as part of the Long Night of Science in Berlin, Germany.Michigan Mobile Phone Ensemble and Stanford Mobile Phone Orchestra web sites! Selected publications:
expressive mobile interactions
Since 2005 I have been interested in turning commodity mobile devices into new (and like-new) musical instruments. Together with Michael Rohs, Ananya Misra, and Martin Roth we systematically explored the sensor and sound capabilities of these devices as they evolved. We used motion camera signals with 2D marker detection, as well as optical flow in CaMus, we explored accelerometer and magnetometer data in ShaMus, we ported MobileSTK to SymbianOS as a first parametric sound synthesis library for this operating system, we reappropriated the microphone as blowing sensor to build mobile wind instruments in MiMus and we explored multitouch in Fendrix. We also looked at ways to support collaborative performance over local bluetooth networks in CaMus2 and explored ways to create rapid on the fly editing on Symbian 12-key devices in SpeedDial. CaMus was mainly used as an interactive MIDI-based remixer and still required an external computer. Thereafter all projects used on-phone audio pipelines only.
structure-preserving physical modeling
We use structure-preserving ideas to model waves in the plane. We already have very successful simulation method for the one-dimensional wave equation called digital waveguides. But can we do as well in 2-D? I explore the structure of the wave equation in the plane to figure out what the exact correspondences between the wave equation in 1-D and 2-D are and how one can find efficient methods while preserving key structures. The picture to the left shows the reduced problem of tracing wave fronts. One can show that only smooth curves and specific sharp edges called cusps can ever exist and this can be seen nicely in the rendered picture. This is of a single point excitation off center after a few reflections.
[video - 2D Wave]Selected publications:
PebbleBox is one in a series of tangible interface designs that explore the role of action and physicality in expressive interfaces (the others have descriptive names like "Scrubber" or "DaGlove"). Jerome Bruner called this connection "enactive" and and psychologists today talk about sensorimotor integration. We look at the malleability of the device by manipulating synthesized sound while keeping the tactile experience the same. This is joint work with Sile O'Modhrain on the designs. Also joint with Charlotte Magnusson, Joakim Eriksson and Sile on experiments.
[video - PebbleBox]Selected publications:
abstract sound synthesis
Circle maps are the simplest non-linear extensions of linear discrete oscillators. Essentially they are just a one-parameter perturbation of the linear case. These are very simple maps that can be implemented as one-step iterations. Yet they already exhibit a wealth of the phenomenology of non-linear effects, like mode-locking, bifurcation and chaos. That actually makes them hard to control, but the plus side is that these are stable for all choices of parameters and a lot of their properties are rather well understood. We look at circle maps and other oscillators for new building blocks for abstract sound synthesis, which are both expressive and complex, yet algorithmically cheap (so that we can run them on our mobile devices without breaking a sweat).
[video - CircleMap]Selected publications:
A lot of stuff - also on other, older projects (pre-2005ish) like banded waveguides, non-propagating excitations and more - can be found in my publications! Link is on the left. If you are interested in my work on synthesizing dispersive media efficiently using banded waveguides, a bit of that is still available at the Sound Lab pages at Princeton.
Some linked papers may be copyrighted. They are provided here solely for previewing for individual, educational, academic purposes. Please contact the cited source directly to acquire a copy.