georg essl projects

home
research
teaching
publications
software

urMus - a meta-environment for mobile interaction design

urMus is a meta-environment for live and interactive application design and programming on and for multi-touch mobile devices such as the iPhone. It's design incorporates a range of new concepts to support rapid prototyping, on-the-fly patching and natural integration of sensors and actuators of mobile devices into interactions and artistic output. It also offers a detailed event-driven engine for multi-touch is a meta-environment. It is designed according to many contemporary principles for universal human-computer interaction such as multi-layered design, design by molding and multiple representations. Check out the urMus web page for more.

Selected publications:
  • Essl, G. “Automated Ad Hoc Networking for Mobile and Hybrid Music Performance,” In Proceedings of the International Computer Music Conference (ICMC), Huddersfield, 2011.
    [pdf]
  • Derbinsky, N. and Essl, G. “Cognitive Architecture in Mobile Music Interactions,” In Proceedings of the International Conference on New Interfaces for Musical Expression (NIME), Oslo, 2011.
    [pdf]
  • Essl, G. “UrMus - An Environment for Mobile Instrument Design and Performance” In Proceedings of the International Computer Music Conference (ICMC), Stony Brooks/New York, June 1-5 2010.
    [pdf]

mobile coding & mobile live coding

mobile devices are fully functional computers. However, programming them directly is as yet poorly supported or understood. This research is interested in investigating programming on mobile devices that takes into account the limited screen space and the capabilities of the devices input capabilities such as multi-touch. In order to allow programming as an artistic practice we are also interested in live-coding mobile devices. Here we take two approaches: On-the-fly principles on the device itself and live-coding mobile musical instruments remotely. The realm of live-coding is still in its infancy and there are many important open problems to be resolved in the area. Check out Sang Won's blog for more on his work on live-coding mobile musical instruments.

Selected publications:
  • Lee, S. W., Essl, G. “Live Coding The Mobile Music Instrument” In Proceedings of the International Conference on New Interfaces for Musical Expression (NIME), Daejeon + Seoul, South Korea, p.28-34, 2013.
    [pdf]
  • Essl, G. “Mobile Phones as Programming Platforms” In Proceedings of the First International Workshop on Programming Methods for Mobile and Pervasive Systems, Helsinki, May 17 2010, Pervasive.
    [pdf]
  • Essl, G. “SpeedDial: Rapid and On-The-Fly Mapping of Mobile Phone Instruments” In Proceedings of the International Conference on New Interfaces for Musical Expression, Pittsburgh, June 4-6, 2009.
    [pdf]

two-sided force interactions on mobile devices

How should one act on devices that are hand-sized? One possibility is two-sided interactions. We naturally grab objects with opposing thumb and finger. In addition we actually have quite good control over applying forces in that posture. Our research explores these type of interactions and hardware that allows us to prototype these interactions. The first prototype is called sandwich and uses two iPhones and external force sensing.

Selected publications:
  • Stewart, C., Rohs, M., Essl, G., Kratz, S. "Characteristics of Pressure-Based Input for Mobile Devices" To appear in Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI 2010), Atlanta, Georgia, USA, April 10-15, 2010. Full paper.
    [pdf]
  • Essl, G., Rohs, M., Kratz, S. "Squeezing the Sandwich: A Mobile Pressure-Sensitive Two-Sided Multi-Touch Prototype" 22nd Annual ACM Symposium on User Interface Software and Technology (UIST), Victoria, BC, Canada, October 4-7, 2009. Demo.
    [pdf]

mobile phone orchestras and ensembles

I have been interested in large group performance of mobile devices since this has become possible. I am currently direting the Michigan Mobile Phone Ensemble. And in fact I co-founded and co-directed the first repertoire-based mobile phone orchestra at CCRMA Stanford University, joint with Ge Wang (Stanford) and Henri Penttinen (HUT). We had a bunch of other concerts since, in Genova, Italy, in Belfast, UK and in San Francisco. In 2009 I formed the Berlin Mobile Phone Orhcestra which played as part of the Long Night of Science in Berlin, Germany. Check out the Michigan Mobile Phone Ensemble and Stanford Mobile Phone Orchestra web sites!

Selected publications:
  • Wang, G., Essl, G., Penttinen, H. "Do Mobile Phones Dream of Electric Orchestras?" In Proceedings of the International Computer Music Conference (ICMC), Belfast, UK 24.-29. August, 2008.
    [pdf]

expressive mobile interactions

Since 2005 I have been interested in turning commodity mobile devices into new (and like-new) musical instruments. Together with Michael Rohs, Ananya Misra, and Martin Roth we systematically explored the sensor and sound capabilities of these devices as they evolved. We used motion camera signals with 2D marker detection, as well as optical flow in CaMus, we explored accelerometer and magnetometer data in ShaMus, we ported MobileSTK to SymbianOS as a first parametric sound synthesis library for this operating system, we reappropriated the microphone as blowing sensor to build mobile wind instruments in MiMus and we explored multitouch in Fendrix. We also looked at ways to support collaborative performance over local bluetooth networks in CaMus2 and explored ways to create rapid on the fly editing on Symbian 12-key devices in SpeedDial. CaMus was mainly used as an interactive MIDI-based remixer and still required an external computer. Thereafter all projects used on-phone audio pipelines only. Selected publications:
  • Misra, A., Essl, G., Rohs, M. "Microphone as Sensor in Mobile Phone Performance" In Proceedings of the 8th International Conference on New Interfaces for Musical Expression (NIME 2008), Genova, Italy, June 5-7, 2008
    [pdf]
  • Essl, G., Rohs, M. “Interactivity for Mobile Music Making”, Organised Sound 14:2 197-207, 2009
    [pdf]
I have also been advising the iPhone app startup company Smule during its formative year. Check out all their cool stuff!

structure-preserving physical modeling

We use structure-preserving ideas to model waves in the plane. We already have very successful simulation method for the one-dimensional wave equation called digital waveguides. But can we do as well in 2-D? I explore the structure of the wave equation in the plane to figure out what the exact correspondences between the wave equation in 1-D and 2-D are and how one can find efficient methods while preserving key structures. The picture to the left shows the reduced problem of tracing wave fronts. One can show that only smooth curves and specific sharp edges called cusps can ever exist and this can be seen nicely in the rendered picture. This is of a single point excitation off center after a few reflections.
[video - 2D Wave]

Selected publications:
  • Essl, G., "Computation of Wavefronts on a Disk I: Numerical Experiments," Post-conference publication, "Third Irish Conference on the Mathematical Foundations of Computer Science and Information Technology, Dublin, Ireland, July 22-23, 2004." Electronic Notes in Theoretical Computer Science 161, 25-41, 2006.
    [pdf]
  • Essl, G., “"Whispering" waves and Bate's ridges in numerical experiments” Acoustics Research Letters Online 6:3, 227-231, 2005.
    [pdf]

sensory-rich interfaces

PebbleBox is one in a series of tangible interface designs that explore the role of action and physicality in expressive interfaces (the others have descriptive names like "Scrubber" or "DaGlove"). Jerome Bruner called this connection "enactive" and and psychologists today talk about sensorimotor integration. We look at the malleability of the device by manipulating synthesized sound while keeping the tactile experience the same. This is joint work with Sile O'Modhrain on the designs. Also joint with Charlotte Magnusson, Joakim Eriksson and Sile on experiments.
[video - PebbleBox]

Selected publications:
  • Essl, G. , O'Modhrain, S., "An Enactive Approach to the Design of New Tangible Musical Instruments", Organised Sound 11:3 285-296, 2006.
    [pdf]
  • Essl, G., Magnusson, C., Eriksson, J., O'Modhrain, S. "Towards evaluation of performance, control of preference in physical and virtual sensorimotor integration" In Proceedings of ENACTIVE 2005, the 2nd International Conference on Enactive Interfaces, Genoa, Italy, November 17-18, 2005.
    [pdf]

abstract sound synthesis

Circle maps are the simplest non-linear extensions of linear discrete oscillators. Essentially they are just a one-parameter perturbation of the linear case. These are very simple maps that can be implemented as one-step iterations. Yet they already exhibit a wealth of the phenomenology of non-linear effects, like mode-locking, bifurcation and chaos. That actually makes them hard to control, but the plus side is that these are stable for all choices of parameters and a lot of their properties are rather well understood. We look at circle maps and other oscillators for new building blocks for abstract sound synthesis, which are both expressive and complex, yet algorithmically cheap (so that we can run them on our mobile devices without breaking a sweat).
[video - CircleMap]

Selected publications:
  • Essl, G. "Circle maps as a simple oscillators for complex behavior: II. Experiments" In Proceedings of the International Conference on Digital Audio Effects, Montreal, September 18-20, 2006.
    [pdf]
  • Essl, G. "Circle maps as a simple oscillators for complex behavior: I. Basics" In Proceedings of the International Computer Music Conference, New Orleans, November, 2006.
    [pdf]

A lot of stuff - also on other, older projects (pre-2005ish) like banded waveguides, non-propagating excitations and more - can be found in my publications! Link is on the left. If you are interested in my work on synthesizing dispersive media efficiently using banded waveguides, a bit of that is still available at the Sound Lab pages at Princeton.

Some linked papers may be copyrighted. They are provided here solely for previewing for individual, educational, academic purposes. Please contact the cited source directly to acquire a copy.