Sound objects

A box sits on the desk, with no markings on its surface. Touch it with a finger and the box makes a sound, impact, friction, deformation, depending on where you touch. There are soft buttons on the surface, but you can’t see them. You hear them.

Notes from Mikael Fernström’s lecture at AHO in February 2005. Fernström runs the Interaction Design Centre at the University of Limerick, where the Soundobject project is working on sound in ubiquitous computing, a relatively unexplored area of interaction design.

The aim of the Soundobject research is to liberate interaction design from visual dominance, to free up the eyes, and to do what small displays don’t do well.

Reasons for focusing on sound

Sound is good at attracting attention, which is why alarms and notification systems work. We talked about ‘caller groups’ on mobile phones, assigning different ringtones to people in an address book, and how effectively that changes our relationship with our phones. It’s possible to sleep through unimportant calls: the brain processes and evaluates sound while we sleep.

One thing I hadn’t considered: sound is our fastest sense. It has extremely high temporal resolution, ten times faster than vision, so our ears can hear pulses at a much higher rate than our eyes can watch a flashing light.

Disadvantages of sound objects

Sound is not good for continuous representation, because we cannot shut out sound in the way we can divert our visual attention. It’s also not good for absolute display: pitch, loudness and timbre are relative to most people, and even people with absolute pitch can be affected by contextual sounds. Context is a big issue: loud or quiet environments affect the way sound must be used in interfaces, libraries and aeroplanes for example.

There are also big problems with spatial representation in sound. Techniques that mimic the position of sound based on binaural differences are inaccessible to about a fifth of the population. The perception of space in sound is intricately linked to the position and movement of the head. (See Psychophysical Scaling of Sonification Mappings and some Google searches on spatial representation of sound.)

Cartoonification

‘Filling a bottle with water’ is a sound that could work as part of an interface, representing actions such as downloading, uploading, or in place of a progress bar. The sound can be abstracted into a ‘cartoonification’ that works more effectively: the abstraction separates simulated sounds from everyday sounds.

Fernström cites foley artists working on film sound design as inspiration; they are experienced in emphasising and simplifying sound actions, and in creating dynamic sound environments, especially in animation.

A side effect of cartoonification is that sounds can be generated in simpler ways, reducing processing and memory overhead in mobile devices. All of the Soundobject experiments use parametric sound synthesis with PureData, sound generated on the fly rather than using sampled files, resulting in small, fast, adaptive interface environments. (Sound files and the PD files used to generate the sounds are at the Soundobject site.)

One idea Fernström mentioned: simulate ‘peas in a tin’ to hear how much battery is left in a mobile device. Entirely possible in software using the accelerometer in the Nokia 3220. Imagine one ‘pea’ rattling about, instead of one ‘bar’ on a visual display.

Research conclusions

The most advanced prototype was a box that responded to touch, with invisible soft-buttons on its surface that could only be heard. Synthesised sounds responded to the movement of fingertips across a tactex touchpad. The soft-buttons used a simplified sound model that synthesised impact, friction and deformation. See Human-Computer Interaction Design based on Interactive Sonification.

Testing involved asking users to feel and hear their way around different patterns of soft-buttons, and to draw the objects they found. See these slides for some of the results.

The conclusions: users were almost as good at using sound interfaces as with normal soft-button interfaces, and auditory displays are a viable option for ubiquitous computing, particularly wearable.

More reading