Expressive Gesture in Tangible Acoustic Interfaces

Research Objectives

Development of an open platform (EyesWeb version 4) for integrated multimodal processing of data from Tangible Acoustic Interfaces (TAIs)
Development of multimodal and cross-modal models and algorithms for analysis of expressive gesture in TAIs
Development of early prototypes of TAIs integrating algorithms for in-solid acoustic localisation systems.
Research is carried out in the framework of the EU-IST Project TAI-CHI (Tangible Acoustic Interfaces for Computer Human Interaction)

Conceived for supporting research on multimodal expressive interfaces and interactive systems
Widely employed for designing and developing real-time dance, music, and multimedia apps
Adopted in EU - IST projects: 5FP: MEGA, CARE-HERE, MEDIATE 6FP: STREP TAI-CHI, NoEs Enactive and Humaine
Adopted in research centers, universities and industries
Distributed for free on www.eyesweb.org Supported with public newsgroups

Includes libraries for audio and video processing
Includes libraries for movement analysis
Includes support for commercial applications
Can be freely extended by third parties
Supports consolidated industrial standards(e.g., Steinberg VST and ASIO (support to audio multi-channel h/w boards), OSC, MIDI, tcp/ip, serial, firewire, USB webcam, wide range of sensors and frame grabbers, etc.)

Integration in the EyesWeb language of aspects related to multimodal processing of different data stream. E.g., user do not have to explicitly care about synchronization of streams, the language automatically support this
Subpatching mechanism fully supported
Optimization of the kernel for management of audio and video streams
Support for multiprocessor systems
Complete separation of the kernel from the user interface

Different user interfaces can be developed and employed, depending on the specific application

Standalone applications which do not need user interface and have to operate for days in unsupervised environments can run with a minimal (or without) GUI
Simplified connection of the EyesWeb platform with other software platforms
Simplified hosting of standard plugins
Interaction with hardware managed by a new level of abstraction: devices
Multiple versions allowed on a single PC
Multiple instances allowed at the same time

Portable patches: xml
Better audio support
New portable SDK
Beta release in www.eyesweb.org
Final release under testing in a permanent museum interactive installation (Museo del Mare e della Navigazione, Genova).

E.g., Video processing of visual representations of acoustic signals (e.g., auditory display):

E.g., Video processing of visual representations of acoustic signals (e.g., auditory display):

E.g., algorithms based on the integrated analysis of the sound produced by hands (tapping on a table, handwriting) and the hand gesture (bimodal):

The same module can operate on different datatypes: e.g., the same “segmentation” Eyw module can be used for different streams (audio, movement cues, sensor data ...)
Support to resampling, interpolation, adaptation to simultaneous processing of different streams
Data conversion, synchronization, etc.
Extension toward object-oriented programming
These features are not emerging to the programmer and the user unless requested, but are provided (and extensible) in the EyesWeb visual language

Development of super-modal models for expressive gesture analysis in TAIs
Integration of techniques from partners in the TAI-CHI Project
Validation of the research results and assessment of the developed TAIs with experiments with users.