Semantic Sound-to-Haptic Automatic Conversion: Metaverse, Full-body Haptic Effects, and Accessibility

We develop a real-time, semantic sound-to-haptic conversion algorithm and investigate related human perceptual performances to enhance VR experiences. This also aims to enable people with hearing impairments as well as non-disabled people to interact at the same level as possible with VR. With our system, users can enjoy multi-modal haptic effects that match the sounds in VR across their whole body.

​​​

- Perceptual performance investigation and experiment for sound-haptic stimulation

- Development of a supervised learning-based sound classifier for sound-haptic conversion

- Development of a semantic sound to temporal haptic effect conversion system

- Development of a sound-to-spatiotemporal haptic effect conversion system based on sound source location estimation

- Application and improvement for the hearing impaired