Audiovisual Speech Processing

When we speak, we configure the vocal tract which shapes the visible motions of the face and the patterning of the audible speech acoustics. Similarly, we use these visible and audible behaviors to perceive speech. This book showcases a broad range of research investigating how these two types of signals are used in spoken communication, how they interact, and how they can be used to enhance the realistic synthesis and recognition of audible and visible speech. The volume begins by addressing two important questions about human audio-visual performance: how auditory and visual signals combine to access the mental lexicon and where in the brain this and related processes take place. It then turns to the production and perception of multimodal speech and how structures are coordinated within and across the two modalities. Finally, the book presents overviews and recent developments in machine-based speech recognition and synthesis of AV speech.

Gérard Bailly is a Senior CNRS Research Director at the Speech and Cognition Department, GIPSA-Lab, University of Grenoble, where he is now Head of Department.

Pascal Perrier is a Professor in the GIPSA-Lab at the University of Grenoble.

Eric Vatikiotis-Bateson is Professor and Canada Research Chair in Linguistics and Cognitive Science in the Department of Linguistics at the University of British Colombia.