We propose a bimodal-DFSMN to jointly learn feature fusion and acoustic modeling, and utilize a per-frame dropout approach to enhance the robustness of AVSR system against the missing of visual modality. In this work, based on the recently released NTCD-TIMIT audiovisual corpus, we address the challenges of AVSR through three aspects: 1) optimal integration of acoustic and visual information 2) robust performance with multi-condition training 3) robust modeling against missing visual information during decoding. Compared to audio only speech recognition, the major issues of AVSR include the lack of publicly available audiovisual corpora and the need of robust knowledge fusion of both speech and vision. We also show that the proposed unsupervised audio-visual speech enhancement approach outperforms a state-of-the-art supervised deep learning method.Īudiovisual speech recognition (AVSR) is thought to be one of the potential solutions for robust speech recognition, especially in noisy environments. The results confirm that the proposed audio-visual CVAE effectively fuse audio and visual information, and it improves the speech enhancement performance compared with the audio-only VAE model, especially when the speech signal is highly corrupted by noise. Experiments are conducted with the recently published NTCD-TIMIT dataset. At test time, the audio-visual speech generative model is combined with a noise model based on nonnegative matrix factorization, and speech enhancement relies on a Monte Carlo expectation-maximization algorithm. We develop a conditional VAE (CVAE) where the audio speech generative process is conditioned on visual information of the lip region. In this paper, we propose audio-visual variants of VAEs for single-channel and speaker-independent speech enhancement.

One advantage of this generative approach is that it does not require pairs of clean and noisy speech signals at training. VAEs have been successfully used to learn a probabilistic prior over speech signals, which is then used to perform speech enhancement. Variational auto-encoders (VAEs) are deep generative latent variable models that can be used for learning the distribution of complex data. Using NTCD-TIMIT with its freely available visual features and 37 clean and noisy acoustic signals allows for this study to be a common benchmark, to which novel LVCSR AV-ASR models and approaches can be compared. The task of NTCD-TIMIT is phone recognition in continuous speech. All experiments have been applied to the recently released NTCD-TIMIT audio-visual corpus.

A complete evaluation of these fusion models is conducted using a standard speaker-independent DNN-based LVCSR Kaldi recipe in three experimental setups: a clean-train-clean-test, a clean-train-noisy-test, and a matched-training setup. This paper reviews and compares the performance of five audio-visual fusion models: the feature fusion model, the decision fusion model, the multi-stream hidden Markov model (HMM), the coupled HMM, and the turbo decoders. Even less research work compares audio-visual fusion models for large vocabulary continuous speech recognition (LVCSR) models using deep neural networks (DNNs). However, very few studies can be found in the literature that compare different fusion models for AV-ASR. In the last few decades, many approaches for integrating the audio and video modalities were proposed to enhance the performance of automatic speech recognition in both clean and noisy conditions.

Audio-visual fusion is one of the most challenging tasks that continues to attract substantial research interest in the field of audio-visual automatic speech recognition (AV-ASR).

Timit tor meaning