Skip to article
AHFE AHFE Discoveries AHFE International Open Access Journal
AHFE OPEN ACCESS OPEN ACCESS CONFERENCE PROCEEDINGS PAPER

Lip-Reading Research Based on ShuffleNet and Attention-GRU

School of Information Science and Technology, North China University of Technology, Beijing, 100144, China ↗
Publication year
Digital Object Identifier10.54941/ahfe1004024

Abstract

Human-computer interaction has seen a paradigm shift from textual or display-based control towards more intuitive control such as voice, gesture and mimicry. Particularly, speech recognition has attracted a lot of attention because it is the most prominent mode of communication. However, performance of speech recognition systems varies significantly according to sources of background noise, types of talkers and listener's hearing ability. Therefore, lip recognition technology which detects spoken words by tracking speaker's lip movements comes into being. It provides an alternative way for scenes with high background noise and people with hearing impaired problems. Also, lip reading technology has widespread application in public safety analysis, animation lip synthesis, identity authentication and other fields. Traditionally, most work in lipreading was based on hand-engineered features, that were usually modeled by HMM-based pipeline. Recently, deep learning methods are deployed either for extracting 'deep' features or for building end-to-end architectures. In this paper, we propose a neural network architecture combining convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) with a plug-in attention mechanism. The model consists of five parts: (1). Input: We use Dlib library for detecting 68 landmarks of the face, crop the lip area and extract 29 consecutive frames from the video sequence. The frames go through a simple C3D network for generic feature extraction. (2). CNN: with neural networks becoming deeper and deeper, computation complexity increases significantly as well, which motivated the appearance of the lightweight model architecture design. Then a lightweight CNN named ShuffleNet pre-trained on ImageNet dataset is used in our method to perform spatial downsampling of a single image. The ShuffleNet mainly uses two new operations, namely, pointwise group convolution and channel shuffle, which greatly reduce the computational cost without affecting recognition accuracy. (3) CBAM: In the field of image processing, a feature map contains a variety of important information. The traditional convolutional neural network performs convolution in the same way on all channels but importance of information varies greatly depending on different channels. To improve the performance of convolutional neural networks for feature extraction, we utilize an attention mechanism named Convolutional Block Attention Module (CBAM), which is a simple and effective attention module for feedforward convolutional neural networks and contains two independent sub-modules, namely, Channel Attention Module (CAM) and Spatial Attention Module (SAM), which perform Channel and Spatial Attention respectively. (4) RNN: The traditional Recurrent Neural Network (RNN) is mainly used to process sequential data, but with the extension of the RNN network, it may be unable to connect to all related information which may cause key information loss. It cannot solve the long-distance dependence problem and the performance may drop significantly. Due to this shortcoming of the traditional RNN network, we select the GRU network in this paper, which is a variant of the LSTM. It has a simpler structure and better performance than the LSTM neural network. (5) Outputs: Lastly, we pass the result of the backend to SoftMax for classifying the final word. In our experiment, we compare several model architectures and find that our model achieves a comparable accuracy to the current state-of-the-art model at a lower computational cost.

© 2023. Published by AHFE Open Access. All rights reserved. 194

Lip-Reading Research Based on ShuffleNet and Attention-GRU 195

INTRODUCTION

Lip reading technology, also known as visual speech recognition or speech reading, is an innovative field that focuses on interpreting human speech through the analysis of lip movements and facial expressions. This technol- ogy has advanced significantly in recent years, largely due to breakthroughs in artificial intelligence (AI), computer vision, and deep learning algorithms. The primary goal of lip- reading technology is to bridge the communica- tion gap for individuals with hearing impairments, enabling them to engage more effectively in conversations and social interactions. Additionally, this technology can be utilized in noisy environments, where audio-based speech recognition may struggle to perform accurately. Furthermore, lip reading technology has the potential to enhance security measures, assist in voice- driven device control, and improve teleconferencing experiences. The lip- reading technology has developed through key stages leveraging artificial intelligence, computer vision, and machine learning. First, a diverse dataset of video recordings is collected, capturing various languages, accents, and environments. Next, data preprocessing isolates the relevant lip and facial features. Then, machine learning algorithms, such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs), are employed to train models for recognizing and deciphering lip movements. As these models are continuously trained and refined with new data, their perfor- mance improves. Finally, integration with applications and devices allows the technology to assist in real-world communication and accessibility scenarios.

LIP-READING MODEL ARCHITECTURE The lip-reading technique primarily features two interconnected components: a frontend network and a backend network. The frontend networks com- prise a variety of architectures, such as MobileNet, ShuffleNet, VGGNet, GoogLeNet, and ResNet, while the backend networks encompass Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) models. An attention mechanism is employed to optimize resource allocation by priori- tizing crucial information in the feature map. Our proposed system integrates ShuffleNet, CBAM, and GRU, with the architecture depicted in Figure 1.

Figure 1: Architecture of the lip-reading model.
Original PDF Figure 1:

196 Lu and Fu

The frontend network is ShuffleNet. It is an optimized convolutional neu- ral network (CNN) architecture tailored for mobile and embedded systems. It was proposed by Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun in 2017, aiming to strike a balance between computational efficiency and performance, making it well-suited for real-time applications on devices with limited processing power and energy restrictions.The primary innova- tion of ShuffleNet lies in its “channel shuffle” operations, which enhance the flow of information between feature channels in the network. To achieve this, the input feature channels are split into several groups, and the chan- nels within and across these groups are then rearranged. This process enables more effective cross-group information sharing without substantially increas- ing the computational burden. In addition to channel shuffling, ShuffleNet incorporates depthwise separable convolutions and pointwise group convo- lutions to further minimize computational complexity. Depthwise separable convolution distinguishes between spatial and channel-wise convolutions, resulting in considerable computational savings compared to conventional convolution layers. Pointwise group convolutions, on the other hand, segre- gate channels into groups and apply convolutions individually within each group, further contributing to reduced computation.

Table 1. Measurement of accuracy and GPU speed of four different lightweight models with the same level of FLOPs on COCO object detection. (FLOPs: float-point operations.)

Table available in the original PDF paper.

Model mmAP (%) GPU Speed (Images/s) FLOPs 40M 140M 300M 500M 40M 140M 300M 500M

Xception 21.9 29.0 31.3 32.9 178 131 101 83 ShuffleNet v1 20.9 27.0 29.9 32.9 152 85 76 60 MobileNet v2 20.7 24.4 30.0 30.6 146 111 94 72 ShuffleNet v2 23.7 29.6 32.2 34.2 183 138 105 83

Table 1 demonstrates that among the four distinct architectures, Shuf- fleNet V2 outperforms the others in terms of both accuracy and speed. As a result, we employ ShuffleNet V2 as the front-end component of our model. This architecture primarily incorporates two innovative operations: pointwise group convolution and channel shuffle. These operations signif- icantly decrease the computational burden while maintaining the model’s recognition accuracy.

The attention mechanism is Convolution Block Attention Mechanism (CBAM). It is an attention-based technique developed to boost the perfor- mance of convolutional neural networks (CNNs) by adaptively focusing on critical features in the input. Introduced in 2018 by Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon, CBAM is a versatile and lightweight component that can be effortlessly incorporated into existing CNN struc- tures without substantially impacting model complexity or computational demand. CBAM comprises two consecutive attention mechanisms: channel attention and spatial attention. Channel attention aims to emphasize relevant

Lip-Reading Research Based on ShuffleNet and Attention-GRU 197

channels by leveraging the relationships between channels. This is achieved through global average pooling and global max pooling operations, followed by a common multi-layer perceptron (MLP). The generated channel-wise attention map is subsequently multiplied with the input feature map to accen- tuate crucial channels. After applying channel attention, the spatial attention mechanism concentrates on capturing spatial information by examining the relationships between spatial features within the feature map. It employs both average and max pooling operations over the channels, merges the results, and processes the combination through a convolutional layer with a sigmoid activation function. This process yields a spatial attention map, which is then element-wise multiplied with the channel-focused feature map.

The backend network is Gated Recurrent Unit (GRU). It is a recurrent neu- ral network (RNN) architecture introduced in 2014. GRUs were designed to address the vanishing gradient problem often encountered in traditional RNNs, which hampers the network’s ability to learn long-term dependencies in sequences. GRU has a simpler structure than the Long Short-Term Memory (LSTM) architecture, another widely-used RNN variant, while still offering comparable performance. GRU contains two gates: the reset gate and the update gate. These gates control the flow of information inside the hidden state, enabling the model to capture dependencies across varying time scales. The reset gate determines how much of the previous hidden state should be combined with the new input, allowing the model to decide the extent of the past information to be retained. The update gate, on the other hand, con- trols the extent to which the hidden state is updated with new information, balancing between maintaining the old hidden state and incorporating new insights.

EXPERIMENTAL SETTINGS AND RESULTS In our study, we employ the OpenCV library to extract 29 frames from each video, focusing on the lip region by identifying 68 facial landmarks using the dlib library. We then resize all frames to 112x112 pixels, normalize them, and convert them to grayscale. To build and train our models, we utilize the open-source Tensorflow and Keras libraries, which offer user-friendly APIs.

Our model is trained on servers equipped with four NVIDIA Titan X GPUs. We divide the dataset into training and testing subsets at an 8:2 ratio, setting the epoch and batch size at 6032. The Adam optimizer is employed with an initial learning rate of 3 × 10-4. Both the frontend and backend of our network are pre-trained on the LRW dataset. During training, we apply dropout with a 0.5 probability, and we measure our model’s performance using the standard Cross Entropy loss.

In the Figure 2, as illustrated by the bar chart displaying accuracy and recall rates, the lowest recall rate for lip sign language recognition is associ- ated with the number 0, followed by numbers 3 and 7. The accuracy metric for number 7 is markedly lower compared to the other nine digits. This is pri- marily attributed to the similarity in gestures between numbers 0 and 7. In the lip recognition system, numbers 0 and 7 exhibit low recall rates, while num- bers 6, 7, and 8 have low accuracy rates. This is mainly due to the analogous

198 Lu and Fu

Figure 2: Prediction of pronunciation of English numbers.
Original PDF Figure 2:

lip movement patterns when pronouncing numbers 6, 7, and 8 in English. Consequently, to enhance their accuracy and recall rates, we should augment the training sample size for these specific numbers in the future.

CONCLUSION

Lip recognition technology offers significant potential and application demand in the realms of computer vision and human-computer interaction. Specifically, employing automatic lip recognition technology to enhance the social interactions of individuals with hearing and speech impairments is one of the most promising applications of artificial intelligence in healthcare and rehabilitation. Therefore, Our research primarily utilizes a combination of a lightweight network named ShuffleNet which dramatically reduces com- putational power for practical scenarios and GRU network. Additionally, we insert an attention mechanism into GRU to focus on more important information in feature maps. Furthermore, to validate the algorithm model’s effectiveness, we appropriately apply a lab-generated test dataset for exper- imental testing. The result demonstrates that our model not only achieves comparable recognition accuracy but also reduces computational demands. In the future, we will do more research to improve the speed and accuracy of the model and expand self-built datasets for more tests.

ACKNOWLEDGMENT

This research was supported by the National Natural Science Foundation of China (61971007 and 61571013).

Lip-Reading Research Based on ShuffleNet and Attention-GRU 199

REFERENCES

Assael, Y. M.; Shillingford, B.; Whiteson, S. Lipnet: End-to-end sentence-level lipreading. arXiv 2016, arXiv:1611.01599. Bai, S.; Kolter, J. Z.; Koltun, V. An empirical evaluation of generic convolutional and

recurrent networks for sequence modeling. arXiv 2018, arXiv:1803.01271. He, K.; Zhang, X.; Ren, S.; Sun, J. Deep residual learning for image recognition.

In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778. Hussein, D.; Ibrahim, D. M.; Sarhan, A. M. HLR-Net: A Hybrid Lip-Reading Model Based on Deep Convolutional Neural Networks. Comput. Mater. Contin. 2021, 68, 1531–1549. Lu, H.; Liu, X.; Yin, Y.; Chen, Z. A Patent Text Classification Model Based on

Multivariate Neural Network Fusion. In Proceedings of the 2019 6th International Conference on Soft Computing & Machine Intelligence (ISCMI), Johannesburg, South Africa, 19–20 November 2019. Mcgurk, H.; Macdonald, J. Hearing lips and seeing voices. Nature 1976, 264, 746–748. Nazari, K.; Ebadi, M. J.; Berahmand, K. Diagnosis of alternaria disease and leafminer

pest on tomato leaves using image processing techniques. J. Sci. Food Agric. 2022, 102, 6907–6920. Palecek, K. Utilizing lipreading in large vocabulary continuous speech recognition.

In Proceedings of the International Conference on Speech and Computer, Hat- field, UK, 12–16 September 2017; Springer: Berlin, Germany; Cham, Switzerland, 2017; pp. 767–776. Pattern Recognition, Columbus, OH, USA, 23–28 June 2014; pp. 580–587. Rostami, M.; Berahmand, K.; Nasiri, E.; Forouzandeh, S. Review of swarm intelligence-based feature selection methods. Eng. Appl. Artif. Intell. 2021, 100, 104210. Saberi-Movahed, F.; Rostami, M.; Berahmand, K.; Karami, S.; Tiwari, P.; Ous-

salah, M.; Band, S. S. Dual Regularized Unsupervised Feature Selection Based on Matrix Factorization and Minimum Redundancy with application in gene selection. Knowl. Based Syst.2022, 256, 109884. Szegedy, C.; Liu, W.; Jia, Y.; Sermanet, P.; Reed, S.; Anguelov, D.; Erhan, D.; Vincent, V.; Andrew, R. Going deeper with convolutions. In Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7–12 June 2015; pp. 1–9.

Cover of the proceedings volume Human Interaction & Emerging Technologies (IHIET 2023): Artificial Intelligence & Future Applications

Explore the complete digital volume

Human Interaction & Emerging Technologies (IHIET 2023): Artificial Intelligence & Future Applications

Read this individual research paper free through the original publisher, or explore the full digital proceedings book.

ISBN 978-1-958651-87-2

Explore this ISBN in the AHFE Bookstore Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional

Explore Related Research

Related research 8 matches

More papers from this volume

← Back to Research ExplorerBrowse the full library →