COMPLEX METHOD FOR AUTOMATIC RECOGNITION OF NATURAL LANGUAGE AND EMOTIONAL STATE
DOI:
https://doi.org/10.28925/2663-4023.2023.19.146164Keywords:
automatic speech recognition; ASR; NLP; recurrent neural network; RNNAbstract
Current trends in NLP emphasize universal models and learning from pre-trained models. This article explores these trends and advanced models of pre-service learning. Inputs are converted into words or contextual embeddings that serve as inputs to encoders and decoders. The corpus of the author's publications over the past six years is used as the object of the research. The main methods of research are the analysis of scientific literature, prototyping, and experimental use of systems in the direction of research. Speech recognition players are divided into players with huge computing resources for whom training on large unlabeled data is a common procedure and players who are focused on training small local speech recognition models on pre-labeled audio data due to a lack of resources. Approaches and frameworks for working with unlabeled data and limited computing resources are almost not present, and methods based on iterative training are not developed and require scientific efforts for development. The research aims to develop methods of iterative training on unlabeled audio data to obtain productively ready speech recognition models with greater accuracy and limited resources. A separate block proposes methods of data preparation for use in training speech recognition systems and a pipeline for automatic training of speech recognition systems using pseudo marking of audio data. The prototype and solution of a real business problem of emotion detection demonstrate the capabilities and limitations of owl recognition systems and emotional states. With the use of the proposed methods of pseudo-labeling, it is possible to obtain recognition accuracy close to the market leaders without significant investment in computing resources, and for languages with a small amount of open data, it can even be surpassed.
Downloads
References
Vaswani, A., et al. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998–6008. http://arxiv.org/abs/1706.03762
Firth, J. R. (1957). A Synopsis of Linguistic Theory, 1930–1955.
Luscher, C., et al. (2019) RWTH ASR Systems for LibriSpeech: Hybrid vs Attention—w/o Data Augmentation, 1–5. http://arxiv.org/abs/1905.03072
Devlin, J., et al. (2019). Bert: Pre-Training of Deep Bidirectional Transformers for Language Understanding. http://arxiv.org/abs/1810.04805
Xu, Q., et al. (2020). Iterative Pseudo-Labeling for Speech Recognition, 1–13. https://arxiv.org/abs/2005.09267
Mikolov, T., et al. (2013). Efficient Estimation of Word Representations in Vector Space. In First International Conference on Learning Representations (pp. 1–13). http://arxiv.org/abs/1301.3781
Peters, M., et al. (2018). Deep Contextualized Word Representations. In 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 1 (pp. 2227–2237). https://doi.org/10.18653/v1/n18-1202
Cho, K., et al. (2014). On the Properties of Neural Machine Translation: Encoder–Decoder Approaches. In SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation. https://doi.org/10.3115/v1/w14-4012
Rumelhart, D. E., Hinton, G. E., Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323(6088), 533–536. https://doi.org/10.1038/323533a0
Goodfellow, I., Bengio, Y., Courville, A. (2016). Sequence modeling: recurrent and recursive nets, Deep Learning, 367–415.
Hochreiter, S., Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735
Tanaka, T., et al. (2019). A Joint End-to-End and DNN-HMM Hybrid Automatic Speech Recognition System with Transferring Sharable Knowledge. Interspeech, 2210–2214. https://doi.org/10.21437/interspeech.2019-2263
Wang, D., Wang, X., Lv, S. (2019). An Overview of End-to-End Automatic Speech Recognition. Symmetry, 11(8), 1018. https://doi.org/10.3390/sym11081018
Iosifov, I., Iosifova, O., Sokolov, V. (2020). Sentence Segmentation from Unformatted Text using Language Modeling and Sequence Labeling Approaches. In 2020 IEEE International Conference on Problems of Infocommunications. Science and Technology. https://doi.org/10.1109/picst51311.2020.9468084
Graves, A., et al. (2006). Connectionist Temporal Classification. In 23rd International Conference on Machine Learning (pp. 369–376). https://doi.org/10.1145/1143844.1143891
Hsiao, R. (2020). Online Automatic Speech Recognition with Listen, Attend and Spell Model, 1–5. http://arxiv.org/abs/2008.05514
McDermott, E. (2018). A deep generative acoustic model for compositional automatic speech recognition, In 32nd Conference on Neural Information Processing Systems (pp. 1–17).
Chan, W., et al. (2016). Listen, Attend and Spell: A Neural Network for Large Vocabulary Conversational Speech Recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing. https://doi.org/10.1109/icassp.2016.7472621
Iosifova, O., et al. (2020). Techniques comparison for natural language processing. In Proceedings of the 2nd International Workshop on Modern Machine Learning Technologies and Data Science, I(2631) (pp. 57–67).
Iosifova, O., et al. (2021). Analysis of Automatic Speech Recognition Methods. In Workshop on Cybersecurity Providing in Information and Telecommunication Systems (pp. 252–257).
Radford, A., et al. (2019). Language Models Are Unsupervised Multitask Learners. OpenAI Blog 1(8), 9.
Liu, Y., et al. (2019). Roberta: A robustly optimized bert pretraining approach. https://arxiv.org/abs/1907.11692
Sanh, V., et al. (2019). DistilBERT, A Distilled Version of BERT: Smaller, Faster, Cheaper and Lighter. NeurIPS. https://arxiv.org/abs/1910.01108
Yang, Z., et al. (2019). XLNet: Generalized Autoregressive Pretraining for Language Understanding. https://arxiv.org/abs/1906.08237
Li, T., et al. (2021). A Short Study on Compressing Decoder-Based Language Modelshttps://arxiv.org/abs/2110.08460
Brown, T. B., et al. (2020). Language Models are Few-Shot Learners. https://arxiv.org/abs/2005.14165
Dai, Z., (2019). Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context. https://arxiv.org/abs/1901.02860
Iosifov, I., et al. (2022). Natural Language Technology to Ensure the Safety of Speech Information. In Workshop on Cybersecurity Providing in Information and Telecommunication Systems II (pp. 216–226).
TensorFlow: The Functional API. https://www.tensorflow.org/guide/keras/functional
Apache Spark: ML Pipelines. https://spark.apache.org/docs/latest/ml-pipeline.html
Apache Airflow: DAGs. https://airflow.apache.org/docs/stable/concepts.html
Liao, H., McDermott, E., Senior, A. (2013). Large Scale Deep Neural Network Acoustic Modeling with Semi-Supervised Training Data for YouTube Video Transcription. In 2013 IEEE Workshop on Automatic Speech Recognition and Understanding (pp. 368–373). https://doi.org/10.1109/asru.2013.6707758
Romanovskyi, O., et al. (2021). Automated Pipeline for Training Dataset Creation from Unlabeled Audios for Automatic Speech Recognition. In Advances in Computer Science for Engineering and Education IV (pp. 25–36). https://doi.org/10.1007/978-3-030-80472-5_3
Georgescu, A.-L., et al. (2021). Performance vs. Hardware Requirements In State-of-the-Art Automatic Speech Recognition. EURASIP Journal on Audio, Speech, and Music Processing, 2021(1). https://doi.org/10.1186/s13636-021-00217-4
Dutta, A., Ashishkumar, G., Rao, Ch. V. R. (2021). Improving the Performance of ASR System by Building Acoustic Models using Spectro-Temporal and Phase-Based Features. Circuits, Systems, and Signal Processing, 41(3), 1609–1632. https://doi.org/10.1007/s00034-021-01848-w
Gondi, S., Pratap, V. (2021). Performance and Efficiency Evaluation of ASR Inference on the Edge. Sustainability, 13(22), 12392. https://doi.org/10.3390/su132212392
Li, S., et al. (2019). Improving Transformer-Based Speech Recognition Systems with Compressed Structure and Speech Attributes Augmentation. Interspeech, 2019. https://doi.org/10.21437/interspeech.2019-2112
Kuchaiev, O., et al. (2019). NeMo: A Toolkit for Building AI Applications using Neural Modules, 36–44. https://arxiv.org/abs/1909.09577
Web Framework Benchmarks. https://www.techempower.com/benchmarks/
#section=test&runid=7464e520-0dc2-473d-bd34-dbdfd7e85911
Pa Pa Win, H., Thu Thu Khine, P. (2020). Emotion Recognition System of Noisy Speech in Real World Environment. International Journal of Image, Graphics and Signal Processing, 12(2), 1–8. https://doi.org/10.5815/ijigsp.2020.02.01
Kumar, J. A., Balakrishnan, M., Wan Yahaya, W. A. J. (2016). Emotional Design in Multimedia Learning: How Emotional Intelligence Moderates Learning Outcomes. International Journal of Modern Education and Computer Science, 8(5), 54–63. https://doi.org/10.5815/ijmecs.2016.05.07
Dhar, P., Guha, S. (2021). A System to Predict Emotion from Bengali Speech. International Journal of Mathematical Sciences and Computing, 7(1), 26–35. https://doi.org/10.5815/ijmsc.2021.01.04
Shirani, A., Nilchi, A. R. N. (2016). Speech Emotion Recognition based on SVM as Both Feature Selector and Classifier. International Journal of Image, Graphics and Signal Processing, 8(4), 39–45. https://doi.org/10.5815/ijigsp.2016.04.05
Devi, J. S., Yarramalle, S., Prasad Nandyala, S. (2014). Speaker Emotion Recognition based on Speech Features and Classification Techniques. International Journal of Image, Graphics and Signal Processing, 6(7), 61–77. https://doi.org/10.5815/ijigsp.2014.07.08
Lech, M., et al. (2020). Real-Time Speech Emotion Recognition Using a Pre-trained Image Classification Network: Effects of Bandwidth Reduction and Companding. Frontiers in Computer Science, 2. https://doi.org/10.3389/fcomp.2020.00014
Livingstone, S. R., Russo, F. A. (2018). The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS): A dynamic, multimodal set of facial and vocal expressions in North American English. PLOS ONE, 13(5), e0196391. https://doi.org/10.1371/journal.pone.0196391
Pichora-Fuller, M. K., Dupuis, K. (2020). Toronto emotional speech set (TESS) [Data set]. Borealis. https://doi.org/10.5683/SP2/E8H2MF
Desplanques, B., Thienpondt, J., Demuynck, K. (2020). ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification. Interspeech, 2020. https://doi.org/10.21437/interspeech.2020-2650
Iosifov, I., et al. (2022). Transferability Evaluation of Speech Emotion Recognition Between Different Languages. In Advances in Computer Science for Engineering and Education (pp. 413–426). https://doi.org/10.1007/978-3-031-04812-8_35
Romanovskyi, O., et al. (2022). Prototyping Methodology of End-to-End Speech Analytics Software. In 4th International Workshop on Modern Machine Learning Technologies and Data Science (pp. 76–86).
