Pengenalan Emosi Manusia Melalui Suara Menggunakan Metode Convolutional Neural Network dan Transformer
Keywords:
speech emotion recognition, convolutional neural network, transformer, mel spectogram, deep learningAbstract
Communication between humans and machines currently lacks a natural emotional component, since systems generally understand verbal content without capturing the emotional meaning conveyed by tone and timbre. This study aims to design a Speech Emotion Recognition model by integrating Convolutional Neural Network (CNN) and Transformer architectures to capture spatial features as well as long-range temporal dependencies in speech signals. The dataset combines four public corpora, namely RAVDESS, CREMA-D, SAVEE, and TESS, comprising 12,798 audio files across seven emotion classes (angry, disgusted, fearful, happy, neutral, sad, and surprised). The audio signals were processed through Mel Spectrogram feature extraction along with delta and delta-delta coefficients, balanced through data augmentation, and trained using a 5-Fold Stratified Cross Validation scheme. Testing on an independent test set shows that the hybrid model achieves a 5-fold average accuracy of 75.80%, while the best-performing single-fold model reaches 77.00% accuracy with a Macro Average F1-Score of 78.00%, and all emotion classes obtain an AUC value above 0.90. Applying Ensemble Learning (Soft Voting) across the five fold models further improves the testing accuracy to 81.00% with a Macro Average F1-Score of 0.82, representing the highest overall result achieved in this study. These results demonstrate that the integration of CNN and Transformer is effective and generalizes well on heterogeneous speech data.
Downloads
References
H. A. Elfenbein, "Emotion in Organizations: Theory and Research," Annual Review of Psychology, vol. 74, pp. 489–517, 2022, doi: 10.1146/annurev-psych-032720.
U. Bilotti, C. Bisogni, M. De Marsico, and S. Tramonte, "Multimodal Emotion Recognition via Convolutional Neural Networks: Comparison of different strategies on two multimodal datasets," Engineering Applications of Artificial Intelligence, vol. 130, 2024, doi: 10.1016/j.engappai.2023.107708.
F. P. Anindya, D. E. Herwindiati, and N. J. Perdana, "Pengenalan Suara Manusia Menggunakan Support Vector Classifier (SVC) untuk Proses Otentikasi," Computatio: Journal of Computer Science and Information Systems, vol. 7, no. 1, 2023.
T. B. Putri, S. Saidah, B. Hidayat, F. Qothrunnada, and D. Darwindra, "Deteksi Emosi Berdasarkan Sinyal Suara Manusia Menggunakan Discrete Wavelet Transform (DWT) dengan Klasifikasi Support Vector Machine (SVM)," Jurnal Ilmu Komputer dan Informatika, vol. 3, no. 1, pp. 1–10, 2023, doi: 10.54082/jiki.45.
R. Meiyanti, "Pendeteksi Pengenalan Emosi pada Manusia Menggunakan Hidden Markov Model dan Bidirectional Associative Memory dengan Suara," Jurnal TIKA, vol. 6, pp. 231–237, 2021, doi: 10.51179/tika.v6i03.756.
M. Hosain, M. Yeasmin Arafat, G. Zahirul Islam, J. Uddin, M. Mobarak Hossain, and F. Alam, "Emotional Expression Detection in Spoken Language Employing Machine Learning Algorithms," n.d.
E. H. Houssein, A. Hammad, and A. A. Ali, "Human emotion recognition from EEG-based brain–computer interface using machine learning: a comprehensive review," Neural Computing and Applications, vol. 34, no. 15, pp. 12527–12557, 2022, doi: 10.1007/s00521-022-07292-4.
I. M. A. Wirawan, K. Y. E. Aryanto, I. N. Sukajaya, N. N. M. Agustini, and D. A. W. M. Putri, "Hybrid method for optimizing emotion recognition models on electroencephalogram signals," IAES International Journal of Artificial Intelligence, vol. 14, no. 3, pp. 2302–2314, 2025, doi: 10.11591/ijai.v14.i3.pp2302-2314.
I. M. A. Wirawan, R. Wardoyo, and D. Lelono, "The challenges of emotion recognition methods based on electroencephalogram signals: A literature review," International Journal of Electrical and Computer Engineering, vol. 12, no. 2, pp. 1508–1519, 2022, doi: 10.11591/ijece.v12i2.pp1508-1519.
I. M. A. Wirawan and I. M. A. Indrawan, "Subject Independent Emotion Recognition Using Electroencephalogram Signals with Continuous Capsule Network Method," JUITA, vol. 13, no. 1, pp. 37–46, 2025.
I. M. A. Wirawan, I. G. M. Darmawiguna, and I. B. N. Pascima, "Improving baseline reduction for emotion recognition based on electroencephalogram signals," IAES International Journal of Artificial Intelligence, vol. 13, no. 4, pp. 4263–4272, 2024, doi: 10.11591/ijai.v13.i4.pp4263-4272.
A. S. Alluhaidan, O. Saidani, R. Jahangir, M. A. Nauman, and O. S. Neffati, "Speech Emotion Recognition through Hybrid Features and Convolutional Neural Network," Applied Sciences (Switzerland), vol. 13, no. 8, 2023, doi: 10.3390/app13084750.
A. Ashraf, T. S. Gunawan, F. Arifin, M. Kartiwi, A. Sophian, and M. H. Habaebi, "On the Audio-Visual Emotion Recognition using Convolutional Neural Networks and Extreme Learning Machine," Indonesian Journal of Electrical Engineering and Informatics, vol. 10, no. 3, pp. 684–697, 2022, doi: 10.52549/ijeei.v10i3.3879.
K. Chauhan, K. K. Sharma, and T. Varma, "Speech Emotion Recognition Using Convolution Neural Networks," in Proc. Int. Conf. Artificial Intelligence and Smart Systems (ICAIS), 2021, pp. 1176–1181, doi: 10.1109/ICAIS50930.2021.9395844.
F. Jonatan Tanudjaja, E. Y. Puspaningrum, and V. Via, "Klasifikasi Jenis Emosi Melalui Ucapan Menggunakan Metode Convolutional Neural Network," Teknologi: Jurnal Ilmiah Sistem Informasi, vol. 13, no. 2, pp. 1–11, 2023, doi: 10.26594/teknologi.v13i1.3740.
R. Nurcahyo and M. Iqbal, "Speaker Emotion Recognition Using Convolutional Neural Networks," Jurnal RESTI, vol. 6, no. 1, pp. 115–122, 2022, doi: 10.29207/resti.v6i1.3726.
T. Praveen Blessington, D. Hadke, S. Chavan, P. Chavan, and P. Ingulkar, "Recognition of Emotion through Speech Using Deep Convolution and LSTM," International Journal of Creative Research Thoughts, vol. 11, 2023.
Y. Zhao and X. Shu, "Speech emotion analysis using Convolutional Neural Network (CNN) and gamma classifier-based error correcting output codes (ECOC)," Scientific Reports, vol. 13, no. 1, 2023, doi: 10.1038/s41598-023-47118-4.
F. Kasyidi, R. Ilyas, and N. M. Annisa, "Peningkatan Kemampuan Pengenalan Emosi Melalui Suara dalam Bahasa Indonesia," MIND Journal, vol. 6, no. 2, pp. 194–204, 2021, doi: 10.26760/mindjournal.v6i2.194-204.
C. N. Govardhini, A. Venu, K. Lakshminarayana, J. S. M. Kumar, and A. K. Rout, "Emotion Recognition Using Deep Learning Techniques for User Experience," International Journal of Research Publication and Reviews, vol. 4, no. 10, pp. 2644–2653, 2023, doi: 10.55248/gengpi.4.1023.102819.
R. Rastogi, T. Anand, S. K. Sharma, and S. Panwar, "Emotion Detection via Voice and Speech Recognition," International Journal of Cyber Behavior, Psychology and Learning, vol. 13, no. 1, 2023, doi: 10.4018/IJCBPL.333473.
A. Yazdani, H. Simchi, and Y. Shekofteh, "Speech Emotion Recognition Based on Deep Learning," in Proc. 2021 11th Int. Conf. Computer Engineering and Knowledge (ICCKE), IEEE, 2021.
D. K. R. P. Wisnu, I. K. Purnamawan, and L. J. E. Dewi, "Analisis Komparasi Performa Arsitektur VGG16 dan CNN Kustom pada Klasifikasi Citra Buah Melon Berdasarkan Gambar Kulit," Jurnal Informatika Teknologi dan Sains (JINTEKS), vol. 8, no. 1, pp. 438–448, 2026, doi: 10.51401/jinteks.v8i1.
K. Mountzouris, I. Perikos, and I. Hatzilygeroudis, "Speech Emotion Recognition Using Convolutional Neural Networks with Attention Mechanism," Electronics (Switzerland), vol. 12, no. 20, 2023, doi: 10.3390/electronics12204376.
R. Ullah et al., "Speech Emotion Recognition Using Convolution Neural Networks and Multi-Head Convolutional Transformer," Sensors, vol. 23, no. 13, 2023, doi: 10.3390/s23136212.
G. Akilandasowmya and M. Mayuranathan, "Enhancing speech-based emotion recognition with transformer models and attention mechanisms," International Journal of Speech Technology, vol. 29, no. 3, p. 55, 2026, doi: 10.1007/s10772-026-10280-x.
H. Barik and R. Sridhar, "Inception-based speech emotion recognition," Multimedia Tools and Applications, vol. 85, no. 7, p. 603, 2026, doi: 10.1007/s11042-026-21760-2.
A. Chakhtouna, S. Sekkate, and A. Adib, "Enhancing speech emotion recognition using statistical self-supervised embeddings," Multimedia Tools and Applications, vol. 85, no. 8, p. 634, 2026, doi: 10.1007/s11042-026-21812-7.
N. Banerjee, N. Sethi, and S. Borah, "Deep analysis of MFCC and MEL spectrogram features to recognize and classify stuttered speech," Multimedia Tools and Applications, vol. 84, no. 35, pp. 43827–43846, 2025, doi: 10.1007/s11042-025-20881-4.
R. N. Bashir, M. A. Shahid, T. Rashid, M. Faheem, T. Saidani, O. Saidani, and A. R. Khan, "Voice pathology identification using mel spectrogram features and deep learning," Signal, Image and Video Processing, vol. 19, no. 11, p. 909, 2025, doi: 10.1007/s11760-025-04527-4.
N. K. A. Artalia, I. M. D. Maysanjaya, and P. H. Suputra, "Sentiment Analysis of Bali's Rapid Tourism Growth Using IndoBERT with INSET Lexicon Automatic Labelling," vol. 15, no. 1, 2026.
N. L. P. R. Dewi, I. N. S. W. Wijaya, I. K. Purnamawan, and N. W. Marti, "Model Classifier Judul Berita Pariwisata Indonesia Berdasarkan Sentimen," Jurnal Teknologi Informasi dan Ilmu Komputer, vol. 11, no. 1, pp. 117–124, 2024, doi: 10.25126/jtiik.20241117617.
M. Zielonka, A. Piastowski, A. Czyżewski, P. Nadachowski, M. Operlejn, and K. Kaczor, "Recognition of Emotions in Speech Using Convolutional Neural Networks on Different Datasets," Electronics (Switzerland), vol. 11, no. 22, 2022, doi: 10.3390/electronics11223831.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Ida Bagus Sebali Bayu Nandana, Ni Putu Novita Puspa Dewi, I Nyoman Saputra Wahyu Wijaya
This work is licensed under a Creative Commons Attribution 4.0 International License.
Universitas Harapan Medan
.png)
.png)
.png)
.png)
.png)
