Implementation of BERT in Sentiment Analysis of National Digital Samsat (SIGNAL) User Reviews Based on Machine Learning

Ratna Savitri; Fido Rizki; Ahmad Sobri

doi:10.18860/mat.v17i2.32059

Implementation of BERT in Sentiment Analysis of National Digital Samsat (SIGNAL) User Reviews Based on Machine Learning

Ratna Savitri, Fido Rizki, Ahmad Sobri

Abstract

The SIGNAL application facilitates online vehicle tax payments for the public. The application's quality is frequently evaluated through user reviews on platforms like the Google Play Store. This study aims to analyze the sentiment of SIGNAL user reviews using a Machine Learning-based approach, specifically the BERT (Bidirectional Encoder Representations from Transformers) model. The dataset consists of 20,000 user reviews. After preprocessing, the remaining data comprises 17,287 reviews, categorized into 12,758 positive reviews, 2,160 neutral reviews, and 2,369 negative reviews. To address data imbalance, the Random Over Sampling (ROS) technique was applied. The evaluation was performed using metrics such as accuracy, precision, recall, and F1-score. The results of the study indicate that the IndoBERT model can classify sentiments with an accuracy of 99% and a validation accuracy of 98% after five epochs of training. Confusion matrix analysis shows that the model achieved an overall accuracy of 99.72% on training data and 98.68% on testing data. This study demonstrates that the IndoBERT model is highly effective in classifying sentiment and makes a significant contribution to understanding the user experience of SIGNAL, which can serve as a foundation for future improvements to the application.

Full Text:

PDF

References

[1] W. A. Rahmadhani et al., “Pemanfaatan Website Sebagai Bentuk Digitalisasi Pelayanan Publik Untuk Mewujudkan Transparansi di Dinas Sosial Provinsi Sumatera Utara dan Pemanfaatan Digitalisasi Pada Pendidikan Islam,” Edukasi Islam. J. Pendidik. Islam, vol. 11, no. 1, pp. 1167–1182, 2022, doi: 10.30868/ei.v11i01.2979.

[2] Republik Indonesia, Instruksi Presiden No. 6 Tahun 2001 Tentang Pengembangan Dan Pendayagunaan Telematika Di Indonesia Presiden Republik Indonesia. Indonesia, 2001, pp. 1–11. [Online]. Available: https://jdih.esdm.go.id/peraturan/INPRES NO 6 TH 2001.pdf

[3] N. Nurzaman, N. Suarna, and W. Prihartono, “Analisis Sentimen Ulasan Aplikasi Threads Di Google Playstore Menggunakan Algoritma Naïve Bayes,” JATI (Jurnal Mhs. Tek. Inform., vol. 8, no. 1, pp. 967–974, 2024, doi: 10.36040/jati.v8i1.8708.

[4] R. Indoensia, “Undang Undang Republik Indonesia No. 22 Tahun 2009 Tentang Lalu Lintas dan Angkatan Umum,” vol. 19, p. 19, 2009.

[5] S. Kacung, C. Pamungkas Putra Bagyana, and D. Cahyono, “Analisis Sentimen Terhadap Layanan Samsat Digital Nasional (Signal) Menggunakan Metode Svm,” J. Mnemon., vol. 7, no. 1, pp. 118–122, 2024, doi: 10.36040/mnemonic.v7i1.9557.

[6] I. F. Rahman, A. N. Hasanah, and N. Heryana, “Analisis Sentimen Ulasan Pengguna Aplikasi Samsat Digiital Nasional (Signal) Dengan Menggunakan Metode Naïve Bayes Classifier,” J. Inform. dan Tek. Elektro Terap., vol. 12, no. 2, pp. 963–969, 2024, doi: 10.23960/jitet.v12i2.4073.

[7] F. I. Septian, H. Ivana Lucia Kharisma, and Kamdan, “Implementasi Metode Bidirectional Encoder Representations from Transformers ( BERT ) untuk Analisis Sentimen Komentar Pengguna,” vol. 3, no. 1, 2023.

[8] R. Kusnadi, Y. Yusuf, A. Andriantony, R. Ardian Yaputra, and M. Caintan, “Analisis Sentimen Terhadap Game Genshin Impact Menggunakan Bert,” Rabit J. Teknol. dan Sist. Inf. Univrab, vol. 6, no. 2, pp. 122–129, 2021, doi: 10.36341/rabit.v6i2.1765.

[9] K. Cindy Pradhisa and R. Fajriyah, “Analisis Sentimen Ulasan Pengguna E-commerce di Google Play Store Menggunakan Metode IndoBERT,” Technol. Sci., vol. 6, no. 1, pp. 92–104, 2024, doi: 10.47065/bits.v6i1.5247.

[10] R. Kurniawan, H. O. L. Wijaya, and R. P. Aprisusanti, “Sentiment Analysis of Google Play Store User Reviews on Digital Population Identity App Using K-Nearest Neighbors,” J. Sisfokom (Sistem Inf. dan Komputer), vol. 13, no. 2, pp. 170–178, 2024, doi: 10.32736/sisfokom.v13i2.2071.

[11] Ardi Mursyidi, “Penerapan Bidirectional Encoder Representations From Transformers (Bert) Untuk Analisis Sentimen Vaksin Covid-19 Pada Twitter,” UIN Suska Riau, 2023.

[12] R. Mas, R. W. Panca, K. Atmaja1, and W. Yustanti2, “Analisis Sentimen Customer Review Aplikasi Ruang Guru dengan Metode BERT (Bidirectional Encoder Representations from Transformers),” Jeisbi, vol. 02, no. 3, pp. 55–62, 2021.

[13] M. Haris, A. Suharso, E. H. Nurkifli, P. S. Informatika, U. S. Karawang, and T. Timur, “ANALISIS SENTIMEN PADA GAME EFOOTBALL DI GOOGLE PLAY STORE MENGGUNAKAN ALGORITMA INDOBERT,” vol. 8, no. 6, pp. 12108–12121, 2024.

[14] Ms. Elvis F. Purba, SE and P. . Parulian Simanjuntak, MA, E-Book Metodologi Penelitian, vol. 11, no. 1. 2011, 2011. [Online]. Available: http://scioteca.caf.com/bitstream/handle/123456789/1091/RED2017-Eng-8ene.pdf?sequence=12&isAllowed=y%0Ahttp://dx.doi.org/10.1016/j.regsciurbeco.2008.06.005%0Ahttps://www.researchgate.net/publication/305320484_SISTEM_PEMBETUNGAN_TERPUSAT_STRATEGI_MELESTARI

[15] B. Kurniawan, A. Ari Aldino, and A. Rahman Isnain, “Sentimen Analisis terhadap Kebijakan Penyelenggara Sistem Elektronik (PSE) Menggunakan Algoritma Bidirectional Encoder Representations from Transformers (Bert),” J. Teknol. dan Sist. Inf., vol. 3, no. 4, pp. 98–106, 2022, [Online]. Available: http://jim.teknokrat.ac.id/index.php/JTSI

DOI: https://doi.org/10.18860/mat.v17i2.32059

Refbacks

There are currently no refbacks.

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

The journal is indexed by :

_______________________________________________________________________________________________________________

Editorial Office:
Informatics Engineering Department
Faculty of Science and Technology
Universitas Islam Negeri Maulana Malik Ibrahim Malang
Jalan Gajayana 50 Malang, Jawa Timur, Indonesia 65144
Email: matics@uin-malang.ac.id
_______________________________________________________________________________________________________________

Username
Password
Remember me