A Hybrid Classification Model Based on BERT for Multi-Class Sentiment Analysis on Twitter
DOI:
https://doi.org/10.26555/jiteki.v11i2.30665Keywords:
Sentiment Analysis, BERT, LTSM, CNN, Emotion ClassificationAbstract
Social media is one of the media to convey opinions and sentiments. Sentiment analysis is an important tool for researchers and business people to understand user emotions efficiently and accurately. Choosing the right classification model has a significant impact on sentiment classification performance. However, the diversity of model architectures and training techniques poses its own challenges. In addition, relying on a single classification model often causes noise, bias, data imbalance, and limitations in handling data variations effectively. This study proposes a hybrid classification model where BERT is the baseline. Furthermore, BERT will be hybridized using LSTM, and BERT is hybridized with CNN to improve sentiment analysis on Twitter social media data. The hybrid approach aims to reduce the limitations of a single model classifier by increasing model effectiveness, reducing bias, and optimizing the model on imbalanced data. The following are the steps in this study, data preprocessing, data balancing, tokenization, model training, and performance evaluation. Three models were trained: the baseline BERT model, the BERT-CNN hybrid, and the BERT-LSTM hybrid. Model performance was assessed using accuracy, precision, recall, and F1 score. Experimental results show that the baseline BERT model achieves an accuracy of 91.45%, while BERT-LSTM achieves 91.60%, and BERT-CNN achieves the highest accuracy of 91.80%. However, further analysis is needed to determine whether these improvements are statistically significant and whether the hybrid model offers additional benefits beyond accuracy, such as remembering underrepresented sentiment categories.
References
[1] K. Chakraborty, S. Bhattacharyya, R. Bag, “A Survey of Sentiment Analysis from Social Media Data,” IEEE Transactions on Computational Social Systems, vol. 7, no. 2, pp. 450-464, 2020, https://doi.org/10.1109/TCSS.2019.2956957.
[2] A. Yadav, M. Alahmar, A. Singh, K. Sharma, R. Agrawal, C. B. Sharma, “Analyzing User Behavior in Social Media through Big Data Analytics,” IEEE International Conference on ICT in Business Industry & Government (ICTBIG), pp. 1–5, 2023, https://doi.org/10.1109/ICTBIG59752.2023.10456112.
[3] Simon Kemp, “Twitter Users, Stats, Data & Trends.” [Online]. Available: https://datareportal-com.translate.goog/essential-twitter-stats?_x_tr_sl=en&_x_tr_tl=id&_x_tr_hl=id&_x_tr_pto=tc.
[4] G. Rasool, A. Pathania, “Reading between the lines: untwining online user-generated content using sentiment analysis,” J. Res. Interact. Mark, vol. 15, no. 3, pp. 401–418, 2021, https://doi.org/10.1108/JRIM-03-2020-0045.
[5] A. R. Abas, I. Elhenawy, M. Zidan, M. Othman,“Aspect-based sentiment analysis on social media comments (twitter): the attributes of service robots in the hotel and restaurant industry,” J. Qual. Assur. Hosp. Tour, pp. 1–26, 2024, https://doi.org/10.1080/1528008X.2024.2386590.
[6] C. J. Hartmann, M. Heitmann, C. Siebert, “More than a feeling: Accuracy and application of sentiment analysis,” Int. J. Res. Mark, vol. 40, no. 1, pp. 75–87, 2023, https://doi.org/10.1016/j.ijresmar.2022.05.005.
[7] M. Abas, A. R., Elhenawy, I., Zidan, M., & Othman, “BERT-CNN: A Deep Learning Model for Detecting Emotions from Text,” Comput. Mater. Contin, vol. 71, no. 2, 2022, https://doi.org/10.32604/cmc.2022.021671.
[8] J. Hartmann, M. Heitmann, C. Sieber, "Usability evaluation of a nursing information system by applying cognitive walkthrough method,” Int. J. Med. Inform, vol. 152, p. 104459, 2021, https://doi.org/10.1016/j.ijmedinf.2021.104459.
[9] N. Raghunathan, K. Saravanakumar, “Challenges and issues in sentiment analysis: A comprehensive survey,” IEEE Access, vol. 11, 2023, https://doi.org/69626-69642.
[10] M. Wankhade, A. C. S. Rao, C. Kulkarni, “A survey on sentiment analysis methods, applications, and challenges,” Artif. Intell. Rev, vol. 55, no. 7, pp. 5731–5780, 2022, https://doi.org/10.1007/s10462-022-10144-1.
[11] R. Obiedat et al., “Sentiment analysis of customers’ reviews using a hybrid evolutionary SVM-based approach in an imbalanced data distribution,” IEEE Access, vol. 10, pp. 22260–22273, 2022, https://doi.org/10.1109/ACCESS.2022.3149482.
[12] J. Hartmann et al., “More than a feeling: Accuracy and application of sentiment analysis,” Int. J. Res. Mark., vol. 40, no. 1, pp. 75–87, 2023, https://doi.org/10.1016/j.ijresmar.2022.05.005.
[13] M. F. R. A. Bakar, N. Idris, L. Shuib, N. Khamis, “Sentiment analysis of noisy Malay text: state of art, challenges and future work,” IEEE Access, vol. 8, pp. 24687–24696, 2020, https://doi.org/10.1109/ACCESS.2020.2968955.
[14] L. R. Sultan, “An Enhanced Emotion Classification Scheme for Twits Based on Deep Learning Approach,” Rev. d’Intelligence Artif, vol. 37, no. 5, p. 1203, 2023, https://doi.org/10.18280/ria.370512.
[15] S. Minaee, N. Kalchbrenner, E. Cambria, N. Nikzad, M. Chenaghlu, J. Gao, “Deep learning--based text classification: a comprehensive review,” ACM Comput. Surv, vol. 54, no. 3, pp. 1–40, 2021, https://doi.org/10.1145/3439726.
[16] M. Celik, O. Inik, “Development of hybrid models based on deep learning and optimized machine learning algorithms for brain tumor Multi-Classification,” Expert Syst. Appl, no. 122159, p. 238, 2024, https://doi.org/10.1016/j.eswa.2023.122159.
[17] J. H Joloudari et al., “BERT-deep CNN: State of the art for sentiment analysis of COVID-19 tweets,” Soc. Netw.Anal. Min, vol. 13, no. 1, p. 99, 2023, https://doi.org/10.1007/s13278-023-01102-y.
[18] W. X. Zhao, J. Liu, R. Ren, J. R. Wenn, “Dense text retrieval based on pretrained language models: A survey,” ACM Trans. Inf. Syst., vol. 42, no. 4, pp. 1–60, 2024, https://doi.org/10.1145/3637870.
[19] N. M. Gardazi, A. Daud, M. K. Malik, A. Bukhari, T. Alsahfi, B. Alshemaimri, “BERT applications in natural language processing: a review,” Artif. Intell. Rev., vol. 58, no. 6, pp. 1–49, 2025, https://doi.org/10.1007/s10462-025-11162-5.
[20] A. S. Alammary, “Investigating the impact of pretraining corpora on the performance of Arabic BERT models,” J. Supercomput., vol. 81, no. 1, p. 187, 2025, https://doi.org/10.1007/s11227-024-06698-2.
[21] M. Khazeni, M. Heydari, A. Albadvi, “Persian Slang Text Conversion to Formal and Deep Learning of Persian Short Texts on Social Media for Sentiment Classification,” arXiv Prepr. arXiv, 2024, https://doi.org/10.22061/jecei.2024.10745.731.
[22] F. Miletić, S. S. im Walde, “A systematic search for compound semantics in pretrained BERT architectures,” Proc. 17th Conf. Eur. Chapter Assoc. Comput. Linguist, pp. 1499–1512, 2023, https://doi.org/10.18653/v1/2023.eacl-main.110.
[23] G. Sperduti, A. Moreo, “Misspellings in Natural Language Processing: A survey,” arXiv Prepr. arXiv, 2025, https://doi.org/10.48550/arXiv.2501.16836.
[24] D. Tsirmpas, I. Gkionis, G. T. Papadopoulos, I. Mademlis, “Neural natural language processing for long texts: A survey on classification and summarization,” Eng. Appl. Artif. Intell, p. 133, 2024, https://doi.org/10.1016/j.engappai.2024.108231.
[25] Y. He, “BERT-CNN-BiLSTM: A Hybrid Deep Learning Model for Accurate Sentiment Analysis,” IEEE 5th Int. Conf. Power, Intell. Comput. Syst., pp. 921–926, 2023, https://doi.org/10.1109/ICPICS58376.2023.10235335.
[26] A. S. Talaat, “Sentiment analysis classification system using hybrid BERT models,” J. Big Data, vol. 10, no. 1, 2023, https://doi.org/10.1186/s40537-023-00781-w.
[27] K. L. Tan, C. P. Lee, K. M. Lim, and K. S. M. Anbananthen, “Sentiment Analysis With Ensemble Hybrid Deep Learning Model,” IEEE Access, vol. 10, no. July, pp. 103694–103704, 2022, https://doi.org/10.1109/ACCESS.2022.3210182.
[28] C. N. Dang, M. N. Moreno-García, and F. De La Prieta, “Hybrid Deep Learning Models for Sentiment Analysis,” Complexity, 2021, https://doi.org/10.1155/2021/9986920.
[29] F. A. Acheampong, H. Nunoo-Mensah, W. Chen, “Transformer models for text-based emotion detection: a review of BERT-based approaches,” Artif. Intell. Rev, vol. 54, no. 8, pp. 5789–5829, 2021, https://doi.org/10.1007/s10462-021-09958-2.
[30] A. Onan, K. F. Balbal, “Improving Turkish text sentiment classification through task-specific and universal transformations: an ensemble data augmentation approach,” IEEE Access, vol. 12, pp. 4413–4458, 2024, https://doi.org/10.1109/ACCESS.2024.3349971.
[31] M. Shah, N. Sureja, “A comprehensive review of bias in deep learning models: Methods, impacts, and future directions,” Arch. Comput. Methods Eng, vol. 32, no. 1, pp. 255–267, 2025, https://doi.org/10.1007/s11831-024-10134-2.
[32] J Wang et al., “Generalizing to unseen domains: A survey on domain generalization,” IEEE Trans. Knowl. Data Eng, vol. 35, no. 8, pp. 8052–8072, 2022, https://doi.org/10.1109/TKDE.2022.3178128.
[33] S. Ramakrishnan and L. D. Dhinesh Babu, “"Enhancing Twitter Sentiment Analysis using Attention-based BiLSTM and BERT Embedding,” 9th Int. Conf. Smart Comput. Commun, pp. 36–40, 2023, https://doi.org/10.1109/ICSCC59169.2023.10335010.
[34] C. P. Chai, “Comparison of text preprocessing methods,” Nat. Lang. Eng., vol. 29, no. 3, pp. 509–553, 2023, https://doi.org/10.1017/S1351324922000213.
[35] D. Muhamediyeva, N. Niyozmatova, N. Turgunova, S. Ungalov, N. Almuradova, “Classification of Emoji in Text Documents of Users in Social Networks Using Machine Learning,” IEEE. 2025 6th Int. Conf. Mob. Comput. Sustain. Informatics, pp. 1491–1496, 2025, https://doi.org/10.1109/ICMCSI64620.2025.10883250.
[36] N. Merayo, “Applying machine learning to assess emotional reactions to video game content streamed on Spanish Twitch channels,” Comput. Speech Lang, p. 88, 2024, https://doi.org/10.1016/j.csl.2024.101651.
[37] D. Bino, V. Dhanalakshmi, P. K. Udupi, “Sentiment Analysis and Machine Learning for Tourism Feedback Data Analysis: An Overview of Trends, Techniques, and Applications,” AI Technol. Pers. Sustain. Tour, pp. 215-252., 2025, https://doi.org/10.4018/979-8-3693-5678-4.ch009.
[38] P. Lauren, “Improving subword embeddings in large language models using morphological information,” Artif. Intell. Mach. Learn. Convolutional Neural Networks Large Lang. Model, vol. 1, p. 333, 2024, https://doi.org/10.1515/9783111344126-015.
[39] J. Li, Y. Tao, H. Cong, E. Zhu, T. Cai, “Predicting liver cancers using skewed epidemiological data,” Artif. Intell. Med, vol. 124, no. 102234, 2022, https://doi.org/10.1016/j.artmed.2021.102234.
[40] A. R. Chłopowiec et al., “Counteracting data bias and class imbalance—towards a useful and reliable retinal disease recognition system,” Diagnostics, vol. 13, no. 11, p. 1904, 2023, https://doi.org/10.3390/diagnostics13111904.
[41] I. Araf, A. Idri, I. Chairi, “Cost-sensitive learning for imbalanced medical data: a review,” Artif. Intell. Rev, vol. 54, no. 4, p. 80, 2024, https://doi.org/10.1007/s10462-023-10652-8.
[42] G Citovsky et al., “Batch active learning at scale.,” Adv. Neural Inf. Process. Syst., vol. 34, pp. 11933–11944, 2021, https://proceedings.neurips.cc/paper/2021/hash/64254db8396e404d9223914a0bd355d2-Abstract.html.
[43] D. Alomari, I. Ahmad, “Exploring Character Trigrams for Robust Arabic Text Classification: A Comparative Analysis in the Face of Vocabulary Expansion and Misspelled Words,” IEEE Access, vol. 12, pp. 57103–57116, 2024, https://doi.org/10.1109/ACCESS.2024.3390048.
[44] B. Elizalde, S. Deshmukh, M. Al Ismail, H. Wang, “Clap learning audio concepts from natural language supervision,” ICASSP 2023-2023 IEEE Int. Conf. Acoust. Speech Signal Process, pp. 1–5, 2023, https://doi.org/10.1109/ICASSP49357.2023.10095889.
[45] M. Apidianaki, “From word types to tokens and back: A survey of approaches to word meaning representation and interpretation,” Comput. Linguist, vol. 49, no. 2, pp. 465–523, 2023, https://doi.org/10.1162/coli_a_00474.
[46] K. L. Tan, C. P. Lee, K. S. M. Anbananthen, K. M. Lim, “RoBERTa-LSTM: a hybrid model for sentiment analysis with transformer and recurrent neural network,” IEEE Access, vol. 10, pp. 21517–21525, 2022, https://doi.org/10.1109/ACCESS.2022.3162614.
[47] A. K. Kalusivalingam, A. Sharma, N. Patel, V. Singh, “Leveraging BERT and LSTM for Enhanced Natural Language Processing in Clinical Data Analysis,” Int. J. AI ML, vol. 2, no. 3, 2021, https://doi.org/10.1177/14727978251322656.
[48] X. Chen, P. Cong, S. Lv, “A long-text classification method of Chinese news based on BERT and CNN,” IEEE Access, no. 10, pp. 34046–34057, 2022, https://doi.org/10.1109/ACCESS.2022.3162614.
[49] S. Chen, “Semantic relationship extraction of English long sentences and quality optimization of machine translation based on BERT model,” J. Comput. Methods Sci. Eng, p. 14727978251322656, 2025, https://doi.org/14727978251322656.
[50] S. Almlawi, J. Fang, J. LiEnhancing, "Sentiment Analysis Using MCNN-BRNN Model with BERT,” 3rd Int. Conf. Electron. Inf. Eng. Comput. Commun, pp. 574–579, 2023, https://doi.org/10.1109/EIECC60864.2023.10456641.
[51] T. Bikku, J. Jarugula, L. Kongala, N. D. Tummala, N. V. Donthiboina, “Exploring the Effectiveness of BERT for Sentiment Analysis on Large-Scale Social Media Data,” 3rd Int. Conf. Intell. Technol, pp. 1–4, 2023, https://doi.org/10.1109/CONIT59222.2023.10205600.
[52] Y. Zhou, Q. Zhang, D. Wang, and X. Gu, “Text Sentiment Analysis Based on a New Hybrid Network Model,” Genet. Res. (Camb), p. 6774320, 2022, https://doi.org/10.1155/2022/6774320.
[53] S. Susandri, S. Defit, and M. Tajuddin, “Enhancing Text Sentiment Classification with Hybrid CNN-BiLSTM Model on WhatsApp Group,” J. Adv. Inf. Technol, vol. 15, no. 3, pp. 355–363, 2024, https://doi.org/10.12720/jait.15.3.355-363.
[54] M. M. Rahman, A. I. Shiplu, Y. Watanobe, and M. A. Alam, “RoBERTa-BiLSTM: A Context-Aware Hybrid Model for Sentiment Analysis,” arXiv preprint arXiv:2406.00367, 2024, [Online]. Available: http://arxiv.org/abs/2406.00367.
[55] G. Negi, R. Sarkar, O. Zayed, and P. Buitelaar, “A Hybrid Approach To Aspect Based Sentiment Analysis Using Transfer Learning,” 2024 Jt. Int. Conf. Comput. Linguist. Lang. Resour. Eval. Lr. 2024 - Main Conf. Proc, pp. 647–658, 2024, https://doi.org/10.48550/arXiv.2403.17254.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Shofwatul Uyun, Rizqi Praimadi Rosalin, Luky Vianika Sari, Hanny Handayani Sucinta
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with JITEKI agree to the following terms:
- Authors retain copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License (CC BY-SA 4.0) that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.
This work is licensed under a Creative Commons Attribution 4.0 International License
