Text Classification Using Long Short-Term Memory With GloVe Features

Winda Kurnia Sari, Dian Palupi Rini, Reza Firsandaya Malik


In the classification of traditional algorithms, problems of high features dimension and data sparseness often occur when classifying text. Classifying text with traditional machine learning algorithms has high efficiency and stability characteristics. However, it has certain limitations with regard to large-scale dataset training. Deep Learning is a proposed method for solving problems in text classification techniques. By tuning the parameters and comparing the eight proposed Long Short-Term Memory (LSTM) models with a large-scale dataset, to show that LSTM with features GloVe can achieve good performance in text classification. The results show that text classification using LSTM with GloVe obtain the highest accuracy is in the sixth model with 95.17, the average precision, recall, and F1-score are 95


Recurrent Neural Network; Long Short-Term Memory; Multilabel Classification;Text Classification; GloVe

Full Text:



L. Li, L. Xiao, W. Jin, H. Zhu, & G. Yang, 2018. Text Classification Based on Word2vec and Convolutional Neural Network. Journal of the Society of Mechanical Engineers, 90(823), 758-759. https://doi.org/10.1299/jsmemag.90.823_758.

R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, & C. Potts. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. (October), 1631-1642. https://doi.org/10.1371/journal.pone.0073791

H. Yuan, Y. Wang, X. Feng, & S. Sun. 2018. Sentiment Analysis Based on Weighted Word2vec and Att-LSTM. 420-424. https://doi.org/10.1145/3297156.3297228.

J. Lilleberg, Y. Zhu, & Y. Zhang. 2015. Support vector machines and Word2vec for text classification with semantic features. Proceedings of 2015 IEEE 14th International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2015, 136-140. https://doi.org/10.1109/ICCI-CC.2015.7259377.

K. Chen, Z. Zhang, J. Long, & H. Zhang. 2017.Turning from TF-IDF to TF-IGM for term weighting in text classification. Expert Systems with Applications, 66, 1339-1351. https://doi.org/10.1016/j.eswa.2016.09.009.

R. G. Rossi, A. D. A. Lopes, & S. O. Rezende. 2016. Optimization and label propagation in bipartite heterogeneous networks to improve transductive classification of texts. Information Processing and Management, 52(2), 217-257. https://doi.org/10.1016/j.ipm.2015.07.004.

B. Y. Pratama, & R. Sarno. 2016. Personality classification based on Twitter text using Naive Bayes, KNN and SVM. Proceedings of 2015 International Conference on Data and Software Engineering, ICODSE 2015, 170-174. https://doi.org/10.1109/ICODSE.2015.7436992.

Xu, W., Sun, H., Deng, C., & Tan, Y. (2017, February).Variational autoencoder for semi-supervised text classification.In Thirty-First AAAI Conference on Artificial Intelligence..

Ruangkanokmas, P., Achalakul, T., &Akkarajitsakul, K. (2017).Deep Belief Networks with Feature Selection for Sentiment Classification.Proceedings - International Conference on Intelligent Systems, Modelling and Simulation, ISMS, 9-14. https://doi.org/10.1109/ISMS.2016.9.

M. Azam, T. Ahmed, F. Sabah, F. and M.I. Hussain, 2018, “Feature Extraction based Text Classification using K-Nearest Neighbor Algorithm”. IJCSNS Int. J. Comput. Sci. Netw. Secur, 18, pp.95-101.

L. Jiang, C. Li, S. Wang, and L. Zhang, 2016, “Deep feature weighting for naive Bayes and its application to text classification”. Engineering Applications of Artificial Intelligence, 52, pp.26-39.

S. Xu, 2018, “Bayesian Naïve Bayes classifiers to text classification”. Journal of Information Science, 44(1), pp.48-59.

M. Fanjin, H. Ling, T. Jing, and W. Xinzheng, 2017, “The Research of Semantic Kernel in SVM for Chinese Text Classification”. In Proceedings of the 2nd International Conference on Intelligent Information Processing (p. 8). ACM.

M. Goudjil, M. Koudil, M. Bedda, and N. Ghoggali, 2018, “A novel active learning method using SVM for text classification”. International Journal of Automation and Computing, 15(3), pp.290-298.

A. Onan, S. Korukoğlu, and H. Bulut, 2016, “Ensemble of keyword extraction methods and classifiers in text classification”. Expert Systems with Applications, 57, pp.232-247.

R. G. F. Soares, 2018, “Effort Estimation via Text Classification and Autoencoders”. Proceedings of the International Joint Conference on Neural Networks, 2018-July, 1-8. https://doi.org/10.1109/IJCNN.2018.8489030.

M. Gao, T. Li, and P. Huang, 2018, “Text Classification Research Based on Improved Word2vec and CNN”. In International Conference on Service-Oriented Computing (pp. 126-135). Springer, Cham.

Y. Yan, Y. Wang, WC. Gao, BW. Zhang, C. Yang, and XC. Yin, "LSTM2: Multi-Label Ranking for Document Classification," Neural Processing Letters 47, no. 1, 2018, pp. 117-138.

T. Wiatowski, and H. Bölcskei, 2017, “A mathematical theory of deep convolutional neural networks for feature extraction”. IEEE Transactions on Information Theory, 64(3), pp.1845-1866.

K. Kowsari, D.E. Brown, M. Heidarysafa, K.J. Meimandi, M.S. Gerber, and L.E.Barnes, 2017, “Hdltex: Hierarchical deep learning for text classification”. In 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA) (pp. 364-371). IEEE.

H. Zen, and H. Sak, 2015, “Unidirectional long short-term memory recurrent neural network with recurrent output layer for low-latency speech synthesis”. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 4470-4474). IEEE.

Li, K., Daniels, J., Liu, C., Herrero-Vinas, P. and Georgiou, P., 2019. “Convolutional recurrent neural networks for glucose prediction”. IEEE Journal of Biomedical and Health Informatics.

K. Tseng, C. Ou, A. Huang, R.F. Lin, and X. Guo, 2019, “Genetic and Evolutionary Computing “. Vol. 834. https://doi.org/10.1007/978-981-13-5841-8.

C. Zhou, C. Sun, Z. Liu, and F. Lau, 2015, “A C-LSTM neural network for text classification”. arXiv preprint arXiv:1511.08630.

M. Pota, F. Marulli, M. Esposito, G. De Pietro, and H. Fujita, 2019, “Multilingual POS tagging by a composite deep architecture based on character-level features and on-the-fly enriched Word Embeddings”. Knowledge-Based Systems, 164, pp.309-323.

Y. Kim, 2014. “Convolutional neural networks for sentence classification”. arXiv preprint arXiv:1408.5882.

X. Zhang, J. Zhao, and Y. LeCun, 2015, “Character-level convolutional networks for text classification”. In Advances in neural information processing systems (pp. 649-657).

I. Sutskever, O. Vinyals, and Q.V. Le, 2014, “Sequence to sequence learning with neural networks”. In Advances in neural information processing systems (pp. 3104-3112).

C.C. Chiu, T.N. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen, Z. Chen, A. Kannan, R.J. Weiss, K. Rao, E. Gonina, and N. Jaitly, 2018, “State-of-the-art speech recognition with sequence-to-sequence models”. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 4774-4778). IEEE.

A. Graves, 2013, “Generating sequences with recurrent neural networks”. arXiv preprint arXiv:1308.0850.

A. Kumar, and R. Rastogi, 2019, “Attentional Recurrent Neural Networks for Sentence Classification”. In Innovations in Infrastructure. pp. 549-559. Springer, Singapore.

D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.

T. Tieleman, T. and G. Hinton, 2012, “Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude”. COURSERA: Neural networks for machine learning, 4(2), pp.26-31.

G. Wang, C. Li, W. Wang, Y. Zhang, D. Shen, X. Zhang, R. Henao, and L. Carin, 2018, “Joint embedding of words and labels for text classification”. arXiv preprint arXiv:1805.04174.

D. Shen, G. Wang, W. Wang, M.R. Min, Q. Su, Y. Zhang, C. Li, R. Henao, and L. Carin, 2018, “Baseline needs more love: On simple word-embedding-based models and associated pooling mechanisms”. arXiv preprint arXiv:1805.09843.

T. Mikolov, K.. Chen, K., G. Corrado, & J. Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality.CrossRef Listing of Deleted DOIs, 1, 1-9. https://doi.org/10.1162/jmlr.2003.3.4-5.951

J. Pennington, R. Socher, & C. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532-1543).

DOI: http://dx.doi.org/10.26555/jiteki.v5i2.15021

Article Metrics

Abstract view : 0 times
PDF - 0 times


  • There are currently no refbacks.

Jurnal Ilmiah Teknik Elektro Komputer dan Informatika (JITEKI)
ISSN 2338-3070 (print) | 2338-3062 (online)
Organized by Electrical Engineering Department - Universitas Ahmad Dahlan
Published by Universitas Ahmad Dahlan
Website: http://journal.uad.ac.id/index.php/jiteki
Email 1: jiteki@ee.uad.ac.id (publication issues)
Email 2: sonali@ee.uad.ac.id (info and paper handling issues)

Lisensi Creative Commons

This work is licensed under a Creative Commons Attribution 4.0 International License

View JITEKI stat