Text Classification Using Long Short-Term Memory With GloVe Features

Winda Kurnia Sari, Dian Palupi Rini, Reza Firsandaya Malik

Abstract


In the classification of traditional algorithms, problems of high features dimension and data sparseness often occur when classifying text. Classifying text with traditional machine learning algorithms has high efficiency and stability characteristics. However, it has certain limitations with regard to large-scale dataset training. Deep Learning is a proposed method for solving problems in text classification techniques. By tuning the parameters and comparing the eight proposed Long Short-Term Memory (LSTM) models with a large-scale dataset, to show that LSTM with features GloVe can achieve good performance in text classification. The results show that text classification using LSTM with GloVe obtain the highest accuracy is in the sixth model with 95.17, the average precision, recall, and F1-score are 95

Keywords


Recurrent Neural Network; Long Short-Term Memory; Multilabel Classification;Text Classification; GloVe

Full Text:

PDF

References


L. Li, L. Xiao, W. Jin, H. Zhu, & G. Yang, 2018. Text Classification Based on Word2vec and Convolutional Neural Network. Journal of the Society of Mechanical Engineers, 90(823), 758-759. https://doi.org/10.1299/jsmemag.90.823_758.

R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, & C. Potts. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. (October), 1631-1642. https://doi.org/10.1371/journal.pone.0073791

H. Yuan, Y. Wang, X. Feng, & S. Sun. 2018. Sentiment Analysis Based on Weighted Word2vec and Att-LSTM. 420-424. https://doi.org/10.1145/3297156.3297228.

J. Lilleberg, Y. Zhu, & Y. Zhang. 2015. Support vector machines and Word2vec for text classification with semantic features. Proceedings of 2015 IEEE 14th International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2015, 136-140. https://doi.org/10.1109/ICCI-CC.2015.7259377.

K. Chen, Z. Zhang, J. Long, & H. Zhang. 2017.Turning from TF-IDF to TF-IGM for term weighting in text classification. Expert Systems with Applications, 66, 1339-1351. https://doi.org/10.1016/j.eswa.2016.09.009.

R. G. Rossi, A. D. A. Lopes, & S. O. Rezende. 2016. Optimization and label propagation in bipartite heterogeneous networks to improve transductive classification of texts. Information Processing and Management, 52(2), 217-257. https://doi.org/10.1016/j.ipm.2015.07.004.

B. Y. Pratama, & R. Sarno. 2016. Personality classification based on Twitter text using Naive Bayes, KNN and SVM. Proceedings of 2015 International Conference on Data and Software Engineering, ICODSE 2015, 170-174. https://doi.org/10.1109/ICODSE.2015.7436992.

Xu, W., Sun, H., Deng, C., & Tan, Y. (2017, February).Variational autoencoder for semi-supervised text classification.In Thirty-First AAAI Conference on Artificial Intelligence..

Ruangkanokmas, P., Achalakul, T., &Akkarajitsakul, K. (2017).Deep Belief Networks with Feature Selection for Sentiment Classification.Proceedings - International Conference on Intelligent Systems, Modelling and Simulation, ISMS, 9-14. https://doi.org/10.1109/ISMS.2016.9.

M. Azam, T. Ahmed, F. Sabah, F. and M.I. Hussain, 2018, “Feature Extraction based Text Classification using K-Nearest Neighbor Algorithmâ€. IJCSNS Int. J. Comput. Sci. Netw. Secur, 18, pp.95-101.

L. Jiang, C. Li, S. Wang, and L. Zhang, 2016, “Deep feature weighting for naive Bayes and its application to text classificationâ€. Engineering Applications of Artificial Intelligence, 52, pp.26-39.

S. Xu, 2018, “Bayesian Naïve Bayes classifiers to text classificationâ€. Journal of Information Science, 44(1), pp.48-59.

M. Fanjin, H. Ling, T. Jing, and W. Xinzheng, 2017, “The Research of Semantic Kernel in SVM for Chinese Text Classificationâ€. In Proceedings of the 2nd International Conference on Intelligent Information Processing (p. 8). ACM.

M. Goudjil, M. Koudil, M. Bedda, and N. Ghoggali, 2018, “A novel active learning method using SVM for text classificationâ€. International Journal of Automation and Computing, 15(3), pp.290-298.

A. Onan, S. KorukoÄŸlu, and H. Bulut, 2016, “Ensemble of keyword extraction methods and classifiers in text classificationâ€. Expert Systems with Applications, 57, pp.232-247.

R. G. F. Soares, 2018, “Effort Estimation via Text Classification and Autoencodersâ€. Proceedings of the International Joint Conference on Neural Networks, 2018-July, 1-8. https://doi.org/10.1109/IJCNN.2018.8489030.

M. Gao, T. Li, and P. Huang, 2018, “Text Classification Research Based on Improved Word2vec and CNNâ€. In International Conference on Service-Oriented Computing (pp. 126-135). Springer, Cham.

Y. Yan, Y. Wang, WC. Gao, BW. Zhang, C. Yang, and XC. Yin, "LSTM2: Multi-Label Ranking for Document Classification," Neural Processing Letters 47, no. 1, 2018, pp. 117-138.

T. Wiatowski, and H. Bölcskei, 2017, “A mathematical theory of deep convolutional neural networks for feature extractionâ€. IEEE Transactions on Information Theory, 64(3), pp.1845-1866.

K. Kowsari, D.E. Brown, M. Heidarysafa, K.J. Meimandi, M.S. Gerber, and L.E.Barnes, 2017, “Hdltex: Hierarchical deep learning for text classificationâ€. In 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA) (pp. 364-371). IEEE.

H. Zen, and H. Sak, 2015, “Unidirectional long short-term memory recurrent neural network with recurrent output layer for low-latency speech synthesisâ€. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 4470-4474). IEEE.

Li, K., Daniels, J., Liu, C., Herrero-Vinas, P. and Georgiou, P., 2019. “Convolutional recurrent neural networks for glucose predictionâ€. IEEE Journal of Biomedical and Health Informatics.

K. Tseng, C. Ou, A. Huang, R.F. Lin, and X. Guo, 2019, “Genetic and Evolutionary Computing “. Vol. 834. https://doi.org/10.1007/978-981-13-5841-8.

C. Zhou, C. Sun, Z. Liu, and F. Lau, 2015, “A C-LSTM neural network for text classificationâ€. arXiv preprint arXiv:1511.08630.

M. Pota, F. Marulli, M. Esposito, G. De Pietro, and H. Fujita, 2019, “Multilingual POS tagging by a composite deep architecture based on character-level features and on-the-fly enriched Word Embeddingsâ€. Knowledge-Based Systems, 164, pp.309-323.

Y. Kim, 2014. “Convolutional neural networks for sentence classificationâ€. arXiv preprint arXiv:1408.5882.

X. Zhang, J. Zhao, and Y. LeCun, 2015, “Character-level convolutional networks for text classificationâ€. In Advances in neural information processing systems (pp. 649-657).

I. Sutskever, O. Vinyals, and Q.V. Le, 2014, “Sequence to sequence learning with neural networksâ€. In Advances in neural information processing systems (pp. 3104-3112).

C.C. Chiu, T.N. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen, Z. Chen, A. Kannan, R.J. Weiss, K. Rao, E. Gonina, and N. Jaitly, 2018, “State-of-the-art speech recognition with sequence-to-sequence modelsâ€. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 4774-4778). IEEE.

A. Graves, 2013, “Generating sequences with recurrent neural networksâ€. arXiv preprint arXiv:1308.0850.

A. Kumar, and R. Rastogi, 2019, “Attentional Recurrent Neural Networks for Sentence Classificationâ€. In Innovations in Infrastructure. pp. 549-559. Springer, Singapore.

D. Kingma and J. Ba, “Adam: A method for stochastic optimization,†arXiv preprint arXiv:1412.6980, 2014.

T. Tieleman, T. and G. Hinton, 2012, “Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitudeâ€. COURSERA: Neural networks for machine learning, 4(2), pp.26-31.

G. Wang, C. Li, W. Wang, Y. Zhang, D. Shen, X. Zhang, R. Henao, and L. Carin, 2018, “Joint embedding of words and labels for text classificationâ€. arXiv preprint arXiv:1805.04174.

D. Shen, G. Wang, W. Wang, M.R. Min, Q. Su, Y. Zhang, C. Li, R. Henao, and L. Carin, 2018, “Baseline needs more love: On simple word-embedding-based models and associated pooling mechanismsâ€. arXiv preprint arXiv:1805.09843.

T. Mikolov, K.. Chen, K., G. Corrado, & J. Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality.CrossRef Listing of Deleted DOIs, 1, 1-9. https://doi.org/10.1162/jmlr.2003.3.4-5.951

J. Pennington, R. Socher, & C. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532-1543).




DOI: http://dx.doi.org/10.26555/jiteki.v5i2.15021

Refbacks

  • There are currently no refbacks.


Copyright (c) 2020 Universitas Ahmad Dahlan

Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

 
About the JournalJournal PoliciesAuthor Information
 

Jurnal Ilmiah Teknik Elektro Komputer dan Informatika
ISSN 2338-3070 (print) | 2338-3062 (online)
Organized by Electrical Engineering Department - Universitas Ahmad Dahlan
Published by Universitas Ahmad Dahlan
Website: http://journal.uad.ac.id/index.php/jiteki
Email 1: jiteki@ee.uad.ac.id
Email 2: alfianmaarif@ee.uad.ac.id
Office Address: Kantor Program Studi Teknik Elektro, Lantai 6 Sayap Barat, Kampus 4 UAD, Jl. Ringroad Selatan, Tamanan, Kec. Banguntapan, Bantul, Daerah Istimewa Yogyakarta 55191, Indonesia