Text Classification Using Long Short-Term Memory With GloVe Features

Authors

  • Winda Kurnia Sari Universitas Sriwijaya
  • Dian Palupi Rini Universitas Sriwijaya
  • Reza Firsandaya Malik Universitas Sriwijaya

DOI:

https://doi.org/10.26555/jiteki.v5i2.15021

Keywords:

Recurrent Neural Network, Long Short-Term Memory, Multilabel Classification, Text Classification, GloVe

Abstract

In the classification of traditional algorithms, problems of high features dimension and data sparseness often occur when classifying text. Classifying text with traditional machine learning algorithms has high efficiency and stability characteristics. However, it has certain limitations with regard to large-scale dataset training. Deep Learning is a proposed method for solving problems in text classification techniques. By tuning the parameters and comparing the eight proposed Long Short-Term Memory (LSTM) models with a large-scale dataset, to show that LSTM with features GloVe can achieve good performance in text classification. The results show that text classification using LSTM with GloVe obtain the highest accuracy is in the sixth model with 95.17, the average precision, recall, and F1-score are 95

References

L. Li, L. Xiao, W. Jin, H. Zhu, & G. Yang, 2018. Text Classification Based on Word2vec and Convolutional Neural Network. Journal of the Society of Mechanical Engineers, 90(823), 758-759. https://doi.org/10.1299/jsmemag.90.823_758.

R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, & C. Potts. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. (October), 1631-1642. https://doi.org/10.1371/journal.pone.0073791

H. Yuan, Y. Wang, X. Feng, & S. Sun. 2018. Sentiment Analysis Based on Weighted Word2vec and Att-LSTM. 420-424. https://doi.org/10.1145/3297156.3297228.

J. Lilleberg, Y. Zhu, & Y. Zhang. 2015. Support vector machines and Word2vec for text classification with semantic features. Proceedings of 2015 IEEE 14th International Conference on Cognitive Informatics and Cognitive Computing, ICCI*CC 2015, 136-140. https://doi.org/10.1109/ICCI-CC.2015.7259377.

K. Chen, Z. Zhang, J. Long, & H. Zhang. 2017.Turning from TF-IDF to TF-IGM for term weighting in text classification. Expert Systems with Applications, 66, 1339-1351. https://doi.org/10.1016/j.eswa.2016.09.009.

R. G. Rossi, A. D. A. Lopes, & S. O. Rezende. 2016. Optimization and label propagation in bipartite heterogeneous networks to improve transductive classification of texts. Information Processing and Management, 52(2), 217-257. https://doi.org/10.1016/j.ipm.2015.07.004.

B. Y. Pratama, & R. Sarno. 2016. Personality classification based on Twitter text using Naive Bayes, KNN and SVM. Proceedings of 2015 International Conference on Data and Software Engineering, ICODSE 2015, 170-174. https://doi.org/10.1109/ICODSE.2015.7436992.

Xu, W., Sun, H., Deng, C., & Tan, Y. (2017, February).Variational autoencoder for semi-supervised text classification.In Thirty-First AAAI Conference on Artificial Intelligence..

Ruangkanokmas, P., Achalakul, T., &Akkarajitsakul, K. (2017).Deep Belief Networks with Feature Selection for Sentiment Classification.Proceedings - International Conference on Intelligent Systems, Modelling and Simulation, ISMS, 9-14. https://doi.org/10.1109/ISMS.2016.9.

M. Azam, T. Ahmed, F. Sabah, F. and M.I. Hussain, 2018, “Feature Extraction based Text Classification using K-Nearest Neighbor Algorithmâ€. IJCSNS Int. J. Comput. Sci. Netw. Secur, 18, pp.95-101.

L. Jiang, C. Li, S. Wang, and L. Zhang, 2016, “Deep feature weighting for naive Bayes and its application to text classificationâ€. Engineering Applications of Artificial Intelligence, 52, pp.26-39.

S. Xu, 2018, “Bayesian Naïve Bayes classifiers to text classificationâ€. Journal of Information Science, 44(1), pp.48-59.

M. Fanjin, H. Ling, T. Jing, and W. Xinzheng, 2017, “The Research of Semantic Kernel in SVM for Chinese Text Classificationâ€. In Proceedings of the 2nd International Conference on Intelligent Information Processing (p. 8). ACM.

M. Goudjil, M. Koudil, M. Bedda, and N. Ghoggali, 2018, “A novel active learning method using SVM for text classificationâ€. International Journal of Automation and Computing, 15(3), pp.290-298.

A. Onan, S. KorukoÄŸlu, and H. Bulut, 2016, “Ensemble of keyword extraction methods and classifiers in text classificationâ€. Expert Systems with Applications, 57, pp.232-247.

R. G. F. Soares, 2018, “Effort Estimation via Text Classification and Autoencodersâ€. Proceedings of the International Joint Conference on Neural Networks, 2018-July, 1-8. https://doi.org/10.1109/IJCNN.2018.8489030.

M. Gao, T. Li, and P. Huang, 2018, “Text Classification Research Based on Improved Word2vec and CNNâ€. In International Conference on Service-Oriented Computing (pp. 126-135). Springer, Cham.

Y. Yan, Y. Wang, WC. Gao, BW. Zhang, C. Yang, and XC. Yin, "LSTM2: Multi-Label Ranking for Document Classification," Neural Processing Letters 47, no. 1, 2018, pp. 117-138.

T. Wiatowski, and H. Bölcskei, 2017, “A mathematical theory of deep convolutional neural networks for feature extractionâ€. IEEE Transactions on Information Theory, 64(3), pp.1845-1866.

K. Kowsari, D.E. Brown, M. Heidarysafa, K.J. Meimandi, M.S. Gerber, and L.E.Barnes, 2017, “Hdltex: Hierarchical deep learning for text classificationâ€. In 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA) (pp. 364-371). IEEE.

H. Zen, and H. Sak, 2015, “Unidirectional long short-term memory recurrent neural network with recurrent output layer for low-latency speech synthesisâ€. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 4470-4474). IEEE.

Li, K., Daniels, J., Liu, C., Herrero-Vinas, P. and Georgiou, P., 2019. “Convolutional recurrent neural networks for glucose predictionâ€. IEEE Journal of Biomedical and Health Informatics.

K. Tseng, C. Ou, A. Huang, R.F. Lin, and X. Guo, 2019, “Genetic and Evolutionary Computing “. Vol. 834. https://doi.org/10.1007/978-981-13-5841-8.

C. Zhou, C. Sun, Z. Liu, and F. Lau, 2015, “A C-LSTM neural network for text classificationâ€. arXiv preprint arXiv:1511.08630.

M. Pota, F. Marulli, M. Esposito, G. De Pietro, and H. Fujita, 2019, “Multilingual POS tagging by a composite deep architecture based on character-level features and on-the-fly enriched Word Embeddingsâ€. Knowledge-Based Systems, 164, pp.309-323.

Y. Kim, 2014. “Convolutional neural networks for sentence classificationâ€. arXiv preprint arXiv:1408.5882.

X. Zhang, J. Zhao, and Y. LeCun, 2015, “Character-level convolutional networks for text classificationâ€. In Advances in neural information processing systems (pp. 649-657).

I. Sutskever, O. Vinyals, and Q.V. Le, 2014, “Sequence to sequence learning with neural networksâ€. In Advances in neural information processing systems (pp. 3104-3112).

C.C. Chiu, T.N. Sainath, Y. Wu, R. Prabhavalkar, P. Nguyen, Z. Chen, A. Kannan, R.J. Weiss, K. Rao, E. Gonina, and N. Jaitly, 2018, “State-of-the-art speech recognition with sequence-to-sequence modelsâ€. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 4774-4778). IEEE.

A. Graves, 2013, “Generating sequences with recurrent neural networksâ€. arXiv preprint arXiv:1308.0850.

A. Kumar, and R. Rastogi, 2019, “Attentional Recurrent Neural Networks for Sentence Classificationâ€. In Innovations in Infrastructure. pp. 549-559. Springer, Singapore.

D. Kingma and J. Ba, “Adam: A method for stochastic optimization,†arXiv preprint arXiv:1412.6980, 2014.

T. Tieleman, T. and G. Hinton, 2012, “Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitudeâ€. COURSERA: Neural networks for machine learning, 4(2), pp.26-31.

G. Wang, C. Li, W. Wang, Y. Zhang, D. Shen, X. Zhang, R. Henao, and L. Carin, 2018, “Joint embedding of words and labels for text classificationâ€. arXiv preprint arXiv:1805.04174.

D. Shen, G. Wang, W. Wang, M.R. Min, Q. Su, Y. Zhang, C. Li, R. Henao, and L. Carin, 2018, “Baseline needs more love: On simple word-embedding-based models and associated pooling mechanismsâ€. arXiv preprint arXiv:1805.09843.

T. Mikolov, K.. Chen, K., G. Corrado, & J. Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality.CrossRef Listing of Deleted DOIs, 1, 1-9. https://doi.org/10.1162/jmlr.2003.3.4-5.951

J. Pennington, R. Socher, & C. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) (pp. 1532-1543).

Downloads

Published

2019-12-30

How to Cite

[1]
W. K. Sari, D. P. Rini, and R. F. Malik, “Text Classification Using Long Short-Term Memory With GloVe Features”, J. Ilm. Tek. Elektro Komput. Dan Inform, vol. 5, no. 2, pp. 85–100, Dec. 2019.

Issue

Vol. 5 No. 2 (2019): December

Section

Articles

License

Authors who publish with JITEKI agree to the following terms:

  1. Authors retain copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License (CC BY-SA 4.0) that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
  2. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
  3. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.

Lisensi Creative Commons

This work is licensed under a Creative Commons Attribution 4.0 International License

Similar Articles

1 2 > >> 

You may also start an advanced similarity search for this article.