Proposal of Image generation model using cGANs for sketching faces

Authors

  • Nguyen Phat Huu Hanoi University of Science and Technology (HUST)
  • Nguyet Giap Thi Hanoi University of Science and Technology (HUST)

Keywords:

GANs, cGANs, CNN, Sketching faces, Image processing

Abstract

The transition from sketches to realistic images of human faces has an important application in criminal investigation science to find criminals as depicted by witnesses. However, due to the difference between the sketch image and the real face image in terms of image detail and color, it is challenging and takes time to transform from hand-drawn sketches to actual faces. To solve this problem, we propose an image generation model using the conditional generative adversarial network with autoencoder (cGANs-AE) model to generate synthetic samples for variable length and multi-feature sequence datasets. The goal of the model is to learn how to encode a dataset that reduces its vector size. Using a vector with reducing the dimension, the autoencoder will have to recreate the image similar to the original image. The autoencoder aims to produce output as input and focus only on the essential features. Raw sketches over the cGANS create realistic images that quickly and easily make the sketch images raw images. The results show that the model achieves high accuracy of up to 75%, and PSNR is 25.5 dB that is potentially applicable for practice with only 606 face images. The performance of our proposed architecture is compared with other solutions, and the results show that our proposal obtains competitive performance in terms of output quality (25.5 dB) and efficiency (above 75%).

References

Y. Jo and J. Park, “SC-FEGAN: Face Editing Generative Adversarial Network With User’s Sketch and Color,†2019 IEEE/CVF Int. Conf. Comput. Vis., pp. 1745–1753, Oct. 2019, doi: 10.1109/ICCV.2019.00183.

I. J. Goodfellow et al., “Generative Adversarial Nets,†Adv. Neural Inf. Process. Syst., pp. 2672–2680, Jun. 2014, doi: 10.5555/2969033.2969125.

C.-H. Lee, Z. Liu, L. Wu, and P. Luo, “MaskGAN: Towards Diverse and Interactive Facial Image Manipulation,†2020 IEEE/CVF Conf. Comput. Vis. Pattern Recognit., pp. 5548–5557, Jun. 2020, doi: 10.1109/CVPR42600.2020.00559.

M. Wang et al., “Example-Guided Style-Consistent Image Synthesis From Semantic Labeling,†2019 IEEE/CVF Conf. Comput. Vis. Pattern Recognit., pp. 1495–1504, Jun. 2019, doi: 10.1109/CVPR.2019.00159.

D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros, “Context Encoders: Feature Learning by Inpainting,†2016 IEEE Conf. Comput. Vis. Pattern Recognit., pp. 2536–2544, Jun. 2016, doi: 10.1109/CVPR.2016.278.

P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-Image Translation with Conditional Adversarial Networks,†2017 IEEE Conf. Comput. Vis. Pattern Recognit., pp. 5967–5976, Jul. 2017, doi: 10.1109/CVPR.2017.632.

T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro, “High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs,†2018 IEEE/CVF Conf. Comput. Vis. Pattern Recognit., pp. 8798–8807, Jun. 2018, doi: 10.1109/CVPR.2018.00917.

J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, “Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks,†2017 IEEE Int. Conf. Comput. Vis., pp. 2242–2251, Oct. 2017, doi: 10.1109/ICCV.2017.244.

D. Wu and Q. Dai, “Sketch realizing: lifelike portrait synthesis from sketch,†Proc. 2009 Comput. Graph. Int. Conf., pp. 13–20, 2009, doi: 10.1145/1629739.1629741.

H. V. Dinh, “Building database of human to apply to the portrait of the criminal through descriptions of witnesses and victims,†Quang Ninh province police, 2017. http://cstc.cand.com.vn.

S. A. Israel et al., “Generative Adversarial Networks for Classification,†2017 IEEE Appl. Imag. Pattern Recognit. Work., pp. 1–4, Oct. 2017, doi: 10.1109/AIPR.2017.8457952.

L. Gonog and Y. Zhou, “A Review: Generative Adversarial Networks,†2019 14th IEEE Conf. Ind. Electron. Appl., pp. 505–510, Jun. 2019, doi: 10.1109/ICIEA.2019.8833686.

Y.-J. Cao et al., “Recent Advances of Generative Adversarial Networks in Computer Vision,†IEEE Access, vol. 7, pp. 14985–15006, 2019, doi: 10.1109/ACCESS.2018.2886814.

M. A. Souibgui and Y. Kessentini, “DE-GAN: A Conditional Generative Adversarial Network for Document Enhancement,†IEEE Trans. Pattern Anal. Mach. Intell., pp. 1–1, 2021, doi: 10.1109/TPAMI.2020.3022406.

J. Wang, X. Li, and J. Yang, “Stacked Conditional Generative Adversarial Networks for Jointly Learning Shadow Detection and Shadow Removal,†2018 IEEE/CVF Conf. Comput. Vis. Pattern Recognit., pp. 1788–1797, Jun. 2018, doi: 10.1109/CVPR.2018.00192.

S. Kim and D. Y. Suh, “Recursive Conditional Generative Adversarial Networks for Video Transformation,†IEEE Access, vol. 7, pp. 37807–37821, 2019, doi: 10.1109/ACCESS.2019.2906472.

N. Hubens, “Deep inside: Autoencoders,†2018. https://towardsdatascience.com/deep-inside-autoencoders-7e41f319999f.

Q. P. Nguyen, K. W. Lim, D. M. Divakaran, K. H. Low, and M. C. Chan, “GEE: A Gradient-based Explainable Variational Autoencoder for Network Anomaly Detection,†2019 IEEE Conf. Commun. Netw. Secur., pp. 91–99, Jun. 2019, doi: 10.1109/CNS.2019.8802833.

J. Xue, P. P. K. Chan, and X. Hu, “Experimental study on stacked autoencoder on insufficient training samples,†2017 Int. Conf. Wavelet Anal. Pattern Recognit., pp. 223–229, Jul. 2017, doi: 10.1109/ICWAPR.2017.8076693.

A. Deshpande, J. Lu, M.-C. Yeh, M. J. Chong, and D. Forsyth, “Learning Diverse Image Colorization,†2017 IEEE Conf. Comput. Vis. Pattern Recognit., pp. 2877–2885, Jul. 2017, doi: 10.1109/CVPR.2017.307.

R. Tyleˇcek, “The CMP Facade Database,†Res. Reports C. Czech Tech. Univ. Prague, No. 24, 2012, pp. 1–8, 2013, [Online]. Available: https://cmp.felk.cvut.cz/~tylecr1/facade/CMP_facade_DB_2013.pdf.

A. Martinez and R. Benavente, “The AR Face Database: CVC Technical Report, 24,†1998. Available: Google Scholar.

K. Messer, J. Matas, J. Kittler, J. Luettin, and G. Maitre, “XM2VTSDB: The extended M2VTS database,†Second Int. Conf. audio video-based biometric Pers. authentication, vol. 964, pp. 965–966, 1999. Available: Google Scholar.

A. Radford, L. Metz, and S. Chintala, “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks. BT - 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.†2016, [Online]. Available: http://arxiv.org/abs/1511.06434.

Q. Chen and V. Koltun, “Photographic Image Synthesis with Cascaded Refinement Networks,†2017 IEEE Int. Conf. Comput. Vis., pp. 1520–1529, Oct. 2017, doi: 10.1109/ICCV.2017.168.

K. Shmelkov, C. Schmid, and K. Alahari, “How Good Is My GAN?,†Ferrari V., Hebert M., Sminchisescu C., Weiss Y. Comput. Vis. – ECCV 2018. ECCV 2018. Lect. Notes Comput. Sci., vol. 11206, pp. 218–234, 2018, doi: 10.1007/978-3-030-01216-8_14.

S. Gu, J. Bao, H. Yang, D. Chen, F. Wen, and L. Yuan, “Mask-Guided Portrait Editing With Conditional GANs,†2019 IEEE/CVF Conf. Comput. Vis. Pattern Recognit., pp. 3431–3440, Jun. 2019, doi: 10.1109/CVPR.2019.00355.

V. Carvalho, F. Soares, and R. Vasconcelos, “Artificial intelligence and image processing based techniques: A tool for yarns parameterization and fabrics prediction,†2009 IEEE Conf. Emerg. Technol. Fact. Autom., pp. 1–4, Sep. 2009, doi: 10.1109/ETFA.2009.5347255.

P. N. Huu, T. Tran Van, and N. G. Thi, “Proposing distortion compensation algorithm for determining distance using two cameras,†2019 6th NAFOSTED Conf. Inf. Comput. Sci., pp. 172–177, Dec. 2019, doi: 10.1109/NICS48868.2019.9023875.

P. N. Huu, V. Tran-Quang, and T. Miyoshi, “Energy threshold adaptation algorithms on image compression to prolong WSN lifetime,†2010 7th Int. Symp. Wirel. Commun. Syst., pp. 834–838, Sep. 2010, doi: 10.1109/ISWCS.2010.5624318.

S. S. Kumar, F. Taheri, and M. R. Islam, “Artificial Intelligence and Image Processing Approaches in Damage Assessment and Material Evaluation,†Int. Conf. Comput. Intell. Model. Control Autom. Int. Conf. Intell. Agents, Web Technol. Internet Commer., vol. 1, pp. 307–313, doi: 10.1109/CIMCA.2005.1631284.

S. Shukla, A. Lakhmani, and A. K. Agarwal, “Approaches of artificial intelligence in biomedical image processing: A leading tool between computer vision & biological vision,†2016 Int. Conf. Adv. Comput. Commun. Autom., pp. 1–6, Apr. 2016, doi: 10.1109/ICACCA.2016.7578900.

A. K. Rathinam, Y. Lee, D. N. C. Ling, and R. Singh, “A review of image processing leading to artificial intelligence methods to detect instruments in ultrasound guided minimally invasive surgical procedures,†2017 IEEE Int. Conf. Power, Control. Signals Instrum. Eng., pp. 3074–3079, Sep. 2017, doi: 10.1109/ICPCSI.2017.8392290.

X. Jia, “Image recognition method based on deep learning,†2017 29th Chinese Control Decis. Conf., pp. 4730–4735, May 2017, doi: 10.1109/CCDC.2017.7979332.

J. Ruili, W. Haocong, W. Han, E. O’Connell, and S. McGrath, “Smart Parking System Using Image Processing and Artificial Intelligence,†2018 12th Int. Conf. Sens. Technol., pp. 232–235, Dec. 2018, doi: 10.1109/ICSensT.2018.8603590.

W. Chao, L. Chang, X. Wang, J. Cheng, X. Deng, and F. Duan, “High-Fidelity Face Sketch-To-Photo Synthesis Using Generative Adversarial Network,†2019 IEEE Int. Conf. Image Process., pp. 4699–4703, Sep. 2019, doi: 10.1109/ICIP.2019.8803549.

Downloads

Published

2021-05-07

Issue

Vol. 15 No. 2 (2021): May 2021

Section

Articles

License

Authors who publish with Jurnal Informatika (JIFO) agree to the following terms:

  • Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License (CC BY-SA 4.0) that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
  • Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
  • Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work.

 

 Creative Commons License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.