Advanced 3D Artistic Image Generation with VAE-SDFCycleGAN
Abstract
Generation of a 3-dimensional (3D)-based artistic image from a 2-dimensional (2D) image using a generative adversarial network (GAN) framework is challenging. Most existing artistic GAN-based frameworks lack robust algorithms lack suitable 3D data representations that can fit into GAN to produce high-quality 3D artistic images. To produce 3D artistic images from 2D image that considerably improves scalability and visual quality, this research integrates innovative variational autoencoder signed distance function, cycle generative adversarial network (VAE-SDFCycleGAN). The proposed method feeds a single 2D image into the network to produce a mesh-based 3D shape. The network encodes a 2D image of the 3D object into latent representations, and implicit surface representations of 3D images corresponding to those of 2D images are subsequently generated. VAE extracts feature from the two-dimensional input image and reconstructs a voxel-type grid using a signed distance function. Cycle GAN produces improved and high-quality 3D artistic images from 2D images. The publicly available COCO dataset was used to evaluate the proposed advanced 3D-VAE-SDFCycleGAN. The model produced a peak signal noise ratio (PSNR) of 31.35, mean square error (MSE) of 65.32, and structural similarity index measure (SSIM) of 0.772 which indicates the improved quality of the generated images. The results are compared with other traditional GAN methods and the results obtained show that the proposed method outperforms the others in terms of quantitative and qualitative evaluation metrics.
Downloads
References
P. Salehi, A. Chalechale, and M. Taghizadeh, "Generative Adversarial Networks (GANs): An Overview of Theoretical Model, Evaluation Metrics, and Recent Developments," IEEE Transactions on Visualization and Computer Graphics, vol. 24, no. 6, pp. 216-221, 2018.
D. O. Esan, P. A. Owolawi, and C. Tu, "Image Generation Using StyleVGG19-NST Generative Adversarial Networks " International Journal of Advanced Computer Science and Applications, vol. 15, 2024, doi: 10.14569/IJACSA.2024.0150808.
G. Coiffier, P. Renard, and S. Lefebvre, "3D Geological Image Synthesis From 2D Examples Using Generative Adversarial Networks," Frontier Water, vol. 2, 2020, doi: 10.3389/frwa.2020.560598.
H. A. Alhaija, A. Dirik, A. Knorig, S. Fidler, and M. Shugrina, "XDGAN: Multi-Modal 3D Shape Generation in 2D Space," NNN, pp. 1-14, 2022, doi: doi:10.5244/C.36.NNN.
E. R. Chan et al., "Efficient Geometry-aware 3D Generative Adversarial Networks," 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16102-16112, 2022, doi: 10.1109/CVPR52688.2022.01565.
R. J. Spick, S. Demediuk, and J. A. Walker, "Naive Mesh-to-Mesh Coloured Model Generation using 3D GANs," In Proceedings of Australasian Computer Science Week (ACSW’20). ACM, pp. 1-6, 2020, doi: 10.1145/1122445.1122456.
H.-K. Ko, Gwanmo-Park, H. Jeon, J. Jo, J. Kim, and J. Seo, "Large-Scale Text-to-Image Generation Models for Visual Artists' Creative Work," ACM Symposium on Neural Gaze Detection, pp. 1-15, 1999.
Z. Niu, S. Xiang, and M. Zhang, "Application of Artificial Intelligence Combined with Three-Dimensional Digital Technology in the Design of Complex Works of Art," Hindawi Wireless Communication and Mobile Computing vol. 2022, pp. 1-9, 2022.
A. Karnewar and O. Wang, "MSG-GAN: Multi-Scale Gradients for Generative Adversarial Networks," arXiv:1903.06048v4 [cs.CV] 12 Jun 2020, pp. 1-18, 2020.
R. J. Spick, P. Renard, S. Demediuk, and J. A. Walker, "Naive Mesh-to-Mesh Coloured Model Generation using 3D GANs," Proceedings of Australasian Computer Science Week (ACSW’20), 2019.
D. Victor, "COCO-AFRICA: A Curation Tool and Dataset of Common Objects in the Context of Africa," Conference on Neural Information Processing, 2nd Black in AI Workshop, 2018.
P. A. O. a. C. T. Dorcas. Esan, "Anomalous Detection in Noisy Image Frames using Cooperative Median Filtering and KNN," IAENG International Journal of Computer Science, vol. 49, no. 1, 2022.
E. A. Ajayi, K. M. Lim, S.-C. Chong, and C. P. Lee, "Three-dimensional shape generation via variational autoencoder generative adversarial network with signed distance function," International Journal of Electrical and Computer Engineering (IJECE), vol. 13, no. 4, pp. 4009-4019, 2023, doi: 10.11591/ijece.v13i4.pp4009-4019.
C. Kingkan and K. Hashimoto, "Generating mesh-based shapes from learned latent spaces of point clouds with VAE-GAN," in 2018 24th International Conference on Pattern Recognition (ICPR), pp. 308–313, 2018, doi: 10.1109/ICPR.2018.8546232.
C. M. Jiang and P. Marcus, "Hierarchical detail enhancing mesh-based shape generation with 3D generative adversarial network," arXiv preprint arXiv:1709.07581, 2017.
M. Soloveitchik, E. M. Tzvi Diskin, and A. Wiesel, "Conditional Frechet Inception Distance," arXiv:2103.11521v2 [cs.LG] 28 Feb 2022, pp. 1-11, 2022.
M. J. Chong and D. Forsyth, "Effectively Unbiased FID and Inception Score and Where to Find Them," 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, doi: 10.1109/CVPR42600.2020.00611.
I. Vaccari, V. Orani, A. Paglialonga, E. Cambiaso, and M. Mongelli, "A Generative Adversarial Network (GAN) Technique for Internet of Medical Things Data," Sensors, vol. 3726, no. 21, pp. 1-14, 2021, doi: 10.3390/s21113726.
H. Zhang, H. Li, J. R. Dillman, N. A. Parikh, and L. He, "Multi-Contrast MRI Image Synthesis Using Switchable Cycle-Consistent Generative Adversarial Networks," Diagnostics, vol. 12, no. 816, 2022, doi: 10.3390/diagnostics12040816.
U. Sara, M. Akter, and M. S. Uddin, "Image Quality Assessment through FSIM, SSIM, MSE and PSNR—A Comparative Study," Journal of Computer and Communications vol. 17, no. 3, 2019, doi: 10.4236/jcc.2019.73002.
M. Wenzel, "Generative Adversarial Networks and Other Generative Models," In Colliot O, editor. Machine Learning for Brain Disorders [Internet], 2023, doi: 10.1007/978-1-0716-3195-9_5.
Y. Z. N. Wang, Z. Li, Y. Fu, W. Liu, and Y.-G. Jiang, "Pixel2Mesh: generating 3D mesh models from single RGB images," in Computer Vision ECCV 2018, Springer International Publishing, pp. 55-71, 2018.
L. Tran, X. Yin, and X. Liu, "Disentangled Representation Learning GAN for Pose-Invariant Face Recognition," in Conference: IEEE Computer Vision and Pattern Recognition (CVPR 2017), Honolulu, Hawaii, 2017, vol. Tran, Luan & Yin, Xi & Liu, Xiaoming. (2017). Disentangled Representation Learning GAN for Pose-Invariant Face Recognition. 10.1109/CVPR.2017.141. , pp. 1415–1424, doi 10.1109/CVPR.2017.141.
T. Miyato, Kataoka, T., Koyama, M., and Yoshida, Y, "Spectral Normalization for Generative Adversarial Networks," International Conference on Learning Representations (ICLR), 2020.
X. Mao, L. Qing, H. Xie, R. Y. Lau, Z. Wang, and S. P. Smolley, "Least Squares Generative Adversarial Networks," IEEE International Conference on Computer Vision (ICCV), pp. 2813-2821, 2017, doi: doi: 10.1109/ICCV.2017.304.


Copyright (c) 2024 Journal of Information Systems and Informatics

This work is licensed under a Creative Commons Attribution 4.0 International License.
- I certify that I have read, understand and agreed to the Journal of Information Systems and Informatics (Journal-ISI) submission guidelines, policies and submission declaration. Submission already using the provided template.
- I certify that all authors have approved the publication of this and there is no conflict of interest.
- I confirm that the manuscript is the authors' original work and the manuscript has not received prior publication and is not under consideration for publication elsewhere and has not been previously published.
- I confirm that all authors listed on the title page have contributed significantly to the work, have read the manuscript, attest to the validity and legitimacy of the data and its interpretation, and agree to its submission.
- I confirm that the paper now submitted is not copied or plagiarized version of some other published work.
- I declare that I shall not submit the paper for publication in any other Journal or Magazine till the decision is made by journal editors.
- If the paper is finally accepted by the journal for publication, I confirm that I will either publish the paper immediately or withdraw it according to withdrawal policies
- I Agree that the paper published by this journal, I transfer copyright or assign exclusive rights to the publisher (including commercial rights)