An Experimental Study of The Efficacy of Prompting Strategies In Guiding ChatGPT for A Computer Programming Task
Abstract
In the rapidly advancing artificial intelligence (AI) era, optimizing language models such as Chatbot Generative Pretrained Transformer (ChatGPT) for specialised tasks like computer programming remains a mystery. There are numerous inconsistencies in the quality and correctness of code generated by ChatGPT in programming. This study aims to analyse how the different prompting strategies; text-to-code and code-to-code, impact the output of ChatGPT's responses in programming tasks. The study adopted an experimental design that presented ChatGPT with a diverse set of programming tasks and prompts, spanning various programming languages, difficulty levels, and problem domains. The generated outputs were rigorously tested and evaluated for accuracy, latency, and qualitative aspects. The findings indicated that code-to-code prompting significantly improved accuracy, achieving a 93.55% success rate compared to 29.03% for text-to-code. Code-to-code prompts were particularly effective across all difficulty levels, while text-to-code struggled, especially with harder tasks. Based on these findings, computer programming students need to appreciate and comprehend that ChatGPT prompting is essential for getting the desired output. Using optimised prompting methods, students can achieve more accurate and efficient code generation, enhancing the quality of their code. Future research should explore the balance between prompt specificity and code efficiency, investigate additional prompting strategies, and develop best practices for prompt design to optimize the use of AI in software development.
Downloads
References
J. Msane, M. B. Mutanga, and T. Chani, ‘Students’ Perception of the effect of cognitive factors in determining student’s success in computer programming’, J. Theor. Appl. Inf. Technol., vol. 98, no. 17, pp. 3607–3618, 2020.
M. Thuné and A. Eckerdal, ‘Analysis of Students’ learning of computer programming in a computer laboratory context’, Eur. J. Eng. Educ., vol. 44, no. 5, pp. 769–786, 2019, doi: 10.1080/03043797.2018.1544609.
B. S. Javier, ‘Understanding their Voices from Within: Difficulties and Code Comprehension of Life-Long Novice Programmers’, Int. J. Arts, vol. 1, no. 1, pp. 53–76, 2021.
M. B. Mutanga, P. X. Piyose, and L. S. Ndovela, ‘Factors Affecting Career Preferences and Pathways : Insights from IT Students’, J. Inf. Syst. Informatics, vol. 5, no. 3, pp. 1111–1122, 2023, doi: 10.51519/journalisi.v5i3.556.
N. Islam, G. Shafi Sheikh, R. Fatima, and F. Alvi, ‘A Study of Difficulties of Students in Learning Programming’, J. Educ. Soc. Sci., vol. 7, no. 2, pp. 38–46, 2019, doi: 10.20547/jess0721907203.
E. Y. İnce, ‘Students’ Perceptions on Learning Programming with CodinGame’, Int. J. Technol. Teach. Learn., vol. 17, no. 1, pp. 38–46, 2021, doi: 10.37120/ijttl.2021.17.1.03.
D. A. Egbe, B. M. Mutanga, and T. Chani, ‘Combating Digital Academic Dishonesty: A Scoping Review of Approaches’, no. April, 2021, doi: 10.35940/ijeat.F1268.089620.
C. S. Cheah, ‘Factors contributing to the difficulties in teaching and learning of computer programming: A literature review’, Contemp. Educ. Technol., vol. 12, no. 2, pp. 1–14, 2020, doi: 10.30935/cedtech/8247.
M. B. Mutanga, ‘The effect of cognitive factors in determining studens’ success in computer programming’, J. Theor. Appl. Inf. Technol., vol. 98, no. 17, pp. 3607–3618, 2020.
C.-H. ; Lai, Y.-K. ; Chen, and Y. Wang, ‘Learning Computer Programming for Students with Medical Fields of’, Int. J. Environ. Reasearch Public Heal., vol. 19, pp. 1–17, 2022, doi: 10.3390/ijerph19106005
W. Takerngsaksiri, C. Warusavitarne, C. Yaacoub, M. H. K. Hou, and C. Tantithamthavorn, ‘Students’ Perspective on AI Code Completion: Benefits and Challenges’, arXiv.org, pp. 1–6, 2023.
J. White, S. Hays, Q. Fu, J. Spencer-Smith, and D. C. Schmidt, ‘ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design’, arXiv:2303.07839, pp. 1–35, 2023.
R. Jagerman, H. Zhuang, Z. Qin, X. Wang, and M. Bendersky, Query Expansion by Prompting Large Language Models, vol. 1, no. 1. Association for Computing Machinery, 2023.
E. Kadir, T. Rahman, and S. Barman, ‘Exploring the Competency of ChatGPT in Solving Competitive Programming Challenges’, Int. J. Adv. Trends Comput. Sci. Eng., vol. 13, no. 1, pp. 13–23, 2023, doi: 10.30534/ijatcse/2024/031312024.
R. Yilmaz and F. G. Karaoglan Yilmaz, ‘Augmented intelligence in programming learning: Examining student views on the use of ChatGPT for programming learning’, Comput. Hum. Behav. Artif. Humans, vol. 1, no. 2, p. 100005, 2023, doi: 10.1016/j.chbah.2023.100005.
B. McCann, N. S. Keskar, C. Xiong, and R. Socher, ‘The Natural Language Decathlon: Multitask Learning as Question Answering’, arXiv, pp. 1–23, 2018.
V. Sanh et al., ‘Multitask Prompted Training Enables Zero-Shot Task Generalization’, in ICLR 2022 - 10th International Conference on Learning Representations, 2022.
S. H. Bach et al., ‘PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts’, in Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2022, pp. 93–104. doi: 10.18653/v1/2022.acl-demo.9.
C. Liu et al., ‘Improving ChatGPT Prompt for Code Generation’, arXiv, pp. 1–12, 2023.
S. Wang and P. Jin, ‘A Brief Summary of Prompting in Using GPT Models’, Qeios, no. April, 2023, doi: 10.32388/IMZI2Q.
T. Lehtinen, C. Koutcheme, and A. Hellas, Let’s Ask AI About Their Programs: Exploring ChatGPT’s Answers To Program Comprehension Questions, vol. 1, no. 1. Association for Computing Machinery, 2024. doi: 10.1145/3639474.3640058.
A. J. Spasic and D. S. Jankovic, ‘Using ChatGPT Standard Prompt Engineering Techniques in Lesson Preparation: Role, Instructions and Seed-Word Prompts’, in 2023 58th International Scientific Conference on Information, Communication and Energy Systems and Technologies, ICEST 2023 - Proceedings, IEEE, 2023, pp. 47–50. doi: 10.1109/ICEST58410.2023.10187269.
G. van den Berg and E. du Plessis, ‘ChatGPT and Generative AI: Possibilities for Its Contribution to Lesson Planning, Critical Thinking and Openness in Teacher Education’, Educ. Sci., vol. 13, no. 10, 2023, doi: 10.3390/educsci13100998.
X. Sun et al., ‘Pushing the Limits of ChatGPT on NLP Tasks’, ArXiv, pp. 1–26, 2023.
A. Suárez, V. Díaz-Flores García, J. Algar, M. Gómez Sánchez, M. Llorente de Pedro, and Y. Freire, ‘Unveiling the ChatGPT phenomenon: Evaluating the consistency and accuracy of endodontic question answers’, Int. Endod. J., vol. 57, no. 1, pp. 108–113, 2024, doi: 10.1111/iej.13985.
K. Kusunose, S. Kashima, and M. Sata, ‘Evaluation of the Accuracy of ChatGPT in Answering Clinical Questions on the Japanese Society of Hypertension Guidelines’, Circ. J., vol. 87, no. 7, pp. 1030–1033, 2023, doi: 10.1253/circj.CJ-23-0308.
Y. Kaneda et al., ‘Evaluating ChatGPT’s effectiveness and tendencies in Japanese internal medicine’, J. Eval. Clin. Pract., no. April, pp. 1–7, 2024, doi: 10.1111/jep.14011.
G. Zuccon and B. Koopman, Dr ChatGPT, tell me what I want to hear: How prompt knowledge impacts health answer correctness, vol. 1, no. 1. Association for Computing Machinery, 2023.
E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell, ‘On the dangers of stochastic parrots: Can language models be too big?’, in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 2021, pp. 610–623. doi: 10.1145/3442188.3445922.
HackerRank, ‘HackerRank’, 2024. https://www.hackerrank.com/ (accessed Jun. 26, 2024).
K. Buffardi and P. Valdivia, ‘Measuring Unit Test Accuracy’, in 50th ACM Technical Symposium on Computer Science Education, 2019, pp. 578–584. doi: 10.1145/3287324.3287351.
Tricentis, ‘64 essential testing metrics for measuring quality assurance success.’, 2016. https://www.tricentis.com/blog/64-essential-testing-metrics-for-measuring-quality-assurance-success (accessed Jun. 04, 2024).
J. Cao, M. Li, M. Wen, and S. Cheung, A study on Prompt Design, Advantages and Limitations of ChatGPT for Deep Learning Program Repair, vol. 1, no. 1. Association for Computing Machinery, 2023.
N. Mungoli, ‘Exploring the Synergy of Prompt Engineering and Reinforcement Learning for Enhanced Control and Responsiveness in Chat GPT’, J. Electr. Electron. Eng., vol. 2, no. 3, pp. 201–205, 2023, doi: 10.33140/jeee.02.03.02.
T. B. Brown et al., ‘Language Models are Few-Shot Learners’, no. NeurIPS, 2020.
X. Liu et al., ‘GPT Understands, Too’, AI Open, 2023, doi: 10.1016/j.aiopen.2023.08.012.
S. Dube et al., Students’ Perceptions of ChatGPT in Education: A Rapid Systematic Literature Review, vol. 4, no. 1. Springer Nature, 2024. doi: 10.3390/proceedings47010031.
Khan Academy, ‘Categorizing run time efficiency’, 2024. https://www.khanacademy.org/computing/ap-computer-science-principles/algorithms-101/evaluating-algorithms/a/comparing-run-time-efficiency (accessed Jun. 26, 2024).
S. Sivarajkumar, M. Kelley, A. Samolyk-mazzanti, S. Visweswaran, and Y. Wang, ‘An Empirical Evaluation of Prompting Strategies for Large Language Models in Zero-Shot Clinical Natural Language Processing’, EJMIR Med. Informatics, vol. 12, 2023, doi: 10.2196/55318.
M. Chen et al., ‘Evaluating Large Language Models Trained on Code’, arXiv Prepr., 2021.
R. Yilmaz, F. Gizem, and K. Yilmaz, ‘The effect of generative artificial intelligence (AI) based tool use on students’ computational thinking skills, programming self-efficacy and motivation’, Comput. Educ. Artif. Intell., vol. 4, pp. 1–14, 2023, doi: 10.1016/j.caeai.2023.100147.
S. Peng, E. Kalliamvakou, P. Cihon, and M. Demirer, ‘The Impact of AI on Developer Productivity : Evidence from GitHub Copilot arXiv : 2302 . 06590v1 [ cs . SE ] 13 Feb 2023’, arXiv Prepr., pp. 1–19, 2023.
Abstract views: 183 times
Download PDF: 149 times
Copyright (c) 2024 Journal of Information Systems and Informatics
This work is licensed under a Creative Commons Attribution 4.0 International License.
- I certify that I have read, understand and agreed to the Journal of Information Systems and Informatics (Journal-ISI) submission guidelines, policies and submission declaration. Submission already using the provided template.
- I certify that all authors have approved the publication of this and there is no conflict of interest.
- I confirm that the manuscript is the authors' original work and the manuscript has not received prior publication and is not under consideration for publication elsewhere and has not been previously published.
- I confirm that all authors listed on the title page have contributed significantly to the work, have read the manuscript, attest to the validity and legitimacy of the data and its interpretation, and agree to its submission.
- I confirm that the paper now submitted is not copied or plagiarized version of some other published work.
- I declare that I shall not submit the paper for publication in any other Journal or Magazine till the decision is made by journal editors.
- If the paper is finally accepted by the journal for publication, I confirm that I will either publish the paper immediately or withdraw it according to withdrawal policies
- I Agree that the paper published by this journal, I transfer copyright or assign exclusive rights to the publisher (including commercial rights)
_1.png){kind=logo}
