Evaluating the Efficacy of AI Tools in Systematic Literature Reviews: A Comprehensive Analysis

Phumzile Dorcus Mogoale; Agnieta Beatrijs Pretorius; Refilwe Constance Mogase; Mmatshuene Anna Segooa

doi:10.51519/journalisi.v7i1.1035

Phumzile Dorcus Mogoale Tshwane University of Technology, South Africa https://orcid.org/0000-0003-1770-5739
Agnieta Beatrijs Pretorius Tshwane University of Technology, South Africa
Refilwe Constance Mogase Tshwane University of Technology, South Africa
Mmatshuene Anna Segooa Tshwane University of Technology, South Africa

DOI: 10.51519/journalisi.v7i1.1035

Keywords: Artificial Intelligence, Systematic Literature Review, Software tools, PRISMA

Abstract

Artificial Intelligence (AI) tools can revolutionize literature review practices by transforming the research landscape towards more efficient and reliable review processes. While conducting literature can be challenging and time-consuming, there is a plethora of AI powered tools which uncover potential solutions to the challenge. AI tools may reduce the time spent on repetitive tasks, allowing scholars to focus more on critical analysis and interpretation. Due to the rising abundance of AI tools, it is difficult to decide which AI tools are best for individual research problems or projects. While concerns exist around the ethical and quality consequences of using AI. The study aims to explore the usage of AI tools on the systematics literature review process, specifically focusing on their effectiveness in various stages and ethical concerns. IEEE and MDPI Journal papers from 2020 to 2024 were reviewed using Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. RobotReviewer, Covidence and EPPI-Reviewer are AI tools commonly used. These AI tools are designed to support different aspects of the systematic literature review process by offering capabilities such as problem formulation, literature search, inclusion screening and quality assessment. AI tools demonstrate improved effectiveness of literature searches, screening processes and data extraction. Language and content presentation, incorrect citation and plagiarism, grammar and spelling errors may be ren when utilizing AI. Concerns related to data quality, biases, and the need for human oversight were identified.

Downloads

Download data is not yet available.

References

E. Purssell and N. McCrae, How to Perform a Systematic Literature Review: A Guide for Healthcare Researchers, Practitioners and Students. Cham: Springer, 2020.

J. Hulland and M. B. Houston, "Why Systematic Review Papers and Meta-Analyses Matter: An Introduction to the Special Issue on Generalizations in Marketing," J. Acad. Mark. Sci., vol. 48, no. 3, pp. 351–359, 2020.

B. Mauer and J. Venecek, "Writing the Literature Review," in Strategies for Conducting Literary Research, 2nd ed., 2022.

W. M. Lim, S. Kumar, and F. Ali, "Advancing Knowledge through Literature Reviews: 'What', 'Why', and 'How to Contribute'," Serv. Ind. J., vol. 42, no. 7–8, pp. 481–513, 2022.

Q. N. Hong and P. Pluye, "Systematic Reviews: A Brief Historical Overview," Educ. Inf., vol. 34, no. 4, pp. 261–276, 2018.

C. Pope, N. Mays, and J. Popay, Synthesising Qualitative and Quantitative Health Evidence: A Guide to Methods. Maidenhead: McGraw-Hill Education, 2007.

D. Antons, C. F. Breidbach, A. M. Joshi, and T. O. Salge, "Computational Literature Reviews: Method, Algorithms, and Roadmap," Organ. Res. Methods, vol. 26, no. 1, pp. 107–138, 2023.

M. K. Linnenluecke, M. Marrone, and A. K. Singh, "Conducting Systematic Literature Reviews and Bibliometric Analyses," Aust. J. Manag., vol. 45, no. 2, pp. 175–194, 2020.

A. B. Brendel, S. Trang, M. Marrone, S. Lichtenberg, and L. M. Kolbe, "What to Do for a Literature Review? A Synthesis of Literature Review Practices," in Proc. AMCIS, 2020.

D. Fan, D. Breslin, J. L. Callahan, and M. Iszatt‐White, "Advancing Literature Review Methodology through Rigour, Generativity, Scope and Transparency," Int. J. Manag. Rev., vol. 24, no. 2, pp. 171–180, 2022.

H. Cooper, Research Synthesis and Meta-Analysis: A Step-by-Step Approach, 5th ed. Los Angeles: Sage Publications, 2015.

F. Rowe, "What Literature Review Is Not: Diversity, Boundaries and Recommendations," Eur. J. Inf. Syst., vol. 23, no. 3, pp. 241–255, 2014.

D. E. Leidner, "Review and Theory Symbiosis: An Introspective Retrospective," J. Assoc. Inf. Syst., vol. 19, no. 6, pp. 552–567, 2018.

G. Paré, M.-C. Trudel, M. Jaana, and S. Kitsiou, "Synthesizing Information Systems Knowledge: A Typology of Literature Reviews," Inf. Manag., vol. 52, no. 2, pp. 183–199, 2015.

J. Webster and R. T. Watson, "Analyzing the Past to Prepare for the Future: Writing a Literature Review," MIS Q., vol. 26, no. 2, pp. xiii–xxiii, 2002.

C. Okoli, "A Guide to Conducting a Standalone Systematic Literature Review," Commun. Assoc. Inf. Syst., vol. 37, no. 1, pp. 879–910, 2015.

W. Bandara, E. Furtmueller, E. Gorbacheva, S. Miskon, and J. Beekhuyzen, "Achieving Rigor in Literature Reviews: Insights from Qualitative Data Analysis and Tool-Support," Commun. Assoc. Inf. Syst., vol. 37, no. 1, pp. 154–204, 2015.

M. J. Page et al., "The PRISMA 2020 Statement: An Updated Guideline for Reporting Systematic Reviews," BMJ, vol. 372, p. n71, 2021, doi: 10.1136/bmj.n71.

A. Ramasamy, "PRISMA 2020: Key Changes and Implementation Aspects," J. Oral Maxillofac. Surg., vol. 80, no. 5, pp. 795–797, 2022, doi: 10.1016/j.joms.2021.12.018.

K. Knobloch, U. Yoon, and P. M. Vogt, "Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement and Publication Bias," J. Cranio-Maxillofac. Surg., vol. 39, no. 2, pp. 91–92, 2011, doi: 10.1016/j.jcms.2010.11.001.

T. Innocenti et al., "Adherence to the PRISMA Statement and Its Association with Risk of Bias in Systematic Reviews Published in Rehabilitation Journals: A Meta-Research Study," Braz. J. Phys. Ther., vol. 26, no. 5, p. 100450, 2022, doi: 10.1016/j.bjpt.2022

E. M. da Silva Júnior and M. L. Dutra, “A roadmap toward the automatic composition of systematic literature reviews,” Iberoam. J. Sci. Meas. Commun., 2021.

R. Van De Schoot et al., “An open source machine learning framework for efficient and transparent systematic reviews,” Nat. Mach. Intell., vol. 3, no. 2, pp. 125–133, 2021.

B. Lainjo and H. Tsmouche, “The impact of artificial intelligence on higher learning institutions,” Int. J. Educ. Teach. Soc. Sci., vol. 3, no. 2, pp. 96–113, 2023.

K. Stolpe and J. Hallström, “Artificial intelligence literacy for technology education,” Comput. Educ. Open, vol. 6, p. 100159, 2024, doi: 10.1016/j.caeo.2024.100159.

H. Harrison, S. J. Griffin, I. Kuhn, and J. A. Usher-Smith, “Software tools to support title and abstract screening for systematic reviews in healthcare: an evaluation,” BMC Med. Res. Methodol., vol. 20, pp. 1–12, 2020.

A. M. Cohen, W. R. Hersh, K. Peterson, and P.-Y. Yen, “Reducing workload in systematic review preparation using automated citation classification,” J. Am. Med. Inform. Assoc., vol. 13, no. 2, pp. 206–219, 2006.

G. Wagner, R. Lukyanenko, and G. Paré, “Artificial intelligence and the conduct of literature reviews,” J. Inf. Technol., vol. 37, no. 2, pp. 209–226, 2022.

M. Segooa, F. Modiba, and I. Motjolopane, “Generative artificial intelligence tools to augment teaching scientific research in postgraduate studies,” S. Afr. J. High. Educ., vol. 39, no. 1, pp. 294–314, 2025.

S. K. Boell and D. Cecez-Kecmanovic, “On being ‘systematic’ in literature reviews,” in Formulating Research Methods for Information Systems, vol. 2, pp. 48–78, 2015.

M. Kozlovska, D. Mackova, and M. Spisakova, “Survey of construction management documentation usage in planning and construction of building project,” Procedia Eng., vol. 161, pp. 711–715, 2016.

J. F. Wolfswinkel, E. Furtmueller, and C. P. Wilderom, “Using grounded theory as a method for rigorously reviewing literature,” Eur. J. Inf. Syst., vol. 22, no. 1, pp. 45–55, 2013.

S. Erduran and O. Levrini, “The impact of artificial intelligence on scientific practices: an emergent area of research for science education,” Int. J. Sci. Educ., pp. 1–8, 2024.

C. Stansfield, J. Thomas, and J. Kavanagh, “‘Clustering’ documents automatically to support scoping reviews of research: a case study,” Res. Synth. Methods, vol. 4, no. 3, pp. 230–241, 2013.

A. Y. Tsou, J. R. Treadwell, E. Erinoff, and K. Schoelles, “Machine learning for screening prioritization in systematic reviews: comparative performance of Abstrackr and EPPI-Reviewer,” Syst. Rev., vol. 9, pp. 1–14, 2020.

A. Gates et al., “The semi-automation of title and abstract screening: a retrospective exploration of ways to leverage Abstrackr’s relevance predictions in systematic and rapid reviews,” BMC Med. Res. Methodol., vol. 20, pp. 1–9, 2020.

N. Fabiano et al., “How to optimize the systematic review process using AI tools,” JCPP Adv., vol. 4, no. 2, p. e12234, 2024.

C. Hamel et al., “An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening–impact on reviewer-relevant outcomes,” BMC Med. Res. Methodol., vol. 20, pp. 1–14, 2020.

J. Li, K. Larsen, and A. Abbasi, “TheoryOn: A design framework and system for unlocking behavioral knowledge through ontology learning,” MIS Q., vol. 44, no. 4, 2020.

S. Boell and B. Wang, “an IT artifact supporting exploratory literature searches for information systems research,” itbaskets, vol. 21, 2021.

B. Sturm and A. Sunyaev, “Design principles for systematic search systems: a holistic synthesis of a rigorous multi-cycle design science research journey,” Bus. Inf. Syst. Eng., vol. 61, pp. 91–111, 2019.

I. J. Marshall, J. Kuiper, and B. C. Wallace, “RobotReviewer: evaluation of a system for automatically assessing bias in clinical trials,” J. Am. Med. Inform. Assoc., vol. 23, no. 1, pp. 193–201, 2016.

V. R. Walker et al., “Evaluation of a semi-automated data extraction tool for public health literature-based reviews: Dextr,” Environ. Int., vol. 159, p. 107025, 2022.

R. Van Dinter, B. Tekinerdogan, and C. Catal, “Automation of systematic literature reviews: A systematic literature review,” Inf. Softw. Technol., vol. 136, p. 106589, 2021.

B. Anuyahong, C. Rattanapong, and I. Patcha, “Analyzing the impact of artificial intelligence in personalized learning and adaptive assessment in higher education,” Int. J. Res. Sci. Innov., vol. 10, no. 4, pp. 88–93, 2023.

R. Butson and R. Spronken-Smith, “AI and its implications for research in higher education: a critical dialogue,” High. Educ. Res. Dev., vol. 43, no. 3, pp. 563–577, 2024, doi: 10.1080/07294360.2023.2280200.

S. Pal, “Artificial intelligence in the research panorama: A dissection of global propensities and quintessential practices,” Int. J. Res. Publ. Rev., vol. 4, no. 5, pp. 3439–3442, 2023.

P. Hamet and J. Tremblay, “Artificial intelligence in medicine,” Metabolism, vol. 69, pp. S36–S40, 2017, doi: 10.1016/j.metabol.2017.01.011.

P. Krish, K. Salehuddin, and N. A. Razak, “Engaging postgraduate students in preparing research proposals,” Akademika, vol. 87, no. 1, pp. 243–252, 2017.

C. Winkler, B. Hammoda, E. Noyes, and M. Van Gelderen, “Entrepreneurship education at the dawn of generative artificial intelligence,” Entrep. Educ., vol. 6, pp. 579–589, 2023.

L. J. Mariano and J. Potane, “Benefits and difficulties of postgraduate student-researchers: A qualitative inquiry,” Int. J. Soc. Sci. Humanit. Invent., vol. 9, no. 3, pp. 6852–6861, 2022.

P. Jost, “The Agent's Smile: Impacts of Artificially Generated Pedagogical Agents on Risk-Taking,” Int. Assoc. Dev. Inf. Soc., 2021.

P. Feussi, E. I. Edoun, and L. Kok, “Overcoming the challenges faced by university students pursuing business research in presenting meaningful and relevant literature,” IUP J. Bus. Strategy, vol. 17, no. 4, 2020.

T. Wang et al., “Exploring the potential impact of artificial intelligence (AI) on international students in higher education: Generative AI, chatbots, analytics, and international student success,” Appl. Sci., vol. 13, no. 11, p. 6716, 2023.

M. A. Segooa, I. Motjolopane, and F. S. Modiba, “Development of a design science artefact to teach computing students: A systematic literature review,” in Proc. ECRM 2023 22nd Eur. Conf. Res. Methods Bus. Manag., Academic Conferences and Publishing Limited, 2023.

J. de la Torre-López, A. Ramírez, and J. R. Romero, “Artificial intelligence to automate the systematic review of scientific literature,” Computing, vol. 105, no. 10, pp. 2171–2194, 2023.

H. Müller, S. Pachnanda, F. Pahl, and C. Rosenqvist, “The application of artificial intelligence on different types of literature reviews—A comparative study,” in Proc. 2022 Int. Conf. Appl. Artif. Intell. (ICAPAI), IEEE, pp. 1–7, 2022.

K. Al-Saedi, M. Al-Emran, T. Ramayah, and E. Abusham, “Developing a general extended UTAUT model for M-payment adoption,” Technol. Soc., vol. 62, p. 101293, 2020, doi: 10.1016/j.techsoc.2020.101293.

[S. Vongjaturapat, S. Chaveesuk, N. Chotikakamthorn, and S. Tongkhambanchong, “Analysis of factor influencing the tablet acceptance for library information services: A combination of UTAUT and TTF model,” J. Inf. Knowl. Manag., vol. 14, no. 3, p. 1550023, 2015, doi: 10.1142/s0219649215500239.

S. Jonnalagadda and D. Petitti, “A new iterative method to reduce workload in systematic review process,” Int. J. Comput. Biol. Drug Des., vol. 6, no. 1–2, pp. 5–17, 2013.