How Ready Are Students to Using ChatGPT? A Study on Instrument Validity and Reliability
DOI:
https://doi.org/10.24191/jikm.v15i2.9051Keywords:
ChatGPT adoption, Behavioral intention, UTAUT model, Instrument validity, Instrument reliabilityAbstract
This study reports the construct validity and reliability of an instrument designed to measure students’ behavioral intention to use ChatGPT. The instrument is developed based on the Unified Theory of Acceptance and Use of Technology (UTAUT) model. Data were collected from 65 undergraduate students of the Faculty of Information Management at UiTM Puncak Perdana using proportionate stratified sampling. Analyses conducted with SPSS and Smart PLS confirmed that the instrument demonstrated acceptable construct validity and moderate reliability. Internal consistency values included Cronbach’s alpha (0.810–0.929), composite reliability (0.70–0.90), and average variance extracted above 0.50. Model accuracy and relevance (R² and Q²) were within acceptable and moderate ranges, with no collinearity issues detected. Content validity was further established using CVI, S-CVI, and S-CVI/Ave, which exceeded 0.80, indicating strong agreement. Overall, the instrument achieved robust validity and reliability, offering a useful inventory for future research on ChatGPT adoption in education.
References
Abdullayeva, A., & Musayeva, S. (2023). The use of ChatGPT in higher education: Opportunities and challenges. Journal of Educational Technology and Innovation, 6(2), 45–56.
Ahmad, M. (2023). AI-powered personalized learning: Opportunities and challenges in education. International Journal of Emerging Technologies in Learning, 18(4), 25–37.
Albanna, S. (2023). Integrating generative AI in education: Opportunities and risks. Education and Information Technologies, 28(5), 6031–6048.
Ali, A., Khan, S., & Hussain, M. (2023). Exploring the role of ChatGPT in enhancing students’ motivation toward learning. International Journal of Educational Research, 122, 102185. https://doi.org/10.1016/j.ijer.2023.102185
Aldasoro, J. C., Martín, J. A., & Sanz, A. (2019). Artificial intelligence in logistics and supply chain management: A review. International Journal of Production Research, 57(7), 2164–2182.
Augustine, O., & Ali, I. (2021). Artificial intelligence application in academic libraries in Nigeria. Library Philosophy and Practice, 1–16.
Ausat, A., Putra, D., & Nugroho, A. (2023). Enhancing learning engagement with ChatGPT: A case study in Indonesian universities. Journal of Applied Learning and Technology, 5(1), 77–88.
Balugani, E., Mandelli, A., & Vescovi, T. (2018). Artificial intelligence and big data in decision-making: Toward a new research agenda. Journal of Business Research, 93, 109–118.
Balugani, E., Mandelli, A., & Vescovi, T. (2019). AI-based solutions for business innovation: An integrative framework. European Journal of Innovation Management, 22(3), 423–440.
Benuyenah, V. (2023). The promise and peril of ChatGPT in higher education. Journal of Higher Education Policy and Practice, 7(1), 12–19.
Borji, A. (2023). A categorical archive of ChatGPT failures. arXiv preprint arXiv:2302.03494.
Breja, R., Sharma, D., & Singh, A. (2011). Narrow vs general AI: Applications and future. International Journal of Computer Applications, 30(8), 1–6.
Brown, J. (2013). Artificial intelligence and business creation. Harvard Business Review, 91(7), 56–62.
Byrne, B. M. (2010). Structural equation modeling with AMOS: Basic concepts, applications, and programming (2nd ed.). Routledge.
Chaouali, W., Ben Yahia, I., & Souiden, N. (2016). The interplay of counter-conformity motivation, social influence, and trust in customers’ intention to adopt internet banking services. Journal of Retailing and Consumer Services, 28, 209–218.
Chen, L., Xu, Q., & Zhang, C. (2019). Artificial intelligence applications in supply chain: A literature review. Computers & Industrial Engineering, 137, 106024.
Chin, W. W. (1998). The partial least squares approach to structural equation modeling. In G. A. Marcoulides (Ed.), Modern methods for business research (pp. 295–336). Lawrence Erlbaum Associates.
Choi, J., Lee, J., & Park, H. (2018). General AI and robotics: Bridging the gap. Robotics and Autonomous Systems, 104, 45–55.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum.
Colicchia, C., Creazza, A., & Dallari, F. (2019). Linking supply chain resilience and artificial intelligence: A conceptual model. Supply Chain Management, 24(3), 407–420.
Crawford, K., Paglen, T., & Whittaker, M. (2023). Generative AI in education: Risks of dependency and overreliance. AI & Society, 38, 923–936.
Davis, F. D., Bagozzi, R. P., & Warshaw, P. R. (1989). User acceptance of computer technology: A comparison of two theoretical models. Management Science, 35(8), 982–1003.
De Sousa Jabbour, A. B. L., Jabbour, C. J. C., Foropon, C., & Filho, M. G. (2018). When titans meet—Can industry 4.0 revolutionize the environmentally-sustainable manufacturing wave? The role of critical success factors. Technological Forecasting and Social Change, 132, 18–25.
Deshpande, A., Kurhade, S., & Kale, V. (2018). Applications of AI in decision-making. International Journal of Computer Science and Applications, 15(2), 14–23.
Dolgui, A., Ivanov, D., & Sokolov, B. (2018). Ripple effect in the supply chain: An analysis and recent literature. International Journal of Production Research, 56(1–2), 414–430.
Dolgui, A., Ivanov, D., & Sokolov, B. (2019). Supply chain disruption and recovery: Considerations from supply chain viability perspective. Annals of Operations Research, 283, 711–726.
Dwivedi, Y. K., et al. (2023). Adoption of ChatGPT: A research agenda. International Journal of Information Management, 71, 102642. https://doi.org/10.1016/j.ijinfomgt.2023.102642
Dupont, J., Kim, H., & Lee, S. (2018). Artificial intelligence in autonomous driving: Recent developments. IEEE Transactions on Intelligent Vehicles, 3(2), 123–135.
Emon, M. M. H., Rahman, M., & Sultana, S. (2023). Factors influencing ChatGPT adoption behavior among professionals: A UTAUT model approach. International Journal of Emerging Technologies in Learning, 18(5), 32–46.
Fatorachian, H., & Kazemi, H. (2018). A critical investigation of industry 4.0 in manufacturing: Theoretical operationalisation framework. Production Planning & Control, 29(8), 633–644.
Felfel, H., Ben Yahia, W., Ayadi, O., & Masmoudi, F. (2018). Stochastic multi-site supply chain planning in textile and apparel industry under demand and price uncertainties with risk aversion. Annals of Operations Research, 271(2), 551–574. https://doi.org/10.1007/s10479-018-2980-2
Financial Times/New York Times. (2023). Google and Microsoft race to integrate AI chatbots into search engines. The New York Times/Financial Times.
Fornell, C., & Larcker, D. F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18(1), 39–50.
Fuchs, C. (2023). ChatGPT and the new AI capitalism. Monthly Review, 75(2), 32–47.
Garza-Reyes, J. A. (2018). Green lean and the sustainable manufacturing agenda. International Journal of Lean Six Sigma, 9(2), 151–173.
Gefen, D., Straub, D. W., & Boudreau, M. C. (2000). Structural equation modeling and regression: Guidelines for research practice. Communications of the Association for Information Systems, 4(1), 1–70.
Gopalakrishnan, S. (2023). Challenges of large language models in specialized domains. Journal of AI Research and Practice, 12(3), 211–228.
Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2010). Multivariate data analysis (7th ed.). Pearson.
Hair, J. F., Hult, G. T. M., Ringle, C. M., & Sarstedt, M. (2017). A primer on partial least squares structural equation modeling (PLS-SEM) (2nd ed.). SAGE Publications.
Hair, J. F., Sarstedt, M., Hopkins, L., & Kuppelwieser, V. G. (2014). Partial least squares structural equation modeling (PLS-SEM): An emerging tool in business research. European Business Review, 26(2), 106–121.
Hamdi, F., Lachhab, M., & Bouslama, F. (2018). Expert systems in artificial intelligence: A literature review. Expert Systems with Applications, 95, 233–245.
Hazen, B. T., Boone, C. A., Ezell, J. D., & Jones-Farmer, L. A. (2018). Data quality for data science, predictive analytics, and big data in supply chain management. International Journal of Production Economics, 193, 91–99.
Hertzog, M. A. (2008). Considerations in determining sample size for pilot studies. Research in Nursing & Health, 31(2), 180–191.
Holmes, W., & Tuomi, I. (2022). State of AI in education: Trends and implications. European Journal of Education, 57(4), 567–583.
Hong, S. J., Thong, J. Y. L., & Tam, K. Y. (2008). Understanding continued information technology usage behavior: A comparison of three models in the context of mobile internet. Decision Support Systems, 42(3), 1819–1834.
Horng, J.-S., Liu, C.-H. S., Chou, S.-F., Tsai, C.-Y., & Hu, D.-C. (2018). Developing a sustainable service innovation framework for the hospitality industry. International Journal of Contemporary Hospitality Management, 30(1), 455–474. https://doi.org/10.1108/IJCHM-12-2015-0727
Ibrahim, N., & Tain, J. (2016). Structural equation modeling for instrument validation. Journal of Social Science Research, 12(4), 85–94.
Jackson, L. M. (2019). The psychology of prejudice: From attitudes to social action (2nd ed.). American Psychological Association. https://doi.org/10.1037/0000168-000
Kazancoglu, Y., Sezer, M. D., & Ozkan-Ozen, Y. D. (2018). Analyzing barriers to industry 4.0 in sustainable food supply chains. Sustainability, 10(11), 4080.
Keeble-Ramsay, D. R., & Armitage, A. (2010). Theorising the reflective learner: The role of reflection in work-based learning. Journal of Workplace Learning, 22(8), 505–516.
Kesharwani, A., & Singh Bisht, S. (2012). The impact of trust and perceived risk on internet banking adoption in India: An extension of technology acceptance model. International Journal of Bank Marketing, 30(4), 303–322.
Kock, N., & Lynn, G. S. (2012). Lateral collinearity and misleading results in variance-based SEM: An illustration and recommendations. Journal of the Association for Information Systems, 13(7), 546–580.
Kumar, S. (2019). Artificial intelligence in business strategy. Journal of Business Strategy, 40(6), 11–19.
Kumar, S., et al. (2018). Big data analytics in supply chain and logistics management. Annals of Operations Research, 270, 367–380.
Kumar, S., et al. (2019). Artificial intelligence and supply chain disruption management. Computers & Operations Research, 111, 65–77.
Kwak, D., Lee, H., & Lee, J. (2022). Examining user acceptance of AI-based learning systems. Educational Technology Research and Development, 70(2), 543–559.
Lai, M., & Adebayo, T. (2024). Ethical concerns and challenges of large language models in specialized fields. AI Ethics, 4, 215–230.
Larsson, H., Hedberg, J., & Nilsen, J. (2015). Measuring inter-rater reliability for content validity. Nursing Research, 64(3), 202–208.
Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35(6), 382–386.
Martins, C., Oliveira, T., & Popovič, A. (2014). Understanding the internet banking adoption: A unified theory of acceptance and use of technology and perceived risk application. International Journal of Information Management, 34(1), 1–13.
McCarthy, J., Minsky, M. L., Rochester, N., & Shannon, C. E. (2006). A proposal for the Dartmouth summer research project on artificial intelligence, August 31, 1955. AI Magazine, 27(4), 12–14.
McKinsey & Company. (2022). The state of AI in 2022. https://www.mckinsey.com/featured-insights/mckinsey-on-books/state-of-ai
McKinsey & Company. (2023). The state of AI in 2023. https://www.mckinsey.com/featured-insights/mckinsey-on-books/state-of-ai
Menon, A., Singh, R., & Verma, S. (2023). Users’ intention to adopt ChatGPT using UTAUT model. International Journal of Emerging Technologies in Learning, 18(6), 120–135.
Moor, J. (2006). The Dartmouth conference: The next 50 years. AI Magazine, 27(4), 87–91.
Nasim, S., Rehman, A., & Khan, I. (2022). AI adoption across industries: A systematic review. Technological Forecasting and Social Change, 175, 121345.
OpenAI. (2023). GPT-4 technical report. arXiv preprint arXiv:2303.08774.
Pathak, M. (2023). Awareness of ChatGPT among students: A gender-based analysis. International Journal of Advanced Computer Science and Applications, 14(5), 99–107.
Paul, J., Mittal, A., & Srivastav, G. (2016). Impact of validity measures in instrument development. International Journal of Management Research, 14(3), 112–124.
Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 30(4), 459–467.
Ramayah, T., & Chuah, F. (2017). Partial least squares structural equation modeling (PLS-SEM) using SmartPLS 3.0. Pearson Malaysia.
Rahman, H., Ismail, S., & Ali, N. (2016). Instrument validity in educational research. Journal of Social Science and Humanities, 11(2), 105–112.
Richards, G., Yeoh, W., Chong, A. Y. L., & Popovič, A. (2019). Business intelligence effectiveness and corporate performance management: An empirical analysis. Journal of Computer Information Systems, 59(2), 188–196.
Sandu, R., & Gide, E. (2019). The role of Chatbots in education. International Journal of Learning Technology, 14(3), 185–202.
Schilling, L., Dixon, J. K., & Knafi, K. (2007). Determining content validity in nursing research. Nursing Research, 56(6), 441–445.
Shen, X., Chen, Q., & Liu, H. (2022). Student anxiety in adopting AI-based education platforms. Computers & Education, 184, 104495. https://doi.org/10.1016/j.compedu.2022.104495
Shoufan, A. (2023). Exploring students’ perceptions of ChatGPT: Thematic analysis and follow-up survey. IEEE Access, 11, 38805–38818. https://doi.org/10.1109/ACCESS.2023.3268224
Sreedharan, R., Kumar, R., & Menon, V. (2018). AI in decision science. Journal of Decision Analytics, 15(2), 112–123.
TechTRP. (2023). OpenAI ChatGPT Plus subscription in Malaysia: What you need to know. TechTRP. https://techtrp.com
The Star. (2023, February 15). OpenAI launches ChatGPT Plus in Malaysia for RM85/month. The Star. https://www.thestar.com.my
Tlili, A., Shehata, B., Adarkwah, M. A., Bozkurt, A., Hickey, D. T., Huang, R., & Agyemang, B. (2023). What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learning Environments, 10, Article 15. https://doi.org/10.1186/s40561-023-00237-x
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425–478.
Vieriu, D., & Petrea, R. (2025). Artificial intelligence and higher education: Impacts on learning and research. Journal of Educational Futures, 10(1), 1–15.
Winner, L. (2009). Do artifacts have politics? Daedalus, 109(1), 121–136.
Zhai, C. (2022). Implications of ChatGPT in higher education: A literature review. Educational Review, 74(5), 1–15.
Zhai, X., Chu, X., & Chai, C. S. (2021). Integrating artificial intelligence into education: Trends and challenges. Educational Technology Research and Development, 69(4), 1–23.
Zhou, J., Li, X., & Zhang, H. (2023). Examining ChatGPT adoption using TAM and TPB. Journal of Educational Computing Research, 61(7), 1583–1601
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 A'dillah Mustafa, Mohamad Naim Mohd Nor, Kasmarini Baharuddin
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Copyright of articles that appear in the journal belongs exclusively to Faculty of Information Science, Universiti Teknologi MARA (Publisher). This copyright covers the rights to reproduce the article, including reprints, electronic reproductions or any other reproductions of similar nature.
