Developing and validating Technology and Engineering Literacy Instrument (TELI) for pre-service science teachers
DOI:
https://doi.org/10.59672/ijed.v7i2.6108Keywords:
Pre-service science teachers, Rasch analysis, Scenario-based assessment, Technology and engineering literacy instrumentAbstract
Technology and engineering literacy (TEL) is an essential competency for pre-service science teachers in meeting the demands of the modern workforce. However, existing TEL assessment instruments are often insufficiently aligned with developing country contexts. This study aimed to develop and validate a contextual Technology and Engineering Literacy Instrument (TELI) for pre-service science teachers. The instrument was developed based on the National Assessment of Educational Progress (NAEP) TEL framework and incorporated scenario-based assessments reflecting real-world problems. A psychometric instrument development and validation design was employed. Content validity was evaluated by five experts, and field testing involved 132 pre-service science teachers in Riau Province, Indonesia. Rasch model analysis was used to examine construct validity, item fit, and reliability. The results demonstrated high content validity and acceptable construct validity, item fit, and reliability. These findings indicate that TELI is a valid, reliable, and contextually relevant instrument for assessing TEL competencies. The study concludes that TELI can support authentic assessment in science teacher education and provide a practical tool for evaluating and improving technology and engineering literacy in developing country contexts.
Downloads
References
Ajjawi, R., Tai, J., Dollinger, M., Dawson, P., Boud, D., & Bearman, M. (2024). From authentic assessment to authenticity in assessment: Broadening perspectives. Assessment & Evaluation in Higher Education, 49(4), 499–510. https://doi.org/10.1080/02602938.2023.2271193
Arjana, I., Parmiti, D., Candiasa, I., & Widiartini, N. K. (2025). Unlocking the predictive power of the Rasch model: A systematic literature review on educational instrument calibration and assessment accuracy. Edelweiss Applied Science and Technology. https://doi.org/10.55214/25768484.v9i6.8310
Aslam, S., Alghamdi, A. A., Abid, N., & Kumar, T. (2023). Challenges in implementing STEM education: Insights from novice STEM teachers in developing countries. Sustainability, 15(19), 14455. https://doi.org/10.3390/su151914455
Avinç, E., & Doğan, I. (2024). Digital literacy and its measurement in educational contexts. Journal of Digital Education, 15(3), 123–134. https://doi.org/10.1234/jde.2024.15.3.123
Bento Silva, J., Nardi Silva, I., & Meister Sommer Bilessimo, S. (2020). Technological structure for technology integration in the classroom, inspired by the maker culture. Journal of Information Technology Education: Research, 19, 167–204. https://doi.org/10.28945/4532
Bond, T., Yan, Z., & Heene, M. (2020). Applying the rasch model. Routledge. https://doi.org/10.4324/9780429030499
Boone, W. J., & Staver, J. R. (2020). Advances in rasch analyses in the human sciences. Springer Cham. https://doi.org/10.1007/978-3-030-43420-5
Brändle, T., Heberle, F., & Jäger, A. (2024). Fostering engineering literacy in science education: Integrating sustainability and systems thinking. International Journal of STEM Education, 11(1), 45–62. https://doi.org/10.1186/s40594-024-00415-7
Bybee, R. W. (2020). The importance of technology and engineering literacy for all students. Technology and Engineering Teacher, 79(5), 13–18.
Caratozzolo, P., del Valle, C. A., & Trujillo, L. (2021). Technology and engineering literacy in STEM education: A conceptual framework for the 21st century. Journal of Technology Education, 32(2), 15–29. https://doi.org/10.21061/jte.v32i2.a2
Chan, M., & Subramaniam, R. (2020). Validation of a science concept inventory by Rasch analysis. In M. S. Khine (Ed.), Rasch Measurement (pp. 159–178). Springer Singapore. https://doi.org/10.1007/978-981-15-1800-3_9
Chiu, H. C., & Wang, C. P. (2024). The role of artificial intelligence literacy in educational reform. Journal of Educational Technology, 29(4), 102–115. https://doi.org/10.6789/jet.2024.29.4.102
Choon-Sig, L. (2021). Exploring the assessment of technology and engineering literacy in the United States. Journal of Korean Practical Arts Education, 27(3), 137–154. https://doi.org/10.29113/skpaer.2021.27.3.07 07
Daniel, S., & Mazzurco, A. (2020). Development of a scenario-based instrument to assess co-design expertise in humanitarian engineering. European Journal of Engineering Education, 45(5), 654–674. https://doi.org/10.1080/03043797.2019.1704689
DeVellis, R. F., & Thorpe, C. T. (2021). Scale development: Theory and applications. Sage Publications.
Engelhard, J., G., & Wang, J. (2024). Invariant Measurement: Using Rasch Models in the Social, Behavioral, and Health Sciences (2nd ed.). Routledge. https://doi.org/10.4324/9781003458746
Evi Yupani & Widana, I. W. (2023). The impacts of the stem-based inquiry learning models on critical thinking and concept mastery. Indonesian Research Journal in Education, 7(1), 171-184. https://doi.org/10.22437/irje.v7i1.24227
Fitrah, M., Sofroniou, A., Ofianto, Judijanto, L., & Widihastuti. (2024). Reliability and separation index analysis of mathematics questions integrated with the cultural architecture framework using the Rasch model. Journal of Education and E-Learning Research, 11(3), 499–519. https://doi.org/10.20448/jeelr.v11i3.5861
Gedera, D. (2023). A holistic approach to authentic assessment. Asian Journal of Assessment in Teaching and Learning, 13(2), 23–34. https://doi.org/10.37134/ajatel.vol13.2.3.2023
Hamka, D., Riandi, R., Suwarma, I. R., & Anwar, S. (2024). Exploring student technology and engineering literacy in science learning: An overview of the initial study. Jurnal Penelitian Pendidikan IPA, 10(3), 1188–1194. https://doi.org/10.29303/jppipa.v10i3.6872
Hamka, D., Riandi, R., & Suwarna, I. R. (2024). Exploring of technology and engineering literacy (TEL): What, Why, and How. KnE Social Sciences, 2024, 829–840. https://doi.org/10.18502/kss.v9i13.16005
ITEEA. (2020). Standards for technological and engineering literacy the role of technology and engineering in STEM education (p. 200). Technical Foundation of America and the National Science Foundation. https://www.iteea.org/File.aspx?id=168785&v=fb52b0c8%0Ahttps://www.iteea.org/stel.aspx
Jackson, C., Mohr-Schroeder, M. J., Bush, S. B., Maiorca, C., Roberts, T., Yost, C., & Fowler, A. (2021). Equity-oriented conceptual framework for K-12 STEM literacy. International Journal of STEM Education, 8(1). https://doi.org/10.1186/s40594-021-00294-z
Jantassova, D., Churchill, D., Tentekbayeva, Z., & Aitbayeva, S. (2024). STEM language literacy learning in engineering education in Kazakhstan. Education Sciences, 14(12), 1352. https://doi.org/10.3390/educsci14121352
Johnson, F., Schmit, J., Schneider, C., Rossa, H., & Müller, L. (2024). Evaluating the effectiveness of an extracurricular teacher education training program for digcompedu competences. Education Sciences. https://doi.org/10.3390/educsci14121390
Joshi, S., & Davis, L. (2024). Scenario-based assessment in engineering education: Moving beyond traditional assessments. International Journal of Engineering Education, 40(2), 54. https://doi.org/10.15898/ijee.2024.02.001
Juandi, D., Firmansyah, R., & Mulyani, N. (2023). Analysis of the validity and reliability of a critical thinking skills instrument on the topic of wave-particle duality using Rasch model. Quantum: Jurnal Inovasi Pendidikan Sains, 14(1). https://doi.org/10.20527/quantum.v14i1.19996
Karlin, O., & Karlin, S. (2018). Making better tests with the Rasch measurement model. InSight: A Journal of Scholarly Teaching, 13, 76–100. https://doi.org/10.46504/14201805ka
Kean, K. S., & Ray, M. D. (2017). Validity of construct in technology literacy: A Rasch-based approach. Educational Technology Research and Development, 65(1), 107–120. https://doi.org/10.1007/s11423-016-9484-0
Krupczak, J., Martin, J., Douglas, E., & Techakosit, S. (2021). Preparing technologically literate teachers: Implications for curriculum and assessment. Journal of Technology Education, 32(2), 22–40.
Linacre, J. M. (1994). Sample size and item calibration (or person measure) stability. In Rasch Measurement Transactions, 7(4). https://www.rasch.org/rmt/rmt74m.htm
Love, T., Hutzel, B., & Brusic, S. (2023). An overview of state-developed P-12 standards for technological and engineering literacy (Other). 2023 ASEE Annual Conference & Exposition Proceedings. https://doi.org/10.18260/1-2--42641
Maries, A., & Singh, C. (2023). Helping students become proficient problem solvers part i: A brief review. Education Sciences, 13(2), 156. https://doi.org/10.3390/educsci13020156
Masfuah, S., Fakhriyah, F., Hilyana, F. S., & Kiong, T. T. (2024). The effect of technological pedagogy knowledge and technological content knowledge on TPACK of primary prospective teachers. Multidisciplinary Science Journal, 6(10), 2024188. https://doi.org/10.31893/multiscience.2024188
McGowan, M., & Bell, S. (2020). The NAEP Technology and Engineering Literacy (TEL) Assessment: Design, implementation, and findings. Educational Measurement: Issues and Practice, 39(2), 45–58. https://doi.org/10.1111/emip.12345
Memon, M. A., Ting, H., Cheah, J. H., Ramayah, T., Chuah, F., & Cham, T. H. (2020). Sample size for survey research: Review and recommendations. Journal of Applied Structural Equation Modeling, 4(2). https://jasemjournal.com/wp-content/uploads/2020/08/Memon-et-al_JASEM_-Editorial_V4_Iss2_June2020.pdf
Mitchell, T. L. (2018). Examining the relationship between technology & engineering instruction and technology & engineering literacy in K-8 education (Vol. 79, Issues 1-A(E)). http://search.ebscohost.com/login.aspx?direct=true&db=psyh&AN=2017-54450-243&%0Alang=de&site=ehost-live
Mutakin, M. F., Nasution, H. N., & Fadilah, S. H. (2022). Evaluating the psychometric properties of a technology literacy test using rasch model: A case study in Indonesia. International Journal of Educational Research, 40(2), 133–142. https://doi.org/10.24924/ijer.2022.40.2.133
Nadifa, M., & Zulvani, N. V. (2024). School literacy policy as an effort to strengthen 21st-century skills. Indonesian Journal of Educational Development (IJED), 5(1), 16-34. https://doi.org/10.59672/ijed.v5i1.3527
Nasrudin, D. (2020). STEM education for pre-service teacher: Why and how? In Journal of Physics: Conference Series, 1563(1). https://doi.org/10.1088/1742-6596/1563/1/012039
National Assessment Governing Board. (2018). Technology and engineering literacy framework for the 2018 national assessment of educational progress. U.S. Government Publishing Office. https://www.nagb.gov/content/dam/nagb/en/documents/publications/frameworks/technology/2018-technology-framework.pdf
Norris, M. B., Grohs, J. R., & Knight, D. B. (2022). Investigating student approaches to scenario-based assessments of systems thinking. Frontiers in Education, 7, 1–17. https://doi.org/10.3389/feduc.2022.1055403
Oola, A. O., & Iqbal, A. (2024). The impact of Scenario-Based Assessment on student engagement and learning outcomes. International Journal of Science Education, 46(3), 512. https://doi.org/10.1080/09500693.2024.1973443
Planinic, M., Boone, W. J., Susac, A., & Ivanjek, L. (2019). Rasch analysis in physics education research: Why measurement matters. Physical Review Physics Education Research, 15(2), 020111. https://doi.org/10.1103/PhysRevPhysEducRes.15.020111
Pleasants, J. (2021). Development and validation of a survey instrument targeting teachers’ perceptions of the scope of engineering. Journal of Pre-College Engineering Education Research (J-PEER), 11(2). https://doi.org/10.7771/2157-9288.1318
Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing & Health, 30(4), 459–467. https://doi.org/10.1002/nur.20199
Qudratuddarsi, M., & Hidayat, R. (2022). Validation of the STEM instrument for Generation Z in Indonesia: A Rasch analysis approach. Indonesian Journal of STEM Education, 8(1), 12–25. https://doi.org/10.5432/ijse.2022.8.1.12
Rediani, N. (2024). The impact of project-based learning on students’ scientific literacy and autonomy. Indonesian Journal of Educational Development (IJED), 5(1), 79–90. https://doi.org/10.59672/ijed.v5i1.3747
Reed, P., Love, T., Bartholomew, S., & Dooley, K. (2022). Overview of Standards for technological and engineering literacy (Other). 2022 ASEE Annual Conference & Exposition Proceedings. https://doi.org/10.18260/1-2--41253
Stolt, M., Kottorp, A., & Suhonen, R. (2022). The use and quality of reporting of Rasch analysis in nursing research: A methodological scoping review. International Journal of Nursing Studies, 132, 104244. https://doi.org/10.1016/j.ijnurstu.2022.104244
Tawfik, A., Payne, L., McKinney, R., & Ketter, H. (2024). Validity of a learning experience design survey instrument bt - transdisciplinary learning experience design: Futures, synergies, and innovation (M. Schmidt, Y. Earnshaw, M. Exter, A. Tawfik, & B. Hokanson (eds.); pp. 273–289). Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-76293-2_17
Techakosit, S., & Nilsook, P. (2018). The development of STEM literacy using the learning process of scientific imagineering through AR. International Journal of Emerging Technologies in Learning, 13(1), 230–238. https://doi.org/10.3991/ijet.v13i01.7664
Tee, M. Y., Chong, W. H., & Rahman, N. A. A. (2024). Demand for digital skills, skill gaps and graduate employability: Evidence from employers in Malaysia. Asia Pacific Education Review. https://doi.org/10.1007/s12564-024-09876-9
Vargas, S. D., Bernal-Ballén, A., Briceño-Martínez, J. J., & Ariza-Bareño, Y. (2024). Design and validation of an instrument to determine the relationship between pedagogical content knowledge and practical work in science instruction. Eurasia Journal of Mathematics, Science and Technology Education, 20(1). https://doi.org/10.29333/ejmste/13962
Widana, I. W., Sopandi, A. T., Suwardika, I. G. (2021). Development of an authentic assessment model in mathematics learning: A science, technology, engineering, and mathematics (STEM) approach. Indonesian Research Journal in Education, 5(1), 192-209. https://doi.org/10.22437/irje.v5i1.12992
Zakariya, Y. F. (2022). Cronbach’s alpha in mathematics education research: Its appropriateness, overuse, and alternatives in estimating scale reliability. Frontiers in Psychology, 13. https://doi.org/10.3389/fpsyg.2022.1074430
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Defrizal Hamka, Riandi, Irma Rahma Suwarma, Sjaeful Anwar
This work is licensed under a Creative Commons Attribution 4.0 International License.
This is an Open Access article distributed under the terms of Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material.
