Development and validation of multimedia-based learning product instruments: SEM and Rasch model approaches

Sofyan; Erisa Kurniati; I Wayan Widana; Sabariah; Mazni Binti Muhammad

doi:10.59672/ijed.v7i1.6343

Authors

Sofyan Universitas Jambi
Erisa Kurniati Universitas Jambi
I Wayan Widana Universitas PGRI Mahadewa Indonesia
Sabariah Universitas PGRI Adi Buana
Mazni binti Muhammad Universiti Islam Melaka Malaysia

DOI:

https://doi.org/10.59672/ijed.v7i1.6343

Keywords:

Instrument validation, Learning evaluation, Multimedia-based learning, Rasch model, SEM

Abstract

Despite the rapid expansion of multimedia-based learning in education, the availability of psychometrically validated evaluation instruments remains limited, underscoring the urgent need for reliable, standardized measurement tools. This study aims to develop and validate a multimedia-based learning product evaluation instrument that measures three main constructs: feasibility, practicality, and attractiveness. The study used an instrument development design with an integrated psychometric approach that combines classical measurement theory (Classical Test Theory), Structural Equation Modeling (SEM), and the Rasch Model. Content validity was assessed using expert judgment, as measured by Aiken's V coefficient and the Content Validity Ratio (CVR). Construct validity was tested through Exploratory Factor Analysis (EFA) and Confirmatory Factor Analysis (CFA). Structural analysis was conducted to examine the relationships between constructs, and multi-group analysis (teachers and students) was conducted to test model invariance. Data were collected from 257 respondents, comprising 153 students and 104 teachers. The results showed that the instrument had high content validity (mean Aiken's V = 0.89), very high reliability (α = 0.93), and a good measurement model fit (CFI = 0.96; RMSEA = 0.054). Rasch analysis showed item reliability of 0.96, and no serious misfit was found. Multi-group analysis showed the model was invariant between teachers and students. Thus, this instrument meets modern psychometric standards and is suitable for use in evaluating multimedia-based learning products.

Downloads

Download data is not yet available.

References

Agustian, M., Jerica, Ernestya, T. G., & Ekarina. (2025). Development of interactive multimedia based on cultural literacy for Pancasila education in elementary schools. Indonesian Journal of Educational Development (IJED), 6(3), 1085–1101. https://doi.org/10.59672/ijed.v6i3.5349

Aiken, L. R. (1985). Three coefficients for analyzing the reliability and validity of ratings. Educational and Psychological Measurement, 45(1), 131–142. https://doi.org/10.1177/0013164485451012

Bond, M., Buntins, K., Bedenlier, S., Zawacki-Richter, O., & Kerres, M. (2020). Mapping research in student engagement and educational technology. International Journal of Educational Technology in Higher Education, 17(2). https://doi.org/10.1186/s41239-019-0176-8

Bond, T. G., & Fox, C. M. (2015). Applying the rasch model: Fundamental measurement in the human sciences (3rd ed.). Routledge.

Branch, R. M. (2019). Instructional design: The ADDIE approach. Springer. https://doi.org/10.1007/978-0-387-09506-6

Baber, H. (2021). Modelling the acceptance of e-learning during the COVID-19 pandemic. The International Journal of Management Education, 19(2), 100503. https://doi.org/10.1016/j.ijme.2021.100503

Bedenlier, S., Bond, M., Buntins, K., & Zawacki-Richter, O. (2021). Facilitating student engagement through educational technology. Educational Technology Research and Development, 69, 165–190. https://doi.org/10.1007/s11423-020-09873-4

Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2020). Best practices for developing and validating scales. Frontiers in Public Health, 8, 149. https://doi.org/10.3389/fpubh.2020.00149

Bond, T. G., & Fox, C. M. (2015). Applying the Rasch model. Routledge.

Bond, M., Buntins, K., Bedenlier, S., Zawacki-Richter, O., & Kerres, M. (2020). Mapping research in student engagement and educational technology. International Journal of Educational Technology in Higher Education, 17(2). https://doi.org/10.1186/s41239-019-0176-8

Cheng, Y.-M. (2020). Students’ satisfaction and continuance intention. Computers in Human Behavior, 107, 105993. https://doi.org/10.1016/j.chb.2019.105993

Chen, O., et al. (2022). Multimultimedia design and cognitive load. Computers & Education, 180, 104435. https://doi.org/10.1016/j.compedu.2022.104435

Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008

DeVellis, R. F. (2017). Scale development: Theory and applications (4th ed.). Sage Publications.

Dick, W., Carey, L., & Carey, J. O. (2015). The systematic design of instruction (8th ed.). Pearson.

Eka Setiawati, N. L., Era Marsakawati, N. P., & Ramendra, D. P. (2024). Authentic assessment in action: English teachers’ perception and practices for assessing students’ productive skills at senior high school. Indonesian Journal of Educational Development (IJED), 5(3), 366–375. https://doi.org/10.59672/ijed.v5i3.4219

Habibi, A., Mukminin, A., & Sofyan, S. (2024). Access to digital technology in vocational schools. Education and Information Technologies, 29, 4197–4222. https://doi.org/10.1007/s10639-023-12006-x

Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2019). Multivariate data analysis (8th ed.). Cengage.

Keller, J. M. (2010). Motivational design for learning and performance. Springer. https://doi.org/10.1007/978-1-4419-1250-3

Koo, T. K., & Li, M. Y. (2021). Guidelines for selecting and reporting intraclass correlation coefficients. Journal of Chiropractic Medicine, 20(2), 155–163. https://doi.org/10.1016/j.jcm.2021.02.012

Leppink, J. (2020). Cognitive load theory in educational research. Educational Psychology Review, 32, 1–15. https://doi.org/10.1007/s10648-019-09500-0

Martin, F., Sun, T., & Westine, C. (2020). Systematic review of online teaching. Computers & Education, 159, 104009. https://doi.org/10.1016/j.compedu.2020.104009

Mayer, R. E. (2014). Cognitive theory of multimultimedia learning. Cambridge University Press. https://doi.org/10.1017/CBO9781139547369.005

Megawaty, M., Sulistiyo, U., Wachyuni, S., & Sofyan. (2025). Development of M.E.G.A: A self-assessment expert system to improve English correspondence skills for job-seeking university students. Indonesian Journal of Educational Development (IJED), 6(2), 393–407. https://doi.org/10.59672/ijed.v6i2.5099

Sofyan, S., Habibi, A., Sofwan, M., et al. (2023). TPACK–UoTI validation. Humanities and Social Sciences Communications, 10, 55. https://doi.org/10.1057/s41599-023-01533-0

Sweller, J., Ayres, P., & Kalyuga, S. (2011). Cognitive load theory. Springer. https://doi.org/10.1007/978-1-4419-8126-4

Taber, K. S. (2023). Improving the use of cronbach’s alpha. Research in Science Education, 53, 1021–1035. https://doi.org/10.1007/s11165-022-10052-4

Teo, T., & Zhou, M. (2021). Explaining intention to use technology. Journal of Computing in Higher Education, 33, 301–325. https://doi.org/10.1007/s12528-020-09259-1

Widana, I. W., Sopandi, A. T., Suwardika, I. G. (2021). Development of an authentic assessment model in mathematics learning: A science, technology, engineering, and mathematics (STEM) approach. Indonesian Research Journal in Education, 5(1), 192-209. https://doi.org/10.22437/irje.v5i1.12992

Widana, I. W. & Ratnaya, I. G. (2021). Relationship between divergent thinking and digital literacy on teacher ability to develop HOTS assessment. Journal of Educational Research and Evaluation, 5(4), 516-524. https://doi.org/10.23887/jere.v5i4.35128

Zamanzadeh, V., et al. (2020). Design and implementation of a content validity study: Development of an instrument. Nursing Practice Today, 7(4), 280–289. https://doi.org/10.18502/npt.v7i4.4031