Validating a diagnostic reading test for junior high school EFL learners in Indonesia's English massive program using QUEST

Nur Hidayati; Erna Andriyanti

doi:10.30603/al.v10i2.6708

Authors

Nur Hidayati English Education Department, Faculty of Languages, Arts and Cultures, Universitas Negeri Yogyakarta
Erna Andriyanti English Education Department, Faculty of Languages, Arts and Cultures, Universitas Negeri Yogyakarta

DOI:

https://doi.org/10.30603/al.v10i2.6708

Keywords:

Diagnostic assessment;, literacy;, QUEST;, reading comprehension

Abstract

Background: Non-formal English education programs often lack rigorous assessment tools, resulting in challenges in evaluating student performance and guiding instructional improvements. One such program, the English Massive Program (EMAS) in Kediri, East Java, Indonesia, serves as a community-driven initiative focused on enhancing English proficiency. However, the quality of its assessments, especially in reading comprehension, remains a critical concern.

Aims: This study aims to investigate how item analysis using the QUEST application can enhance the quality of diagnostic assessment and instructional strategies in non-formal English education, specifically within the EMAS program. The focus is on analysing reading comprehension tests to identify weaknesses and propose improvements in test construction.

Methods: This exploratory study analysed 30 multiple-choice reading comprehension items completed by 26 junior high school students participating in the EMAS program. The QUEST application was employed to assess item difficulty, discrimination, and distractor efficiency.

Results: The results showed that while most discrimination indices were within acceptable ranges, many items, especially the distractors, were too simple and ineffective. This resulted in insufficient and unbalanced discrimination values, indicating that the test items did not optimally differentiate among varying student ability levels.

Implications: The study underscores the importance of integrating psychometric-based diagnostic tools in community education settings. It demonstrates how such analysis can empower educators with practical insights to improve test design, thereby enhancing assessment quality and pedagogy. The research calls for more advanced diagnostic assessment methods to support literacy and instructional planning in low-resource, non-formal educational environments.

Downloads

Download data is not yet available.

References

Adams, R. J., & Khoo, S.-T. (1996). ACER Quest: The interactive test analysis system. Melbourne, Australia: ACER Press. https://research.acer.edu.au/measurement/3/

Afflerbach, P. (2025). Understanding and using reading assessment, K-12. Guilford Publications.https://www.guilford.com/books/Understanding-and-Using-Reading-Assessment-K-12/Peter-Afflerbach/9781462556120

Almeida, F., & Morais, J. (2024). Non-formal education as a response to social problems in developing countries. E-Learning and Digital Media, 22, 1–17. https://doi.org/10.1177/20427530241231843

Arikunto, S. (2018). Dasar-dasar evaluasi pendidikan (Edisi ke-3). Jakarta: PT Bumi Aksara. https://books.google.co.id/books/about/Dasar_Dasar_Evaluasi_Pendidikan_Edisi_3.html?id=j5EmEAAAQBAJ&redir_esc=y

Bayley, T., Wheatley, D., & Hurst, A. (2021). Assessing a novel problem‐based learning approach with game elements in a business analytics course. Decision Sciences Journal of Innovative Education, 19(3), 185–196. https://doi.org/10.1111/dsji.12246

Bhat, S. K., & Prasad, K. H. L. (2021). Item analysis and optimizing multiple-choice questions for a viable question bank in ophthalmology: A cross-sectional study. Indian Journal of Ophthalmology, 69(2), 343–346. https://doi.org/10.4103/ijo.IJO_1610_20

Black, P., & Wiliam, D. (2009). Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability, 21(1), 5–31. https://doi.org/10.1007/s11092-008-9068-5

Brookhart, S. M. (2024). Educational assessment knowledge and skills for teachers revisited. Education Sciences, 14(7). 1-15. https://doi.org/10.3390/educsci14070751

Brown, H. D., & Brown, H. (2018). Language assessment: Principles and classroom practices. Pearson Education. https://books.google.co.id/books?id=a7nqswEACAAJ

Bruner, J. (1966). Toward a theory of instruction. Harvard University Press. https://books.google.co.id/books?id=F_d96D9FmbUC

Callahan, C. M. (2023). Evaluation for decision-making: The Practitioner’s guide to program evaluation. In Systems and models for developing programs for the gifted and talented (pp. 119–142). Routledge.

Carliner, S. (2023). Informal learning basics. ASTD Press. https://books.google.co.id/books/about/Informal_Learning_Basics.html?id=WvnJEAAAQBAJ&redir_esc=y

Catts, H. W. (2022). Rethinking how to promote reading comprehension. American Educator, 45(4), 26. https://files.eric.ed.gov/fulltext/EJ1322088.pdf

Choi, Y., & Zhang, D. (2021). The relative role of vocabulary and grammatical knowledge in L2 reading comprehension: A systematic review of literature. International Review of Applied Linguistics in Language Teaching, 59(1), 1–30. https://doi.org/10.1515/iral-2017-0033

Cook, K. S., Fogelberg, K., Butterbrodt, P., Jolley, K., Raghavan, M., & Smith, J. R. (2023). Assessing student learning: Exams, quizzes, and remediation. In K. Fogelberg (Ed.), Educational Principles and Practice in Veterinary Medicine (pp. 287–312). Wiley-Blackwell.

Dewi, H. H., Damio, S. M., & Sukarno, S. (2023). Item analysis of reading comprehension questions for English proficiency test using Rasch model. REID (Research and Evaluation in Education), 9(1), 24–36. https://doi.org/10.21831/reid.v9i1.53514

Ebel, R.L. (1965). Measuring educational achievement. Prentice-Hall.

Fan, T., Song, J., & Guan, Z. (2021). Integrating diagnostic assessment into curriculum: A theoretical framework and teaching practices. Language Testing in Asia, 11(1), 2. https://doi.org/10.1186/s40468-020-00117-y

Firdaus, A. R., Wulandarie, E., Cantika, M., & Suryana, T. G. S. (2025). Development and validation of diagnostic instrument to identify student misconceptions in Vector material. Journal of Innovative Physics Education Research, 1(1), 1–14. https://doi.org/10.61142/jiper.v1i1.195

Gkintoni, E., Antonopoulou, H., Sortwell, A., & Halkiopoulos, C. (2025). Challenging cognitive load theory: The role of educational Neuroscience and artificial intelligence in redefining learning efficacy. Brain Sciences, 15(2). https://doi.org/10.3390/brainsci15020203

Gore, J., Jaremus, F., & Miller, A. (2022). Do disadvantaged schools have poorer teachers? Rethinking assumptions about the relationship between teaching quality and school-level advantage. The Australian Educational Researcher, 49(4), 635–656. https://doi.org/10.1007/s13384-021-00460-w

Gunawardena, M., Bishop, P., & Aviruppola, K. (2024). Personalized learning: The simple, the complicated, the complex and the chaotic. Teaching and Teacher Education, 139, 104429. https://doi.org/10.1016/j.tate.2023.104429

Harris, J. (2022). Adult English learners with limited or interrupted formal education in diverse learning settings. In L. J. Pentón Herrera (Ed.), English and students with limited or interrupted formal education: Global perspectives on teacher preparation and classroom practices (pp. 43–59). Springer. https://doi.org/10.1007/978-3-030-86963-2_4

Ikhsanudin, I., Novaliah, N., Hidayatullah, H., & Almizi, M. (2023). A practical using of the quest program to analyze the characteristics of the test items in educational measurement. JISAE: Journal of Indonesian Student Assessment and Evaluation, 9(1), 37–43. https://doi.org/10.21009/jisae.v9i1.31163

Izard, J. (2005). Trial testing and item analysis in test construction (Module 7, Quantitative research methods in educational planning). UNESCO International Institute for Educational Planning. https://unesdoc.unesco.org/ark:/48223/pf000014260

Levy-Feldman, I. (2025). The Role of assessment in improving education and promoting educational equity. Education Sciences, 15(2). https://doi.org/10.3390/educsci15020224

Liu, C.-C., Liu, S.-J., Hwang, G.-J., Tu, Y.-F., Wang, Y., & Wang, N. (2023). Engaging EFL students’ critical thinking tendency and in-depth reflection in technology-based writing contexts: A peer assessment-incorporated automatic evaluation approach. Education and Information Technologies, 28(10), 13027–13052. https://doi.org/10.1007/s10639-023-11697-6

Murphy, D. H., Little, J. L., & Bjork, E. L. (2023). The value of using tests in education as tools for learning—not just for assessment. Educational Psychology Review, 35(3), 89. https://doi.org/10.1007/s10648-023-09808-3

Obama, P. B., & Dewey, J. (2022). Assessment: Formal and informal. Teaching Middle Level Social Studies: A Practical Guide for 4th-8th Grade, 141.

Robillard, J. M., Jun, J. H., Lai, J.-A., & Feng, T. L. (2018). The QUEST for quality online health information: Validation of a short quantitative tool. BMC Medical Informatics and Decision Making, 18(1), 87. https://doi.org/10.1186/s12911-018-0668-9

Sukarno, S., Putro, N. H. P. S., Fitrianingsih, I., Alsamiri, Y. A., Gharamah, F. M. A., & Tatin, I. A. G. (2024). Exploring the perceptions of literacy in assessment for learning among high school English teachers. REID (Research and Evaluation in Education), 10(2), 143–154. https://doi.org/10.21831/reid.v10i2.71324

Suskie, L. (2018). Assessing student learning: A common sense guide (3rd ed.). Jossey-Bass.

Suwarto. (2007). Tingkat kesukaran, daya beda, dan reliabilitas tes menurut teori tes klasik. Jurnal Pendidikan, 16(2), 166–178. http://portalgaruda.fti.unissula.ac.id/?ref=browse&mod=viewarticle&article=268287

Tsagari, D., & Armostis, S. (2025). Contextualizing language assessment literacy: A comparative study of teacher beliefs, practices, and training needs in Norway and Cyprus. Education Sciences, 15(7). https://doi.org/10.3390/educsci15070927

Tukiyo, T., Efendi, M., Solissa, E. M., Yuniwati, I., & Pranajaya, S. A. (2023). The development of a two-tier diagnostic test to detect student’s misconceptions in learning process. Mudir: Jurnal Manajemen Pendidikan, 5(1), 92–96. https://doi.org/10.55352/mudir.v5i1.33

Vygotsky, L. S., & Cole, M. (1978). Mind in society: Development of higher psychological processes. Harvard University Press. https://books.google.co.id/books?id=RxjjUefze_oC

Zhang, Y., Su, Y., Liu, Y., Wang, X., Burgess, J., Sui, E., Wang, C., Aklilu, J., Lozano, A., & Wei, A. (2025). Automated generation of challenging multiple-choice questions for vision language model evaluation. arXiv Preprint arXiv:2501.03225