Abstracts
Abstract
Despite the grand demand to receive diagnostic information about students’ difficulties in reading, there are very few tests specifically designed for diagnostic purposes. Therefore, many researches in cognitive diagnostic approach (CDA) use large-scale test results to provide fine and reliable diagnostic feedback on the strengths and weaknesses of students other than the total scores or percentiles ranks, which allow appropriate intervention. This study shows an example of the application of diagnostic modeling using data from 4,762 Canadian students who completed booklet 13 of the PIRLS test in 2011. The results highlight the potential for detailed diagnostic feedback of students’ strengths and weaknesses on the underlying skills identified in the test.
Keywords:
- cognitive diagnostic approach (CDA),
- reading,
- DINA,
- G-DINA,
- diagnostic classification models (DCM),
- large-scale tests
Résumé
Malgré une importante demande de recevoir des informations diagnostiques sur les difficultés en lecture des élèves, il existe très peu d’outils d’évaluation conçus spécifiquement pour cet usage. Plusieurs recherches en approche diagnostique cognitive (ADC) utilisent donc les résultats d’épreuves à grande échelle pour fournir de la rétroaction diagnostique fine et fiable sur les forces et les faiblesses des élèves. Les modélisations de données permettent de s’éloigner des scores ou des rangs percentiles habituellement obtenus, et de fournir des pistes d’intervention appropriées. Cette étude vise à vérifier la faisabilité d’appliquer des modélisations à visée diagnostique aux résultats de 4762 élèves canadiens ayant fait le cahier 13 du test du PIRLS de 2011. Les résultats suggèrent un potentiel de recevoir de la rétroaction diagnostique détaillée de leurs forces et faiblesses sur les habiletés sous- jacentes du test.
Mots-clés :
- approche diagnostique cognitive (ADC),
- lecture,
- DINA,
- G-DINA,
- modèles de classification diagnostique (MCD),
- épreuves à grande échelle
Resumo
Apesar da importante procura por informações diagnósticas sobre as dificuldades de leitura dos alunos, existem muito poucas ferramentas de avaliação concebidas especificamente para este uso. Diversas investigações em abordagem de diagnóstica cognitiva (ADC) utilizam, portanto, os resultados de testes em larga escala para fornecer feedback diagnóstico detalhado e fiável sobre os pontos fortes e fracos dos alunos. As modelizações de dados torna possível afastar-se das pontuações ou dos níveis percentuais normalmente obtidos e fornecer pistas de intervenção apropriadas. Este estudo tem como objetivo verificar a viabilidade da aplicação da modelização diagnóstica aos resultados de 4.762 alunos canadianos que realizaram o caderno 13 do teste do PIRLS de 2011. Os resultados realçam o potencial para um feedback diagnóstico detalhado dos pontos fortes e fracos dos alunos em relação às habilidades subjacentes ao teste.
Palavras chaves:
- abordagem diagnóstica cognitiva (ADC),
- leitura,
- DINA,
- G-DINA,
- modelos de classificação diagnóstica (MCD),
- testes em larga escala
Appendices
Références
- Adams, M. J., & Collins, A. M. (1979). A schema-theoretic view of reading. In R. O. Fredolle (Ed.), Discourse processing: Multidisciplinary Perspectives (pp. 1-22). Norwood, NJ: Ablex.
- Afflerbach, P., & Cho, B. (2009). Identifying and describing constructively responsive comprehension strategies in new and traditional forms of reading. In S. E. Israel & G. G. Duffy (Eds.), Handbook of research on reading comprehension (pp. 69-90). New York, NY: Routledge.
- Alderson, J. C. (2005). Assessing reading. Stutgart, Denmark: Ernst Klett Sprachen.
- Alderson, J. C. (2010). “Cognitive diagnosis and Q-matrices in language assessment”: A commentary. Language Assessment Quarterly, 7(2), 96-103. doi: 10.1080/15434300903426748
- Alderson, J. C., Percsich, R., & Szabo, G. (2000). Sequencing as an item type. Language Testing, 17(4), 423-447. doi: 10.1177/026553220001700403
- Alexander, P. A., & Jetton, T. L. (2000). Learning from text: A multidimensional and developmental perspective. Handbook of reading research, 3, 285-310. doi: 10.4324/9781410605023.ch19
- An, S. (2013). Schema theory in reading. Theory and Practice in Language Studies, 3(1), 130-134. doi: 10.4304/tpls.3.1.130-134
- Anderson, R. C., & Pearson, P. D. (1984). A schema-theoretic view of basic processes in reading comprehension. In P. D. Pearson (Ed.), Handbook of reading research (vol. 1, pp. 255-291). New York, NY: Longman.
- Arican, M., & Sen, S. (2015). A diagnostic comparison of Turkish and Korean students’ mathematics performances on the TIMSS 2011 assessment. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi/Journal of Measurement and Evaluation in Education and Psychology, 6(2), 238-253. doi: 10.13140/RG.2.1.1262.5362
- Basokcu, T. O. (2014). Classification accuracy effects of Q-matrix validation and sample size in DINA and G-DINA models. Journal of Education and Practice, 5(6), 220-230. Retrieved from https://www.iiste.org/Journals/index.php/JEP/article/view/11253/11543
- Buck, G., & Tatsuoka, K. (1998). Application of the rule-space procedure to language testing: Examining attributes of a free response listening test. Language Testing, 15(2), 119-157. doi: 10.1177/026553229801500201
- Buck, G., Tatsuoka, K., & Kostin, I. (1997). The subskills of reading: Rule-space analysis of a multiple-choice test of second language reading comprehension. Language Learning, 47(3), 423-466. doi: 10.1111/0023-8333.00016
- Carrell, P. L. (1983). Some issues in studying the role of schemata, or background knowledge, in second language comprehension. Reading in a Foreign Language, 1(2), 81-92. Retrieved from http://nflrc.hawaii.edu/rfl/PastIssues/rfl12carrell.pdf
- Carrell, P. L., Devine, J., & Eskey, D. E. (Eds.). (1988). Interactive approaches to second language reading. Cambridge, UK: Cambridge University Press.
- Chen, J., de la Torre, J., & Zhang, Z. (2013). Relative and absolute fit evaluation in cognitive diagnosis modeling. Journal of Educational Measurement, 50(2), 123-140. doi: 10.1111/j.1745-3984.2012.00185.x
- Clay, M. M. (1991). Becoming literate: The construction of inner control. Portsmouth, NH: Heinemann.
- Cui, Y., Gierl, M. J., & Chang, H.-H. (2012). Estimating classification consistency and accuracy for cognitive diagnostic assessment. Journal of Educational Measurement, 49(1), 19-38. doi: 10.1111/j.1745-3984.2011.00158.x
- de la Torre, J. (2009). A cognitive diagnosis model for cognitively based multiple- choice options. Applied Psychological Measurement, 33(3), 163-183. doi: 10.1177/0146621608320523
- de la Torre, J. (2011). The generalized DINA model framework. Psychometrika, 76(2), 179-199. doi: 10.1007/s11336-011-9214-8
- de la Torre, J., & Douglas, J. A. (2008). Model evaluation and multiple strategies in cognitive diagnosis: An analysis of fraction subtraction data. Psychometrika, 73(4), 595-624. doi: 10.1007/s11336-008-9063-2
- Desrosiers, H. & Tétreault, K. (2012). Les facteurs liés à la réussite aux épreuves obligatoires de français en sixième année du primaire: un tour d’horizon. Québec, QC: Institut de la statistique du Québec. Repéré à http://www.stat.gouv.qc.ca/statistiques/education/prescolaire-primaire/reussite-epreuve-francais.html.
- DiBello, L. V., Roussos, L. A., & Stout, W. (2007). Review of cognitively diagnostic assessment and a summary of psychometric models. In C. V. Rao & S. Sinharay (Eds.), Handbook of statistics (pp. 979-1027). Amsterdam, Netherlands: Elsevier.
- Dogan, E., & Tatsuoka, K. (2008). An international comparison using a diagnostic testing model: Turkish students’ profile of mathematical skills on TIMSS-R. Educational Studies in Mathematics, 68(3), 263-272. doi: 10.1007/s10649-007-9099-8
- Dubin, F., Eskey, D. E., Grabe, W., & Savignon, S. (1986). Teaching second language reading for academic purposes. Reading, MA: Addison-Wesley.
- Evran, D. (2019). An application of cognitive diagnosis modeling in TIMSS: A comparison of intuitive definitions of Q-Matrices. International Journal of Modern Education Studies, 3(1), 4-17. Retrieved from https://www.ijonmes.net/index.php/ijonmes/article/view/33
- Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378-382. doi: 10.1037/h0031619
- Freedle, R., & Kostin, I. (1993). The prediction of TOEFL reading item difficulty: Implications for construct validity. Language Testing, 10(2), 134-169. doi: 10.1177/026553229301000203
- Gao, L. (2006). Toward a cognitive processing model of MELAB reading test item performance. In J. S. Johnson (Ed.), Spaan fellow working papers in second or foreign language assessment (vol. 4, pp. 1-39). Ann Arbor, MI: University of Michigan. Retrieved from https://www.researchgate.net/profile/Shudong_Wang4/publication/251842392_Validation_and_Invariance_of_Factor_Structure_of_the_ECPE_and_MELAB_across_Gender/links/0f31753889dcc4ef54000000/Validation-and-Invariance-of-Factor-Structure-of-the-ECPE-and-MELAB-across-Gender.pdf
- García, G. (1991). Factors influencing the English reading test performance of Spanish- speaking Hispanic children. Reading Research Quarterly, 26(4), 371-392. doi: 10.2307/747894
- Giasson, J. (1996). La compréhension en lecture. Bruxelles, Belgique: De Boeck Supérieur.
- Gierl, M. J., Cui, Y., & Hunka, S. (2008). Using connectionist models to evaluate examinees’ response patterns to achievement tests. Journal of Modern Applied Statistical Methods, 7(1), 234-245. doi: 10.22237/jmasm/1209615480
- Grabe, W. (1991). Current developments in second language reading research. TESOL Quarterly, 25(3), 375-406. doi: 10.2307/3586977
- Grabe, W. (2000). Reading research and its implications for reading assessment. In A. J. Kunnan (Ed.), Fairness and validation in language assessment (pp. 226-262). Cambridge, UK: Cambridge University Press.
- Hartz, S. M. (2002). A Bayesian framework for the unified model for assessing cognitive abilities: Blending theory with practicality (Unpublished doctoral dissertation). University of Illinois Urbana-Champaign, Champaign, IL.
- Huang, H. Y., & Wang, W. C. (2014). The random-effect DINA model. Journal of Educational Measurement, 51(1), 75-97. doi: 10.1111/jedm.12035
- Im, S., & Park, H. J. (2010). A comparison of US and Korean students’ mathematics skills using a cognitive diagnostic testing method: Linkage to instruction. Educational Research and Evaluation, 16(3), 287-301. doi: 10.1080/13803611.2010.523294
- Irwin, J. W. (1991). Teaching reading comprehension processes. Englewood, NJ: Prentice- Hall.
- Jang, E. E. (2005). A validity narrative: Effects of reading skills diagnosis on teaching and learning in the context of NG TOEFL (Unpublished doctoral dissertation). University of Illinois Urbana-Champaign, Champaign, IL.
- Jang, E. E. (2009). Cognitive diagnostic assessment of L2 reading comprehension ability: Validity argument for fusion model application to LanguEdge assessment. Language Testing, 26(1), 31-73. doi: 10.1177/0265532208097336
- Junker, B. W., & Sijtsma, K. (2001). Cognitive assessment models with few assumptions, and connections with nonparametric item response theory. Applied Psychological Measurement, 25(3), 258-272. doi: 10.1177/01466210122032064
- Kasai, M. (1997). Application of the rule-space model to the reading comprehension section of the Test of English as a Foreign Language (TOEFL) (Unpublished doctoral dissertation). University of Illinois Urbana-Champaign, Champaign, IL.
- Kintsch, W. (1988). The role of knowledge in discourse comprehension: A construction- integration model. Psychological Review, 95(2), 163-182. doi: 10.1037/0033-295X.95.2.163
- Labrecque, M., Chuy, M., Brochu, P., & Houme, K. (2012). PIRLS 2011: le contexte au Canada. Toronto, ON: CMEC.
- Langer, J. A. (1990). The process of understanding: Reading for literary and informative purposes. Research in the Teaching of English, 24(3), 229-260. Retrieved from http://www.jstor.org/stable/40171165.
- Lee, Y.-S., Johnson, M., Park, J. Y., Sachdeva, R., Zhang, J., & Waldman, M. (2013, April). A multidimensional scaling (MDS) approach for investigating students’ cognitive weakness and strength on the TIMSS 2007 mathematics assessment. Paper presented at the 2013 Annual Conference of the American Educational Research Association, San Francisco, CA. Retrieved from https://www.researchgate.net/publication/244988418_A_MDS_Approach_for_Investigating_Student’s_Cognitive_Weakness_and_Strength_on_the_TIMSS_2007_Mathematics_Assessment
- Lee, Y.-S., Park, Y. S., & Taylan, D. (2011). A cognitive diagnostic modeling of attribute mastery in Massachusetts, Minnesota, and the U.S. national sample using the TIMSS 2007. International Journal of Testing, 11(2), 144-177. doi: 10.1080/15305058.2010.534571
- Lee, Y.-W., & Sawaki, Y. (2009). Cognitive diagnosis approaches to language assessment: An overview. Language Assessment Quarterly, 6(3), 172-189. doi: 10.1080/15434300902985108
- Lee, Y.-W, & Sawaki, Y. (2011). Application of three cognitive diagnosis models to ESL reading and listening assessments. Language Assessment Quarterly, 6(3), 239-263. doi: 10.1080/15434300903079562
- Leighton, J. P., & Gierl, M. J. (2007). Defining and evaluating models of cognition used in educational measurement to make inferences about examinees’ thinking processes. Educational Measurement: Issues and Practice, 26(2), 3-16. doi: 10.1111/j.1745-3992.2007.00090.x
- Li, H. (2011). A cognitive diagnostic analysis of the MELAB reading test. In J. S. Johnson (Ed.), Spaan fellow working papers in second or foreign language assessment (vol. 9, pp. 17-46). Ann Arbor, MI: University of Michigan. Retrieved from https://www.academia.edu/9788619/A_cognitive_diagnostic_analysis_of_the_MELAB_reading_test
- Li, H., & Suen, H. K. (2013). Constructing and validating a Q-matrix for cognitive diagnostic analyses of a reading test. Educational Assessment, 18(1), 1-25. doi: 10.1080/10627197.2013.761522
- Loye, N. (2010). 2010, odyssée des modèles de classification diagnostique (MCD). Mesure et évaluation en éducation, 33(3), 75-98. doi: 10.7202/1024892ar
- Loye, N., & Lambert-Chan, J. (2016). Au coeur du développement d’une épreuve en mathématique dotée d’un potentiel diagnostique. Mesure et évaluation en éducation, 39(3), 29-57. doi: 10.7202/1040136ar
- Ma, W., Iaconangelo, C., & de la Torre, J. (2016). Model similarity, model selection, and attribute classification. Applied Psychological Measurement, 40(3), 1-18. doi: 10.1177/0146621615621717
- Ma, X., & Meng, Y. (2014). Towards personalized English learning diagnosis: Cognitive diagnostic modelling for EFL listening. Asian Journal of Education and e-Learning, 2(5), 336-348. Retrieved from https://ajouronline.com/index.php/AJEEL/article/view/1669
- Mullis, I. V., Martin, M. O., Kennedy, A. M., Trong, K. L., & Sainsbury, M. (2009). PIRLS 2011 assessment framework. Amsterdam, Netherlands: International Association for the Evaluation of Educational Achievement.
- Pagani, L. S., Fitzpatrick, C., Belleau, L., & Janosz, M. (2011). Prédire la réussite scolaire des enfants en quatrième année à partir de leurs habiletés cognitives, comportementales et motrices à la maternelle. Étude longitudinale du développement des enfants du Québec (ÉLDEQ 1998-2010): de la naissance à 10 ans. Repéré à http://www.jesuisjeserai.stat.gouv.qc.ca/publications/fascicule_reussite_scol_fr.pdf
- Ravand, H., Barati, H., & Widhiarso, W. (2013). Exploring diagnostic capacity of a high stakes reading comprehension test: A pedagogical demonstration. Iranian Journal of Language Testing, 3(1), 11-37. Retrieved from https://www.researchgate.net/publication/281558416_Exploring_Diagnostic_Capacity_of_a_High_Stakes_Reading_Comprehension_Test_A_Pedagogical_Demonstration
- Rumelhart, D. E. (1978). Schemata: The building blocks of cognition. San Diego, CA: Center for Human Information Processing, University of California.
- Rupp, A., Templin, J., & Henson, R. A. (2010). Diagnostic measurement: Theory, methods, and applications. New York, NY: Guilford Press.
- Samuels, S. J., & Kamil, M. L. (1984). 7 Models of the reading process. In P. D. Pearson (Ed.), Handbook of reading research (pp. 185-224). Mahwah, NJ: Lawrence Erlbaum.
- Scott, H. S. (1998). Cognitive diagnostic perspectives of a second language reading test (Unpublished doctoral dissertation). University of Illinois Urbana-Champaign, Champaign, IL.
- Sinharay, S. (2004). Experiences with Markov chain Monte Carlo convergence assessment in two psychometric examples. Journal of Educational and Behavioral Statistics, 29(4), 461-488. doi: 10.3102/10769986029004461
- Snow, C. (2002). Reading for understanding: Towards an R&D program in reading comprehension. Santa Monica, CA: Rand.
- Stanovich, K. E. (1980). Toward an interactive-compensatory model of individual differences in the development of reading fluency. Reading Research Quarterly, 16(1), 32-71. doi: 10.2307/747348
- Templin, J., & Henson, R. A. (2006). Measurement of psychological disorders using cognitive diagnosis models. Psychological Methods, 11(3), 287-305. doi: 10.1037/1082-989X.11.3.287
- Terzi, R., & Sen, S. (2019). A nondiagnostic assessment for diagnostic purposes: Q-matrix validation and item-based model fit evaluation for the TIMSS 2011 assessment. SAGE Open, 9(1), 1-11. doi: 10.1177/2158244019832684
- Tjoe, H., & de la Torre, J. (2014). The identification and validation process of proportional reasoning attributes: An application of a cognitive diagnosis modeling framework. Mathematics Educational Research Journal, 26(2), 237-255. doi: 10.1007/s13394-013- 0090-7
- Toker, T., & Green, K. (2012, April). An application of cognitive diagnostic assessment on TIMMS-2007 8th grade mathematics items. Paper presented at the Annual Meeting of the American Educational Research Association, Vancouver, BC. Retrieved from https://files.eric.ed.gov/fulltext/ED543803.pdf
- van Dijk, T. A., & Kintsch, W. (1983). Strategies of discourse comprehension. New York, NY: Academic Press.
- von Davier, M. (2008). A general diagnostic model applied to language testing data. Princeton, NJ: ETS.
- Wafa, M. N. (2019). Assessing school students’ mathematic ability using DINA and DINO models. International Journal of Mathematics Trends and Technology (IJMTT), 65(12), 153-165. Retrieved from http://www.ijmttjournal.org/Volume-65/Issue-12/IJMTT-V65I12P517.pdf
- Wafa, M. N., Hussaini, S. A. M., & Pazhman, J. (2020). Evaluation of students’ mathematical ability in Afghanistan’s schools using cognitive diagnosis models. Eurasia Journal of Mathematics, Science and Technology Education, 16(6), em1849. doi: 10.29333/ejmste/7834
- Yamaguchi K., & Okada, K. (2018). Comparison among cognitive diagnostic models for the TIMSS 2007 fourth grade mathematics assessment. PLOS ONE, 13(2), e0188691. doi: 10.1371/journal.pone.0188691
- Yang, X., & Embretson, S. E. (2007). Construct validity and cognitive diagnostic assessment. In J. P. Leighton & M. J. Gierl (Eds.), Cognitive diagnostic assessment for education: Theory and applications (pp. 119-145). Cambridge, UK: Cambridge University Press. Retrieved from https://pdfs.semanticscholar.org/32a5/00670eced66b3023e45a8b0929ad4c1f46e3.pdf
10.7202/1024892ar