Abstracts
Abstract
Performance-based assessments (PBAs) are essential tools in veterinary medical education for evaluating students’ clinical and professional competencies. This reflection on assessment is grounded in the first author’s experience as a veterinary educator navigating the complexities of PBAs and in the second author’s experience as a psychometrician studying validity claims about raters’ cognitive processes as they use PBAs. By integrating personal insights with existing literature, this work contributes to the ongoing discourse on assessment design and defensibility in health professional education. Our aim is to highlight the need to explore how rater response processes influence assessment outcomes and to advocate for more robust, contextually relevant validity evidence in veterinary medical education.
While multiple sources of validity evidence are needed to support decisions based on PBA scores, the response processes of raters—how they interpret and apply assessment criteria—remain underexplored in health profession education. Without this type of evidence, the defensibility of assessment scores, as well as the decisions based on them, may be compromised. We argue that understanding rater decision making is crucial for strengthening the validity argument of PBAs.
Keywords:
- Performance-based Assessments,
- Validity Evidence,
- Rater Response Processes,
- Veterinary Education
Appendices
Bibliography
- American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
- Apramian, T., Cristancho, S., Sener, A., & Lingard, L. (2018). How do thresholds of principle and preference influence surgeon assessments of learner performance? Annals of Surgery, 268(2), 385–390. https://doi.org/10.1097/SLA.0000000000002284
- Ark, T. K., Ark, N., & Zumbo, B. D. (2014). Validation practices of the Objective Structured Clinical Examination (OSCE). In B. D. Zumbo & E. K. H. Chan (Eds.), Validity and validation in social, behavioral, and health sciences (pp. 267–288). Springer. https://doi.org/10.1007/978-3-319-07794-916
- Bok, H. G. J., de Jong, L. H., O’Neill, T., Maxey, C., & Hecker, K. G. (2018). Validity evidence for programmatic assessment in competency-based education. Perspectives on Medical Education, 7(6), 362–372. https://doi.org/10.1007/s40037-018-0481-2
- Cook, D. A., & Hatala, R. (2016). Validation of educational assessments: A primer for simulation and beyond. Advances in Simulation, 1, 1–12. https://doi.org/10.1186/s41077-016-0033-y
- Foreman, J. H., Coleman, M., Danielson, J. A., Fogelberg, K., Frost, J., Gates, C., Hinckley-Boltax, A., Hodgson, J. L., Lyon, S., Matthew, S. M., Read, E., & Schoenfeld-Tacher, R. (2024). Competency-based veterinary education: Toolkit. Association of American Veterinary Medical Colleges. https://www.aavmc.org/wp-content/uploads/2025/09/CBVEToolkit2024April.logo25added.pdf
- Gingerich, A., Kogan, J., Yeates, P., Govaerts, M., & Holmboe, E. (2017). Seeing the “black box” differently: Assessing the assessment of professional competence. Medical Education, 48(11), 1055–1068. https://doi.org/10.1111/medu.12546
- Harden, R. M., Stevenson, M., Wilson Downie, W., & Wilson, G. M. (1975). Assessment of clinical competence using objective structured examination. British Medical Journal, 1, 447–451. https://doi.org/10.1136/bmj.1.5955.447
- Hecker, K. G., Adams, C. L., & Coe, J. B. (2012). Assessment of first-year veterinary students’ communication skills using an objective structured clinical examination: The importance of context. Journal of Veterinary Medical Education, 39(3), 304–310. https://doi.org/10.3138/jvme.0312.022R
- Hubley, A. M., & Zumbo, B. D. (2017). Response processes in the context of validity: Setting the stage. In B. D. Zumbo & A. M. Hubley (Eds.), Understanding and investigating response processes in validation research (pp. 1–12). Springer. https://doi.org/10.1007/978-3-319-56129-5_1
- McMillan, C. J., Read, M. R, Hecker, K. G., Skorobohach, B. J., & Banse, H. E. (2022). Building clinical skills teaching materials: A study of cognitive task analysis versus free recall instruction to teach canine indirect ophthalmoscopy. Journal of Veterinary Medical Education, 49(5), 584–593. https://doi.org/10.3138/jvme-2020-0139
- Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749. https://doi.org/10.1037/0003-066X.50.9.741
- Meyer, E. G., Harvey, E., Durning, S. J., & Uijtdehaage, S. (2022). Pre-clerkship EPA assessments: A thematic analysis of rater cognition. BMC Medical Education, 22, 1–11. https://doi.org/10.1186/s12909-022-03402-x
- Padilla, J. L., & Leighton, J. P. (2017). Cognitive interviewing and think aloud methods. In B.D. Zumbo & A. M. Hubley (Eds.), Understanding and investigating response processes in validation research (pp. 211–228). Springer. https://doi.org/10.1007/978-3-319-56129-5_12
- Park, Y. S. (2020). Chapter 3: Reliability. In R. Yudkowsky, Y.S. Park, & S. M. Downing (Eds.), Assessment in health professions education (2nd ed., pp. 33–50). Routledge.
- Read, E. K., & Baillie, S. (2013). Using cognitive task analysis to create a teaching protocol for bovine dystocia. Journal of Veterinary Medical Education, 40(4), 397–402. https://doi.org/10.3138/jvme.0213-033R
- Roberts, M. R., Cook, M., & Chao, I. C. I. (2020). Exploring assessor cognition as a source of score variability in a performance assessment of practice-based competencies. BMC Medical Education, 20, 1–14. https://doi.org/10.1186/s12909-020-02077-6
- Royal, K. D., & Hecker, K. G. (2016). Rater errors in clinical performance assessments. Journal of Veterinary Medical Education, 43(1), 5–8. https://doi.org/10.3138/jvme.0715-112R
- Shankar, S., St-Onge, C., & Young, M. E. (2022). When I say… response process validity evidence, Medical Education, 56(9), 878–880. https://doi.org/10.1111/medu.14853
- Shaver, S. L., & Fogelberg, K. (2023). Assessing clinical skills. In K. Fogelberg (Ed.), Educational principles and practice in veterinary medicine (pp. 313–27). John Wiley & Sons. https://doi.org/10.1002/9781119852865.ch11
- Simons, M. C., Hunt, J. A., & Anderson, S. L. (2022). What’s the evidence? A review of current instruction and assessment in veterinary surgical education. Veterinary surgery, 51(5), 731–743. https://doi.org/10.1111/vsu.13819
- St-Onge, C., Chamberland, M., Lévesque, A., & Varpio, L. (2016). Expectations, observations, and the cognitive processes that bind them: Expert assessment of examinee performance. Advances in Health Sciences Education, 21(3), 627–642. https://doi.org/10.1007/s10459-015-9656-3
- Tekian, A., & Norcini, J. J. (2016). Faculty development in assessment: What the faculty need to know and do. In P. Wimmers & M. Mentkowski (Eds.), Assessing competence in professional performance across disciplines and professions. Springer. https://doi.org/10.1007/978-3-319-30064-1_16