Abstracts
Abstract
Objective – The Behaviours in Dementia Toolkit is an online library to support dementia-related mood and behaviour changes. It is intended to be used by two key caregiving audiences: 1) formal, professional care partners, and 2) informal care partners. The purpose of this study was to assess the content tone and usability of the website to meet the needs of its two user groups.
Methods – The multi-method design included electronic surveys and interviews. Content tone was assessed using word reaction and highlighter tests. Task-specific usability was assessed with the single-ease question (SEQ), time on task, and completion rate. The System Usability Scale (SUS) measured overall usability before and after design changes based on initial results.
Results – Content tone was friendly, trustworthy, desirable, and clear, but informal care partners found medical jargon confusing. Task-specific usability was generally acceptable. Time one SUS score for formal care partners was assessed at a letter grade A, but for informal care partners was a C. Content tone and usability testing drove critical changes to the library. Time two SUS scores subsequently equalized for the user groups to an A- for formal care partners and a B for informal care partners. Divergent requirements for formal and informal care partners were identified and addressed using pragmatic human-centred design methods.
Conclusion – Jargon, information overload, and search flow require careful attention in an online health information library. Digital libraries and other projects facilitating access to health information seeking groups may find these usability methods useful to consider at the outset of their project planning given the actionable insights they can provide. These techniques are suitable for projects with a limited budget and tight timeframe.
Appendices
Bibliography
- Albrecht, L., Ubels, N., Martinussen, B., Naglie, G., Rapoport, M., Hatch, S., Seitz, D., Checkland, C., & Conn, D. (2025). The Behaviours in Dementia Toolkit: A descriptive study on the reach and early impact of a digital health resource library about dementia-related mood and behaviour changes. Geriatrics (Basel, Switzerland), 10(3), 79. https://doi.org/10.3390/geriatrics10030079
- Allison, R., Hayes, C., McNulty, C., & Young, V. A. (2019). Comprehensive framework to evaluate websites: Literature review and development of GoodWeb. JMIR Formative Research, 3(4). https://doi.org/10.2196/14372
- Barbara, A. M., Dobbins, M., Haynes, R. B., Iorio, A., Lavis, J. N., Raina, P., & Levinson, A. J. (2016). The McMaster Optimal Aging Portal: Usability evaluation of a unique evidence-based health information website. JMIR Human Factors, 3(1), e4800. https://doi.org/10.2196/humanfactors.4800
- Brett, K. R., Lierman, A., & Turner, C. (2016). Lessons learned: A Primo usability study. Information Technology and Libraries, 35(1), 7–25. https://doi.org/10.6017/ital.v35i1.8965
- Brooke, J. (1986). SUS ‘quick and dirty’ usability scale. In P. W. Jordan, B. Thomas, B. A. Weerdmeester, & A. L. McClelland (Eds.), Usability evaluation in industry. Taylor and Francis.
- Burgdorf, J. G., Reckrey, J., & Russell, D. (2023). “Care for me, too”: A novel framework for improved communication and support between dementia caregivers and the home health care team. The Gerontologist, 63(5), 874–886. https://doi.org/10.1093/geront/gnac165
- Canadian Coalition for Seniors’ Mental Health (CCSMH). (2024, March). Canadian clinical practice guidelines for assessing and managing behavioural and psychological symptoms of dementia (BPSD). https://ccsmh.ca/wp-content/uploads/2024/03/CCSMH-BPSD-Clinical-Guidelines_FINAL_March-2024.pdf
- Cerejeira, J., Lagarto, L., & Mukaetova-Ladinska, E. B. (2012). Behavioral and psychological symptoms of dementia. Frontiers in Neurology, 3, 73. https://doi.10.3389/fneur.2012.0007
- Denton, A. H., Moody, D. A., & Bennett, J. C. (2016). Usability testing as a method to refine a health sciences library website. Medical Reference Services Quarterly, 35(1), 1–15. https://doi.org/10.1080/02763869.2016.1117280
- Emde, J. Z., Morris, S. E., & Claassen-Wilson, M. (2009). Testing an academic library website for usability with faculty and graduate students. Evidence Based Library and Information Practice, 4(4), 24–36. https://doi.org/10.18438/B8TK7Q
- Gale, P. (2014, September 2). A simple technique for evaluating content. User Research in Government UK. https://userresearch.blog.gov.uk/2014/09/02/a-simple-technique-for-evaluating-content/
- Hampshire, N., Califano, G., & Spinks, D. (2022). Reaction card method. Mastering collaboration in a product team (pp. 94–95). Apress. https://doi.org/10.1007/978-1-4842-8254-0_47
- Harte, R., Glynn, L., Rodríguez-Molinero, A., Baker, P., Scharf, T., Quinlan, L., & ÓLaighin, G. (2017). A human-centered design methodology to enhance the usability, human factors, and user experience of connected health systems: A three-phase methodology. JMIR Human Factors, 4(1). https://doi.org/10.2196/humanfactors.5443
- Hassan, A. (2020). Challenges and recommendations for the deployment of information and communication technology solutions for informal caregivers: Scoping review. JMIR Aging, 3(2), e20310. https://aging.jmir.org/2020/2/e20310
- Hill, J., Brown, J., Campbell, N., & Holden, R. (2021). Usability-in-place—Remote usability testing methods for homebound older adults: Rapid literature review. JMIR Formative Research, 5(11). https://doi.org/10.2196/26181
- Hyzy, M., Bond, R., Mulvenna, M., Bai, L., Dix, A., Leigh, S., & Hunt, S. (2022). System Usability Scale benchmarking for digital health apps: Meta-analysis. JMIR Mhealth, 10(8). https://mhealth.jmir.org/2022/8/e37290
- Isik, A. T., Soysal, P., Solmi, M., & Veronese, N. (2018). Bidirectional relationship between caregiver burden and neuropsychiatric symptoms in patients with Alzheimer’s disease: A narrative review. International Journal of Geriatric Psychiatry, 34(9), 1326–1334. https://doi.org/10.1002/gps.4965
- ISO. (2018). ISO 9241-11:2018(en), Ergonomics of human-system interaction—Part 11: Usability: Definitions and concepts. https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-2:v1:en
- ISO. (2019). ISO 9241-210:2019(en), Ergonomics of human-system interaction. Part 210: Human-centred design for interactive systems. https://www.iso.org/standard/77520.html
- Jagoda, T., Dharmaratne, S., & Rathnayake, S. (2023). Informal carers’ information needs in managing behavioural and psychological symptoms of people with dementia and related mHealth applications: A systemic integrative review to inform the design of an mHealth application. BMJ Open, 13, e06978. https://doi.org/10.1136/bmjopen-2022-069378
- Jilani, M. A., Sheikh, A., Shah, F., & Saqlain, S. M. (2025). Usability evaluation of academic library websites: A systematic literature review. Information Discovery and Delivery, 54(1), 50–70. https://doi.org/10.1108/IDD-09-2024-0144
- Kales, H. C., Gitlin, L. N., & Lyketsos, C. G. (2015). Assessment and management of behavioural and psychological symptoms of dementia. British Medical Journal, 350, h369. https://doi.org/10.1136/bmj.h369
- Kaplan, K. (2023, February 12). When to use empathy maps: 3 options. NNGroup. https://www.nngroup.com/articles/using-empathy-maps/
- Kous, K., Pušnik, M., Heričko, M., & Polančič, G. (2018). Usability evaluation of a library website with different end user groups. Journal of Librarianship and Information Science, 52(1), 75–90. https://doi.org/10.1177/0961000618773133
- Lewis, J., & Sauro, J. (2018). Item benchmarks for the System Usability Scale. Journal of User Experience, 13(5), 158–167. https://uxpajournal.org/item-benchmarks-system-usability-scale-sus/
- Mutatina, B., Basaza, R., Sewankambo, N. K., & Lavis, J. N. (2019). Evaluating user experiences of a clearing house for health policy and systems. Health Information and Libraries Journal, 36(2), 168–178. https://doi.org/10.1111/hir.12257
- Neilson, C., & Wilson, V. (2011). “We want it now and we want it easy”: Usability testing of an online health library for healthcare practitioners. Journal of the Canadian Health Libraries Association, 32, 51–59. https://doi.org/10.5596/c11-024
- Nielsen, J. (2001, January 20). Usability metrics. NNGroup. https://www.nngroup.com/articles/usability-metrics
- Nimmanterdwong, Z., Boonviriya, S., & Tangkijvanich, P. (2022). Human-centered design of mobile health apps for older adults: Systematic review and narrative synthesis. JMIR Mhealth Uhealth, 10(1). https://doi.org/10.2196/29512
- Oakley-Girvan, I., Davis, S. W., Kurian, A., Rosas, L. G., Daniels, J., Palesh, O. G., Mesia, R. J., Kamal, A. H., Longmire, M., & Divi, V. (2021). Development of a mobile health app (TOGETHERCare) to reduce care partner burden: Product design study. JMIR Formative Research, 5(8). https://doi.org/10.2196/22608
- Okhovati, M., Karami, F., & Khajouei, R. (2017). Exploring the usability of the central library websites of medical sciences universities. Journal of Librarianship and Information Science, 49(3), 246–255. https://doi.org/10.1177/0961000616650932
- Polenick, C. A., Struble, L. M., Stanislawski, B., Broderick, B., Gitlin, L. N., & Kales, H. C. (2018). ‘The filter is kind of broken’: Family caregivers’ attributions about behavioural and psychological symptoms of dementia. The American Journal of Geriatric Psychiatry, 26(5), 548–556. https://doi.org/10.1016/j.jagp.2017.12.004
- Poll, R. (2007). Evaluating the library website: Statistics and quality measures. Quality Issues in Libraries, 74. http://www.ifla.org/iv/ifla73/index.htm
- Poncette, A. K., Mosch, L., Stablo, L., Spies, C., Schieler, M., Weber-Carstens, S., Feufel, M., & Balzer, F. (2021). Human-centered design with usability evaluation of a remote patient monitoring system for intensive care medicine: A mixed-methods study. JMIR Human Factors, 9(10). https://doi.org/10.2196/30655
- Reeder, B., Le, T., Thompson, H. J., & Demiris, G. (2013). Comparing information needs of health care providers and older adults: Findings from a wellness study. Studies in Health Technology and Informatics, 192, 18–22. https://pubmed.ncbi.nlm.nih.gov/23920507
- Rosala, M. (2022, August 17). How to analyze qualitative data from UX research: Thematic analysis. NNGroup. https://www.nngroup.com/articles/thematic-analysis/
- Rutkowski, R. A., Ponnala, S., Younan, L., Weiler, D. T., Bykovskyi, A. G., & Werner, N. E. (2021). A process-based approach to exploring the information behavior of informal caregivers of people living with dementia. International Journal of Medical Informatics, 145, 10431. https://doi.org/10.1016/j.ijmedinf.2020.104341
- Sauro, J. (2010, March 2). If you could only ask one question, use this one. MeasuringU. https://measuringu.com/single-question
- Sauro, J. (2011). Measuring usability with the System Usability Scale (SUS). MeasuringU. https://measuringu.com/sus/
- Schmidt, M., Earnshaw, Y., Tawfik, A. A., & Jahnke, I. (2020). Methods of user centered design and evaluation for learning designers. In M. Schmidt, A. A. Tawfik, I. Jahnke, & Y. Earnshaw (Eds.), Learner and user experience research: An introduction for the field of learning design and technology. EdTech Books.
- Schnelli, A., Hirt, J., & Zeller, A. (2020). Persons with dementia as Internet users: What are their needs? A qualitative study. Journal of Clinical Nursing, 30(5-6), 849–860. https://doi.org/10.1111/jocn.15629
- Silva, A., Caravau, H., Martins, A., Almeida, A., Silva, T., Ribeiro, Ó., Santinha, G., & Rocha, N. (2021). Procedures of user-centered usability assessment for digital solutions: Scoping review of reviews reporting on digital solutions relevant for older adults. JMIR Human Factors, 8(1). https://doi.org/10.2196/22774
- Steinberg, M., Shao, H., Zandi, P., Lyketsos, C. G., Welsh-Bohmer, K. A., Norton, M. C., Breitner, J. C. S., Steffens, D. C., & Tschanz, J. T. (2008). Point and 5-year period prevalence of neuropsychiatric symptoms in dementia: The Cache County study. International Journal of Geriatric Psychiatry, 23(2), 170–177. https://doi.org/10.1002/gps.1858
- Statistics Canada. (2021, December 3). How have Canadians been using the Internet during the COVID-19 pandemic? https://www.statcan.gc.ca/o1/en/plus/198-how-have-canadians-been-using-internet-during-covid-19-pandemic
- Taylor, H. A., Sullivan, D., Mullen, C., & Johnson, C. M. (2011). Implementation of a user-centered framework in the development of a web-based health information database and call center. Journal of Biomedical Informatics, 44(5), 897–908. https://doi.org/10.1016/j.jbi.2011.03.001
- Ubels, N., & Albrecht, L. (2024). Evidence-based principles to accelerate health information flow and uptake among older adults. Evidence Based Library and Information Practice, 19(2), 109–118. https://doi.org/10.18438/eblip30529
- Ubels, N., Tan, L., Albrecht, L., & Long, A. (2024). Developing person-centered metadata: A case study of the Behaviours in Dementia Toolkit. Knowledge Organization, 51(7). https://doi.org/10.5771/0943-7444-2024-7-478
- van den Haak, M., & van Hooijdonk, C. (2010). Evaluating consumer health information websites: The importance of collecting observational, user-driven data. 2010 IEEE International Professional Communication Conference, 333–338. https://doi.org/10.1109/IPCC.2010.5530031
- van Velsen, L., Ludden, G., & Grünloh, C. (2022). The limitations of user-and human-centered design in an eHealth context and how to move beyond them. Journal of Medical Internet Research, 24(10). https://doi.org/10.2196/37341
- Wilson, J., Heinsch, M., Betts, D., Booth, D., & Kay-Lambkin, F. (2021). Barriers and facilitators to the use of e-health by older adults: A scoping review. BMC Public Health, 21(1), 1556. https://doi.org/10.1186/s12889-021-11623-w
- Wittenberg, E., Kerr, A. M., & Goldsmith, J. (2020). Exploring family caregiver communication difficulties and caregiver quality of life and anxiety. American Journal of Hospice and Palliative Medicine, 38(2), 147–153. https://doi.org/10.1177/1049909120935371
- World Health Organization. (2023, March 31). Dementia factsheet. https://www.who.int/news-room/fact-sheets/detail/dementia
- Ye, B., Chu, C. H., Bayat, S., Babineau, J., How, T. V., & Mihailidis, A. (2023). Researched apps used in dementia care for people living with dementia and their informal caregivers: Systematic review on app features, security, and usability. JMIR, 25, e46188. https://doi.org/10.2196/46188

