Usability Testing of an Academic Library Website: A Study of Student Navigation Behaviours With AI-Assisted Data Analysis

Wegener, Debby R.; Chew, Yu Ying

doi:https://doi.org/10.18438/eblip30836

Browse the articles in this issue

The Role of Research in Undergraduate Studio Practice: A Qualitative Study Illustrating How Creative Research Aligns With Queer Theory and Indigenous Ways of Knowing

Article outline

Back to top
Abstract
Bibliography

Toolbox

PDFDownload
Cite this article
Share

Abstracts

Abstract

Objective – This study investigated the search strategies and navigation behaviours of students on an academic library website, with the focus being on identifying usability challenges and opportunities for improvement. The researchers also wanted to test the effectiveness of generative artificial intelligence (GenAI) in analysing qualitative usability data.

Methods – A mixed-methods usability study was conducted with 21 second-year university students. Participants completed three task scenarios on the library website while their screen activity and think-aloud comments were recorded. Quantitative data was collected using a modified usability questionnaire, and the qualitative data was analyzed using multiple GenAI platforms following established prompt frameworks, with human review for accuracy.

Results – Students encountered significant barriers in navigation, layout, content accessibility, and functionality. Many preferred external tools like Google over the library website, citing complexity and difficulty in locating resources. The average usability questionnaire score was 50 out of 100, with higher scores among students who had attended library training. GenAI tools expedited qualitative analysis but required careful prompt engineering and human oversight to ensure accuracy and context relevance.

Conclusion – The findings highlight the need for ongoing, user-centered design improvements to library websites, emphasizing clear navigation, accessible content, and reduced complexity. Regular usability testing and integration of user feedback are recommended to enhance digital library services. In addition, while GenAI can accelerate qualitative analysis, human expertise currently remains essential for nuanced interpretation and validation.

Appendices

Bibliography

Alhadreti, O. (2021). Comparing two methods of usability testing in Saudi Arabia: Concurrent think-aloud vs. co-discovery. International Journal of Human–Computer Interaction, 37(2), 118–130. https://doi.org/10.1080/10447318.2020.1809152
10.1080/10447318.2020.1809152 Google Scholar
Allard, E., & Mucz, D. (2023). RGO library & learning commons usability testing October 2022: Findings and recommendations (pp. 1–23). http://hdl.handle.net/1880/115671
Google Scholar
Ashiq, M., Jabeen, F., & Mahmood, K. (2022). Transformation of libraries during Covid-19 pandemic: A systematic review. Journal of Academic Librarianship, 48(4), 1–10. https://doi.org/10.1016/j.acalib.2022.102534
10.1016/j.acalib.2022.102534 Google Scholar
Bangor, A., Kortum, P., & Miller, J. (2009). Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of Usability Studies, 4, 114–123. https://dl.acm.org/doi/10.5555/2835587.2835589
Google Scholar
Breland, M., Venturella, K., & Shapiro, S. (2023). Looking back, looking ahead: Lessons learned from the pandemic (are we prepared for the next pandemic)? Library Leadership & Management, 36(4), 1–16. https://doi.org/10.5860/llm.v37i2.7557
10.5860/llm.v37i2.7557 Google Scholar
British Standards Institution. (2019). Ergonomics of human-system interaction. Part 210: Human-centred design for interactive systems (Standard No. BS EN ISO 9241-201:2019).
Google Scholar
Brooke, J. (2013). SUS: A retrospective. Journal of Usability Studies, 8(2), 29–40. https://uxpajournal.org/wp-content/uploads/sites/7/pdf/JUS_Brooke_February_2013.pdf
Google Scholar
Chen, Y.-H., Germain, C. A., & Rorissa, A. (2023). More than a decade later: Library web usability practices at ARL academic libraries in 2007 and 2020. College and Research Libraries, 84(2), 203–227. https://doi.org/10.5860/crl.84.2.203
10.5860/crl.84.2.203 Google Scholar
Cirelli, M. C., & Long, B. A. (2020). Doing away with “database:” Exploring medical library website design. Journal of Electronic Resources in Medical Libraries, 17(3–4), 77–91. https://doi.org/10.1080/15424065.2020.1823291
10.1080/15424065.2020.1823291 Google Scholar
Dahrén, K., Meier, S., & Wahlgren, I. (2021). Benefits of remote usability testing: Our experiences [PDF]. https://pub.epsilon.slu.se/27101/1/Dahr_n_Meier_Wahlgren_LibPMC_2021.pdf
Google Scholar
Francis, F. (2023). Usage of e-resources in academic libraries. Journal of Online Engineering Education, 14(2), 13–36. https://onlineengineeringeducation.com/index.php/joee/article/view/95
Google Scholar
Haries, L., Kusumo, D. S., & Sardi, I. L. (2022, November 22-23). Analysis and redesign web navigation Telkom University using usability testing and trunk test [Paper Presentation]. 2022 1st International Conference on Software Engineering and Information Technology (ICoSEIT), Bandung, Indonesia. https://doi.org/10.1109/ICoSEIT55604.2022.10029960
10.1109/ICoSEIT55604.2022.10029960 Google Scholar
Hitch, D. (2024). Artificial intelligence augmented qualitative analysis: The way of the future? Qualitative Health Research, 34(7), 595–606. https://doi.org/10.1177/10497323231217392
10.1177/10497323231217392 Google Scholar
Jabbour, J., Dhillon, H. M., Shepherd, H. L., Puma, S., Milross, C., & Clark, J. R. (2022). A web-based comprehensive head and neck cancer patient education and support needs program: Usability testing. Health Informatics Journal, 28(1), 1–15. https://doi.org/10.1177/14604582221087128
10.1177/14604582221087128 Google Scholar
Jamal, A., Tharkar, S., Alenazi, H., Julaidan, B. S., Al Hindawi, D. A., AlAkeel, N. S., AlNuhayer, O. M., & AlDubaikhi, R. H. (2021). Usability analysis of a health sciences digital library by medical residents: Cross-sectional survey. JMIR Formative Research, 5(6), 1–10. https://doi.org/10.2196/23293
10.2196/23293 Google Scholar
Kahili-Heede, M. (2020). Health Sciences library website using LibGuides: A usability study. http://hdl.handle.net/10125/67802
Google Scholar
Klug, B., Amen, B., Brown, K., Jones, C., & Sanchez Duarte, J. (2023). The best RX is better UX: Redesigning, migrating, and usability testing a hospital library website. Journal of Hospital Librarianship, 23(2), 58–82. https://doi.org/10.1080/15323269.2023.2194801
10.1080/15323269.2023.2194801 Google Scholar
Kous, K., Pušnik, M., Heričko, M., & Polančič, G. (2020). Usability evaluation of a library website with different end user groups. Journal of Librarianship and Information Science, 52(1), 75–90. https://doi.org/10.1177/0961000618773133
10.1177/0961000618773133 Google Scholar
LeLaurin, J. H., Sypniewski, C., Wing, K., Freytes, I. M., Findley, K., & Uphold, C. R. (2022). Development, usability testing, and promotion of the English- and Spanish-language RESCUE stroke caregiver websites. American Journal of Health Education, 53(3), 174–185. https://doi.org/10.1080/19325037.2022.2048747
10.1080/19325037.2022.2048747 Google Scholar
Lo, L. S. (2023). The CLEAR path: A framework for enhancing information literacy through prompt engineering. The Journal of Academic Librarianship, 49(4), 1–3. https://doi.org/10.1016/j.acalib.2023.102720
10.1016/j.acalib.2023.102720 Google Scholar
Matsubara, S. (n.d.). Learning about online geometry resources: A usability study on website for McKinley High School teachers. Geometry Resources. https://sites.google.com/mckinleyhs.k12.hi.us/geometryresources/home
Google Scholar
Mnzava, E. E., & Katabalwa, A. S. (2022). Library websites during the COVID-19 pandemic. Library Philosophy and Practice, 31(1-3), 12-22. https://doi.org/10.1177/09557490211058869
10.1177/09557490211058869 Google Scholar
Moody, D., & Best, R. D. (2021). Covid-19 impacts on electronic resource usage at a regional university. The Southeastern Librarian, 69(3), 1–11. https://doi.org/10.62915/0038-3686.1948
10.62915/0038-3686.1948 Google Scholar
Najafgholinejad, A. (2023). User experience and information architecture of the National Digital Library and archives of Iran: A usability investigation and card sorting. The Electronic Library, 41(1), 30–44. https://doi.org/10.1108/EL-07-2022-0172
10.1108/EL-07-2022-0172 Google Scholar
Nesbitt, K., Beleigoli, A., Champion, S., Gebremichael, L. G., Bulamu, N., Du, H., Tirimacco, R., & Clark, R. A. (2023). Development and evaluation of a co-designed website for delivering interactive self-directed cardiac rehabilitation. European Journal of Cardiovascular Nursing, 23(1), 1–9. https://doi.org/10.1093/eurjcn/zvad026
10.1093/eurjcn/zvad026 Google Scholar
Nielsen, J. (2012). Usability 101: Introduction to usability. NN/G Nielsen Norman Group. https://www.nngroup.com/articles/usability-101-introduction-to-usability/
Google Scholar
Nzomo, P. (2023). The impact of COVID-19 on the use of library resources and services by international students: A case study. Journal of Library Administration, 63(5), 722–740. https://doi.org/10.1080/01930826.2023.2219602
10.1080/01930826.2023.2219602 Google Scholar
Ochoa, I. V. (2020). Navigation design and library terminology: Findings from a user-centered usability study on a library website. Information Technology and Libraries, 39(4), 1–15. https://doi.org/10.6017/ital.v39i4.12123
10.6017/ital.v39i4.12123 Google Scholar
Pattyn, F. (2023). Generative AI for qualitative research: A paradigm shift towards cost-effectiveness (4x), Throughput time (15x) and Reliability (Cohen’s Kappa). Advance online publication. https://doi.org/10.13140/RG.2.2.11414.52800
Google Scholar
Perkins, M., & Roe, J. (2024). The use of generative AI in qualitative analysis: Inductive thematic analysis with ChatGPT. Journal of Applied Learning and Teaching, 7(1), 390–395. https://researchonline.jcu.edu.au/86932/
Google Scholar
Quamar, Q.A. (2016, September 9). 4 pillars of design - Part 1. To The New. https://www.tothenew.com/blog/4-pillars-of-design-part-1/
Google Scholar
Ratnawati, S., Widianingsih, L., Anggraini, N., Shofi, I.M., Hakiem, N., & Agustin, F.E.M. (2020, October 23-24). Evaluation of digital library's usability using the system usability scale method of (a case study) [Paper presentation]. 8th International Conference on Cyber and IT Service Management (CITSM), Pangkal, Indonesia. https://doi.org/10.1109/CITSM50537.2020.9268801
10.1109/CITSM50537.2020.9268801 Google Scholar
Resau, K. (2021). Using Zoom for remote usability testing. Computers in Libraries, 41(1), 22–26.
Google Scholar
Sabzalieva, E., & Valentini, A. (2023). ChatGPT and artificial intelligence in higher education: Quick start guide. UNESDOC Digital Library. https://unesdoc.unesco.org/ark:/48223/pf0000385146
Google Scholar
Salah, M., Al Halbusi, H., & Abdelfattah, F. (2023). May the force of text data analysis be with you: Unleashing the power of generative AI for social psychology research. Computers in Human Behavior: Artificial Humans, 1(2), 1–6. https://doi.org/10.1016/J.CHBAH.2023.100006
10.1016/j.chbah.2023.100006 Google Scholar
Steeb, T., Brutting, J., Reinhardt, L., Hoffmann, J., Weiler, N., Heppt, M. V., Erdmann, M., Doppler, A., Weber, C., Schadendorf, D., Meier, F., & Berking, C. (2023). One website to gather them all: Usability testing of the new German SKin cancer INFOrmation (SKINFO) website–A mixed-methods approach. Journal of Cancer Education, 38, 1264–1270. https://doi.org/10.1007/s13187-022-02258-5
10.1007/s13187-022-02258-5 Google Scholar
Sulistiyawan, A. H., Latifa, Q., Adeliani, Z., & Rahayu, P. (2023, September 14-15). Usability evaluation on Universitas Mercu Buana Jakarta library website using SUS [Paper presentation]. 6th International Conference of Computer and Informatics Engineering (IC2IE), Lombok, Indonesia. https://doi.org/10.1109/IC2IE60547.2023.10331150
10.1109/IC2IE60547.2023.10331150 Google Scholar
Tsekea, S. & Chigwada, J. P. (2021). COVID-19: Strategies for positioning the university library in support of e-learning. Digital Library Perspectives, 37(1), 54-64. https://doi.org/10.1108/DLP-06-2020-0058
10.1108/DLP-06-2020-0058 Google Scholar
Tutko, M. (2024, September 16). The 4 pillars of good design or FRUA — a framework for creating products that effectively meet user needs & expectations. Medium. https://medium.com/@martintutko/the-four-pillars-of-good-design-or-frua-a-framework-for-creating-products-that-effectively-meet-cdbef14b768d
Google Scholar
Upadhyay, N., Chakraborty, K., Upadhyay, S. R., & Pandey, B. N. (2022, November 17-19). Usage pattern of e-resources during COVID-19 and post COVID-19: A study [Paper presentation]. 13th International CALIBER-2022, Banaras Hindu University (BHU), Varanasi, India. http://ir.inflibnet.ac.in/handle/1944/2391
Google Scholar
Vindrola-Padros, C., & Johnson, G. A. (2020). Rapid techniques in qualitative research: A critical review of the literature. Qualitative Health Research, 30(10), 1596-1604. https://doi.org/10.1177/1049732320921835
10.1177/1049732320921835 Google Scholar
Wang, W., Choi, D., & Yu, C. H. (2023). Effective web‑based clinical practice guidelines resources: recommendations from a mixed methods usability study. BMC Primary Care, 24(1), 29. https://doi.org/10.1186/s12875-023-01974-1
10.1186/s12875-023-01974-1 Google Scholar
Yan, L., Echeverria, V., Fernandez-Nieto, G. M., Jin, Y., Swiecki, Z., Zhao, L., Gašević, D., & Martinez-Maldonado, R. (2024). Human-AI collaboration in thematic analysis using ChatGPT: A user study and design recommendations. CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, 1–7. https://doi.org/10.1145/3613905.3650732
10.1145/3613905.3650732 Google Scholar
Zhang, H., Wu, C., Xie, J., Kim, C., & Carroll, J. M. (2023a). QualiGPT: GPT as an easy-to-use tool for qualitative coding. ArXiv. http://arxiv.org/abs/2310.07061
Google Scholar
Zhang, H., Wu, C., Xie, J., Lyu, Y., Cai, J., & Carroll, J. M. (2023b). Redefining qualitative analysis in the AI Era: Utilizing ChatGPT for efficient thematic analysis. ArXiv. http://arxiv.org/abs/2309.10771
Google Scholar

Abstracts

Abstract

Appendices

Bibliography

Citation Tools

Cite this article

Export the record for this article