Abstracts
Abstract
Objective – The purpose of this research was to explore user sentiment on Ask a Librarian, a consortial chat service for university libraries in Ontario, Canada, between 2019 to 2021. We tested how the characteristics of the chat (such as year, semester, user type, operator type, affiliation mismatch, and user complaints) and the onset of the COVID-19 pandemic affected sentiment scores.
Methods – The researchers analyzed 3,339 chat transcripts using VADER, a free, open-source Python natural language processing library for sentiment analysis. We tested the significance of relationships between study variables and sentiment score using either a two-samples t-test or ANOVA.
Results – Between 2019 to 2021, overall sentiment on Ask a Librarian was positive and higher among operators than users. There was a significant relationship between sentiment scores and operator type, affiliation mismatch, and complaints respectively. The year, semester, and pandemic status of the chat were also significantly associated with sentiment score. Chats that took place during the COVID-19 pandemic had a significantly higher overall sentiment score than pre-pandemic chats. Average user sentiment score was also higher during the pandemic, but there were no significant differences in average operator sentiment score.
Conclusion – The COVID-19 pandemic had a significant effect on the emotional tone of the overall chat interaction, as well as the sentiment within the user’s messages. Practitioners can replicate our approach to understand user emotions, opinions, attitudes, or appraisals during times of disruption or emergency, as well as for regular service assessment.
Appendices
Bibliography
- Barrett, K., Crichton, C., & Logan, J. (2024). An analysis of user complaints on chat reference during the COVID-19 pandemic: Insights into user priorities. Internet Reference Services Quarterly. Advance online publication. https://doi.org/10.1080/10875301.2024.2379816
- Barrett, K., & Pagotto, S. (2019). Local users, consortial providers: Seeking points of dissatisfaction with a collaborative virtual reference service. Evidence Based Library and Information Practice, 14(4), 2-20. https://doi.org/10.18438/eblip29624
- Barrett, K., & Pagotto, S. (2021). Pay (no) attention to the man behind the curtain: The effects of revealing institutional affiliation in a consortial chat service. Partnership, 16(2), 1-21. https://doi.org/10.21083/partnership.v16i2.6651
- Brousseau, C., Johnson, J., & Thacker, C. (2021). Machine learning based chat analysis. Code4Lib, 50.
- Catalano, A. J., Glasser, S., Caniano, L., Caniano, W., & Paretta, L. (2018). An analysis of academic libraries’ participation in 21st century library trends. Evidence Based Library and Information Practice, 13(3), 4-16. https://doi.org/10.18438/eblip29450
- Chen, X., & Wang, H. (2019). Automated chat transcript analysis using topic modeling for library reference services. Proceedings of the Association for Information Science & Technology, 56(1), 368-371. https://doi.org/10.1002/pra2.31
- Chow, A. S., & Croxton, R. A. (2014). Usability evaluation of academic virtual reference services. College & Research Libraries, 75(3), 309-361. https://doi.org/10.5860/crl13-408
- Cohn, S., & Hyams, R. (2021). Our year of remote reference: COVID19’s impact on reference services and librarians. Internet Reference Services Quarterly, 25(4), 127-144. https://doi.org/10.1080/10875301.2021.1978031
- Connaway, L. S., & Radford, M. L. (2011). Seeking synchronicity: Revelations and recommendations for virtual reference. Online Computer Library Center, Inc. https://www.oclc.org/research/publications/2011/synchronicity.html
- De Groote, S., & Scoulas, J. M. (2021). Impact of COVID-19 on the use of the academic library. Reference Services Review, 49(3/4), 281-301. https://doi.org/10.1108/RSR-07-2021-0043
- Decker, E. N., & Chapman, K. (2022). Launching chat service during the pandemic: Inaugurating a new public service under emergency conditions. Reference Services Review, 50(2), 163-178. https://doi.org/10.1108/RSR-08-2021-0051
- George, L. E., & Birla, L. (2018). A study of topic modeling methods. In 2018 Second International Conference on Intelligent Computing and Control Systems (pp. 109-113). IEEE. https://doi.org/10.1109/ICCONS.2018.8663152
- Hervieux, S. (2021). Is the library open? How the pandemic has changed the provision of virtual reference services. Reference Services Review, 49(3/4), 267-280. https://doi.org/10.1108/RSR-04-2021-0014
- Hutto, C. J. (2014). VADER-Sentiment-Analysis [Repository]. GitHub. https://github.com/cjhutto/vaderSentiment
- Hutto, C. J., & Gilbert, E. (2014). VADER: A parsimonious rule-based model for sentiment analysis of social media text. Proceedings of the International AAAI Conference on Web and Social Media, 8(1), 216-225. https://doi.org/10.1609/icwsm.v8i1.14550
- Kathuria, S. (2021). Library support in times of crisis: An analysis of chat transcripts during COVID-19. Internet Reference Services Quarterly, 25(3), 107-119. https://doi.org/10.1080/10875301.2021.1960669
- Koh, H., & Fienup, M. (2021). Topic modeling as a tool for analyzing library transcripts. Information Technology and Libraries, 40(3). https://doi.org/10.6017/ital.v40i3.13333
- Kohler, E. (2017). What do your library chats say?: How to analyze webchat transcripts for sentiment and topic extraction. In F. Baudino, K. Hart, & C. Johnson (Eds.), 17th Annual Brick & Click: An Academic Library Conference (pp. 138-148). Northwest Missouri State University. https://eric.ed.gov/?id=ED578189
- Kohler, E. (2020). Mining library chats: Sentiment analysis and topic extraction. In 14th International Conference on Performance Measurement in Libraries (pp. 220-229).
- Lamba, M., & Madhusudhan, M. (2022). Text mining for information professionals: An uncharted territory. Springer. https://doi.org/10.1007/978-3-030-85085-2
- Liu, H., Chatterjee, I., Zhou, M., Lu, X. S., & Abusorrah, A. (2020). Aspect-based sentiment analysis: A survey of deep learning methods. IEEE Transactions on Computational Social Systems, 7(6), 1358-1375. https://doi.org/10.1109/TCSS.2020.3033302
- Lund, B. D. (2020). Assessing library topics using sentiment analysis in R: A discussion and code sample. Public Services Quarterly, 16(2), 112-123. https://doi.org/10.1080/15228959.2020.1731402
- Mawhinney, T. (2020). User preferences related to virtual reference services in an academic library. The Journal of Academic Librarianship, 46(1), 102094. https://doi.org/10.1016/j.acalib.2019.102094
- Munip, L., Tinik, L., Borrelli, S., Randone, G. R., & Paik, E. J. (2022). Lessons learned: A meta-synthesis examining library spaces, services and resources during COVID-19. Library Management, 43(1/2), 80-92. https://doi.org/10.1108/LM-08-2021-0070
- Murphy, J. E., Lewis, C. J., McKillop, C. A., & Stoeckle, M. (2022). Expanding digital academic library and archive services at the University of Calgary in response to the COVID-19 pandemic. IFLA Journal, 48(1), 83-98. https://doi.org/10.1177/03400352211023067
- Osorio, N., & Droog, A. (2021). Exploring the impact of the pandemic on reference and research services: A literature review. New Review of Academic Librarianship, 27(3), 280-300. https://doi.org/10.1080/13614533.2021.1990092
- Ozeran, M., & Martin, P. (2019). “Good night, good day, good luck”: Applying topic modeling to chat reference transcripts. Information Technology and Libraries, 38(2), 49-57. https://doi.org/10.6017/ital.v38i2.10921
- Paulus, T. M., Wise, A. F., & Singleton, R. (2019). How will the data be analyzed? Part one: Quantitative approaches including content analysis, statistical modeling, and computational methods. In T. M. Paulus & A. F. Wise (Eds.), Looking for insight, transformation, and learning in online talk (pp. 127-159). Routledge. https://doi.org/10.4324/9781315283258
- Radford, M. L., Costello, L., & Montague, K. (2020). Chat reference in the time of COVID-19: Transforming essential user services. Association for Library and Information Science Education (ALISE) 2020 Conference. http://hdl.handle.net/2142/108820
- Radford, M. L., Costello, L., & Montague, K. E. (2022). “Death of social encounters”: Investigating COVID-19’s initial impact on virtual reference services in academic libraries. Journal of the Association for Information Science and Technology, 73(11), 1594-1607. https://doi.org/10.1002/asi.24698
- Schiller, S. Z. (2016). CHAT for chat: Mediated learning in online chat virtual reference service. Computers in Human Behavior, 65, 651-665. https://doi.org/10.1016/j.chb.2016.06.053
- Sharma, A., Barrett, K., & Stapelfeldt, K. (2022). Natural language processing for virtual reference analysis. Evidence Based Library and Information Practice, 17(1), 78-93. https://doi.org/10.18438/eblip30014
- Sobol, B., Goncalves, A., Vis-Dunbar, M., Lacey, S., Moist, S., Jantzi, L., Gupta, A., Mussell, J., Foster, P. L., & James, K. (2023). Chat transcripts in the context of the COVID-19 pandemic: Analysis of chats from the AskAway consortia. Evidence Based Library and Information Practice, 18(2), 73-92. https://doi.org/10.18438/eblip30291
- Turp, C., & Hervieux, S. (2023). Exploring an automated method for the analysis of virtual reference interactions. Reference Services Review. Advance online publication. https://doi.org/10.1108/RSR-05-2023-0050
- Walker, J., & Coleman, J. (2021). Using machine learning to predict chat difficulty. College & Research Libraries, 82(5), 683-707. https://doi.org/10.5860/crl.82.5.683
- Wang, Y. (2022). Using machine learning and natural language processing to analyze library chat reference transcripts. Information Technology and Libraries, 41(3). https://doi.org/10.6017/ital.v41i3.14967
- Watson, A. P. (2023). Pandemic chat: A comparison of pandemic-era and pre-pandemic online chat questions at the University of Mississippi Libraries. Internet Reference Services Quarterly, 27(1), 25-36. https://doi.org/10.1080/10875301.2022.2117757
- Yatcilla, J. K., & Young, S. (2021). Library responses during the early days of the pandemic: A bibliometric study of the 2020 LIS literature. Journal of Library Administration, 61(8), 964-977. https://doi.org/10.1080/0193082