The impact of nonverbal information on subtitle translators’ cognitive effort: An eye-tracking and key-logging study - Huang, Jie (Meta)

(untitled) (Les Presses de l’Université de Montréal, 2024)
This study investigates the impact of nonverbal information on the cognitive effort of translators in audiovisual translation (AVT), specifically in subtitling. While the importance of nonverbal elements in AVT is widely recognised, understanding how the amount of nonverbal information directly affects translators’ cognitive effort remains a challenge. To address this, methods from Translation Process Research (TPR) are employed, including eye-tracking, keystroke-logging and subjective reflection. Based on the multimodal transcription method, texts are differentiated according to their level of nonverbal information by analysing key features such as kinesics, camera content and acoustic diegesis. Through the experiment, the cognitive processes and perceived effort of translators are examined throughout the subtitle translation process, including video watching. Results show that an increase in the amount of nonverbal information does not lead to increased cognitive effort among subtitle translators, as evidenced by reduced fixations, keystroke operations and pauses. The perceived effort by translators aligns with this finding. Additionally, the distribution of attention exhibits a consistent pattern, with only minor differences due to varying levels of nonverbal information. Thus, this study highlights the potential to examine the role of nonverbal information in shaping translators’ cognitive processes and confirms the viability of incorporating TPR methods into AVT process research.