Listenership in human-agent collectives: a study of unidirectional instruction-giving

Ofemile, Abdulmalik Caxton Yusuf (2018) Listenership in human-agent collectives: a study of unidirectional instruction-giving. PhD thesis, University of Nottingham.

[img] PDF (Thesis - as examined) - Repository staff only - Requires a PDF viewer such as GSview, Xpdf or Adobe Acrobat Reader
Available under Licence Creative Commons Attribution.
Download (7MB)


Research in nonverbal listenership behaviour and instruction-giving has focused on interaction with people while paying inadequate attention to human-agent interaction even as recent research indicates that, increasing pervasive computing is significantly changing how humans interact with intelligent software agents and extending the boundaries of discourse to contexts including satellite navigation systems giving directions to drivers, self-checkout machines in supermarkets and intelligent personal assistants on smartphones. This thesis reports studies that use spontaneous listener facial actions and gestures to understand the nature and pattern of spontaneous nonverbal listenership behaviours, identification and communication in instruction-giving contexts. The research methodology used is as follows. Participants who are all L1 speakers of English (forty-eight in Study 1, six in Study 2) were tasked with assembling two Lego models using vague verbal instructions from a computer interface in Study 1 and a human instructor in Study 2 with a 15-minute time limit per iteration. The interface in study1uses three voices of which two are synthesised and one is non-synthesised human recording by a voice actor while Study 2 used a live human voice. A 24-hour long multimodal corpus was built and analysed from interactions between participants and the interface in Study 1 while a 3-hour multimodal corpus was developed from Study 2. The multimodal corpus was annotated for marked facial actions and gestures occurring at points when participants requested that instructors repeat instructions. Participant requests were nonverbal in HAI and a combination of nonverbal and verbal instructions in HHI contexts. The repetitions were quantified and classified into nine typologies. The results reveal key findings regarding the use of spontaneous nonverbal listenership behaviours as pragmatic markers indicating listener comprehension or incomprehension of instructions, perception of instructor-identities, projection of attitudes, meaning-development, task-execution strategies and interaction management even though, the agent could not attend to them in the same way a human can. Using these results, the thesis submits that there are potentials for applied linguistics theories and research to be used to identify and understand pathways to make agents more responsive to human behaviour, make human-agent interaction more credible and provides a theoretical foundation for future multidisciplinary research.

Item Type: Thesis (University of Nottingham only) (PhD)
Supervisors: Adolphs, Svenja
Rodden, Tom
Keywords: Listenership; Human-Agent Interaction; Facial Actions; Gestures; Instruction-Giving;Multimodal Interaction; Assembly Task;Listening Comprehension; Vague Language; Nonverbal Feedback; Third Space of Hybridity; Hybrid space
Subjects: P Language and literature > P Philology. Linguistics
Faculties/Schools: UK Campuses > Faculty of Arts > School of English
Item ID: 49255
Depositing User: Ofemile, Abdulmalik
Date Deposited: 15 Mar 2018 08:57
Last Modified: 05 May 2018 09:00

Actions (Archive Staff Only)

Edit View Edit View