State-of-the-art dialogue systems focus on two-person dialogue and explicit indication of who is speaking. However, for using LLM-based AI systems in multi-speaker environments, it is necessary to determine who is speaking next. In this work, we investigate how the task can be formulated and how multi-modal LLMs are able to perform it. First, we frame the task as multi-modal Next Speaker Prediction (mNSP) and create a multi-modal, multi-speaker NSP dataset from multiple existing dialogue datasets with different task setups. Second, we build dedicated systems for the mNSP task using multi-modal LLMs, and finally perform automatic evaluation, comparing them to rule-based baselines.

Discrete motion tokenizers encode motion as atomic units and are widely used for co-speech gesture generation. It remains unclear which motion properties, especially those relevant to gesture semantics, are recoverable from these codebooks. We probe a reconstruction-trained codebook using 19 co-speech gesture descriptors spanning from raw motion to abstract communicative function. Results show that geometry and handedness are readily decodable from token embeddings, while motion category is only weakly decoded despite showing systematic differences in discrete code usage. This gap between reconstruction quality and descriptor decodability suggests that reconstruction objectives alone do not guarantee that gesture semantics are captured, and that evaluating codebooks on such properties can guide the design of more semantic motion tokenizers.

Large-scale multimodal communication research increasingly depends on computational methods capable of extracting reliable gesture annotations from video data. Yet it remains unclear to what extent automatically generated annotations can support the linguistic investigation of naturally occurring co-speech gesture and where expert interpretation remains indispensable. This paper presents a framework for the automatic analysis and annotation of co-speech gesture in media corpora, developed to support multimodal communication research at scale. We build our framework on top of the existing computer vision frameworks for keypoint extraction and active speaker detection, with post-processing steps carefully designed to mitigate noisy output. Building on the processed keypoint trajectories, we implement a configurable rule-based annotation system developed through close collaboration between computer vision researchers and linguists. We demonstrate the usefulness of the framework through a case study investigating vertical hand movement accompanying English expressions of future and past, examining which aspects of this gestural pattern can be captured through automatic annotation and which require expert multimodal interpretation. We find that at corpus scale, greater mean amplitude and higher mean peak velocity converge in will-future, suggesting greater kinematic prominence in the multimodal expression of futurity. More broadly, the case study demonstrates how computational pipelines can support, rather than replace, human analysis by enabling multimodal hypotheses to be investigated on datasets that would be impractical to analyse manually.

Engagement detection in meetings has focused mostly on speakers, who produce explicit prosodic and acoustic cues. The engagement of non-speaking participants is harder to read because it surfaces only in nonverbal behaviour. We predict how engaged a silent listener is, on a three-level scale (Engaged, Intermittent, Disengaged), in five-second windows of remote Zoom meetings from the GEstures and Head Movements in Language (GEHM) corpus. Each window is described by three feature families: the listener’s own nonverbal behaviour, the listener’s behavioural alignment with the others present, including the active speaker, and that speaker’s prosody. Interpersonal coordination is an established correlate of rapport and involvement. We ask the narrower question of whether a compact window-level alignment measure adds predictive value over listener-only features when confident labels are scarce. Under leakage-free cross-validation, listener-speaker alignment is the most useful addition we test, and the speaker’s terminal pitch slope is an interpretable correlate of listener engagement. The gains are modest and their bootstrap intervals include zero. Unsupervised analysis finds no engagement structure. Two findings are firmer: the task is effectively binary, actively engaged versus not, bounded by the scarcity of confident disengaged labels, and the pitch-slope correlate holds despite the small sample.

We investigate the alignment between humans and multimodal large language models (MLLMs) when judging interpersonal rapport –a feeling of ease and harmony in conversation– in naturalistic dyadic interactions. Based on a peer-tutoring corpus with crowd-sourced thin-slice rapport annotations, we compare human and model judgments across a range of open- and closed-source models under two complementary framings: Likert-scales and pairwise comparison of segments. Beyond measuring agreement, we analyze the models’ reasoning traces to examine what they attend to. We find a moderate model–human agreement on absolute ratings, comparable to that among human annotators and correlated with the degree of inter-annotator agreement. However, this surface-level agreement hides systematic limitations, as models show a leniency bias and a gender disparity in agreement, rely on a narrow set of mostly nonverbal cues, and over-interpret isolated signals while overlooking the reciprocity between interlocutors that rapport theory emphasizes. Under pairwise comparison, agreement rises for both people and models, with annotators showing higher agreement than in absolute rating and models reaching near-perfect alignment with human pairwise judgments on high-contrast pairs. Our findings support pairwise comparison as a more reliable approach for the computational modeling of rapport, but also lead to questions about perfect inter-rater alignment as a goal, and about current MLLMs as judges. Traces of model reasoning further illuminate rapport theory, as well as the biases of current MLLMs.

Reading a social situation often depends on behavior, not words alone. We introduce FriendBench, a benchmark for inferring whether two people are already familiar or are meeting as strangers, from a 20-second clip of a dyadic ice-breaker conversation. Every pair answers the same type of prompt, so only the manner of interaction can reveal the answer. Across text, audio, and video, we compare 26 models from seven companies against matched human panels over 96 balanced dyads. The best model and the human crowd are statistically indistinguishable on accuracy in every modality, but reach it differently: humans stay balanced across the two answers, while the strongest models favor “stranger.” This is a difference in effective prior, not in discrimination. Richer channels help both unequally, and only humans gain from visible behavior on top of speech. We release the stimuli, human ratings, and model predictions.

In collaborative tasks with asymmetric information, participants coordinate their understanding through interaction. We ask whether gaze provides evidence about grounding across two such tasks. Working from discrete behavioral annotations, we map HCRC MapTask (Anderson et al., 1991) and MUNDEX (Türk et al., 2023) into a shared partner/task/away vocabulary and compute gaze features around task-relevant dialogue units. In both corpora, aligned reference interpretations (MapTask) and UND (understood) judgments (MUNDEX) are associated with more task-directed gaze and with less partner-directed gaze, lower gaze entropy, and fewer gaze transitions. The associations are clearest for the participant leading the task: in giver-produced references, and in explainer judgments, which also co-vary with the explainee’s gaze. In same-speaker MapTask reference chains, the speaker’s gaze entropy is lower at the mention where a previously non-aligned referent becomes aligned. The best gaze feature groups improve modestly over controls under grouped cross-validation: temporal features in MapTask and raw proportions in MUNDEX. Because effects are small and several weaken when recurring participants rather than dialogues are the unit of inference, we treat gaze as one contributing cue to grounding, to be interpreted alongside task and dialogue context.

Robots that search for objects on their own must decide what to do when the search fails, and how they signal that failure to a human partner can determine whether the interaction turns into collaboration. Prior work on human–robot interaction has studied nonverbal help-seeking and social gaze separately, but rarely compares gaze against a matched verbal failure signal inside a live, real-time embodied search task. We present an embodied dialogue system, built on the Retico incremental processing framework, where the Cozmo robot searches for a specified object and signals failure either verbally or through gaze towards the human partner. We first compare a rule-based planner against a vision-language model (VLM) planner that reasons over scene semantics, finding that the VLM planner succeeds far more often (87.2% vs. 65.9%). We then test whether gaze can close this gap without adding reasoning capability to the rule-based planner. Adding gaze as the stuck signal raises success to 87.87%, matching the VLM planner, while a verbal stuck signal only raises it slightly (68.75%). Results further show that this effect is largely implicit: participants helped more in the gaze condition without consistently reporting that they felt invited to do so. Our findings suggest that gaze can substitute for reasoning capability a robot lacks, helping it elicit assistance from a human partner.

Language technology resources are limited for signed languages. In order to improve these technologies, large amounts of labeled language data stored in annotated corpora are necessary. Many existing annotated corpora in American Sign Language (ASL) do not include non-manual markers (NMMs), i.e., language features that do not involve the hands. This work underscores the importance of including NMMs in sign language research by describing their plurifunctional nature in signed conversation, specifying their link to sign language prosody, examining the current state of ASL corpus research, and recommending a systematized method of annotation which captures the critical but complex variations in NMM production in ASL.

Storytelling in face-to-face dialogue involves coordinating multiple modalities, including speech and gaze. When the addressee is a social robot rather than a human, narrators’ speech and gaze behavior may differ. However, few studies have examined both modalities within the same narratives, and it remains unclear how different aspects of multimodal behavior vary depending on whether the interaction partner is a social robot or a human. In this study, we examined how adults narrated picture-based stories to a social robot and a human interlocutor. Two modalities of narrative production were considered: pausing, separated into filled and unfilled pauses, and gaze behavior during the narration, directed either toward the story pictures or toward the listener. Critically, we examined whether shyness, considered as a continuous dimension of individual variation, relates to pausing and gaze behavior. Results revealed that speakers produced fewer filled pauses when narrating to the robot than to the human, while unfilled pauses did not differ. Additionally, higher shyness was associated with longer, though not more frequent, filled pauses in human-directed narration, although shyness did not reliably predict the magnitude of the partner effect. We discuss how our findings may contribute to a more nuanced understanding of multimodal interactions with embodied agents.

Turn-taking is a fundamental component of spoken interaction, and while humans naturally rely on both verbal and non-verbal signals, dialogue systems usually depend on audio cues alone. This paper investigates whether visual features from face-to-face conversations can enhance turn-taking prediction beyond what is achievable from audio-only. We extend the Voice Activity Projection (VAP) model, a self-supervised transformer-based model for predicting future voice activity, by incorporating visual features extracted from the large-scale Meta Seamless Interaction dataset of dyadic face-to-face conversations. The visual features include gaze direction, head movement, body and hand pose, and facial action units (FAU). For incorporating the visual features, we explore concatenation, cross-attention fusion, delta features, and trainable gating mechanisms. Results show that visual information improves performance over the audio-only baseline, with FAU being significantly more informative than other feature groups. Body and gaze features nevertheless contribute complementary information, as the model combining all features performs best. Furthermore, results indicate that performance on specific tasks varies depending on whether training and test data come from improvised (acted) or naturalistic (non-acted) conversations.

Whispering differs from voiced speech in the lack of fundamental frequency (F0) and is characterized by substantially reduced amplitude. This production study investigates how head movements are modulated in whispering when speakers are visible or not, while producing polar questions with rising intonation versus statements with flat intonation. An experiment with 15 German speakers shows that head movements are more pronounced in questions than in statements, with the strongest effects observed in normally spoken questions under mutual visibility. Overall, the findings suggest head movements serve as a flexible multimodal prosodic resource, aligning with acoustic cues in normal speech under favorable communicative conditions while functioning more independently as compensatory mechanism in whispered speech.