Turn-Taking

• Also called: turn management
• Category: Control › Human in the loop
• Tags: human-robot-interaction, spoken-dialogue, speech, conversation, social-robots

The coordination by which participants in a conversation decide who speaks when, including how a speaker yields the floor and how the next speaker knows when to start.

Definition

Turn-taking is the way people in a conversation organize who is speaking and when the floor passes to someone else. Its classic description comes from Sacks, Schegloff and Jefferson (1974), who showed that speakers build turns out of units that have recognizable possible completion points, and that the next speaker usually starts at or near one of these points. Across languages, gaps between turns are typically very short, on the order of a couple of hundred milliseconds, which is less time than it takes to plan an utterance. Listeners must therefore predict when the current turn will end rather than wait for silence.

People use many cues to predict turn ends and turn shifts: whether the utterance is syntactically and pragmatically complete, prosody such as falling pitch and lengthened final syllables, gaze, breathing, and gestures. Speakers can also hold the floor through a pause with filled pauses such as "um" or continuing intonation. Overlaps and interruptions occur and are handled with their own repair practices, and in multi-party conversation gaze and address terms help select who speaks next.

Many spoken dialogue systems and robots still rely on a fixed silence threshold: when the user has been silent for a set time, the system assumes the turn is over. This produces both slow responses and interruptions when users pause mid-sentence. Predictive turn-taking models aim to do better. Some use the words recognized so far to estimate whether a turn is complete, as in text-based models such as TurnGPT, while others, such as voice activity projection models, predict upcoming speech activity of both participants directly from audio. On robots these models must run in real time, cope with noise, the robot's own voice and multiple people, and can be combined with visual cues such as gaze.

Why it matters

Poor turn-taking is one of the most noticeable problems in spoken interaction with robots: long delays make the robot seem slow or unresponsive, and cutting people off makes it seem rude. As robots use large language models for conversation, the timing of when to speak has become as important to the experience as what is said.

Key applications

• Conversational social and service robots
• Spoken dialogue systems and voice assistants
• Multi-party interaction with robots in public spaces
• Human-robot collaboration using speech instructions
• Telepresence and avatar robots

In practice

A team building a receptionist robot replaces a fixed silence threshold with a voice activity projection model that runs on the microphone stream. The model outputs, every few tens of milliseconds, the probability that the user is about to keep talking or that the robot should take the turn. The dialogue manager starts preparing a response as soon as the probability of a shift rises, and only begins speaking once it crosses a threshold. The robot also looks at the user while listening and briefly looks away when it starts speaking. In tests the team logs response delays and the number of times the robot talks over a user, and tunes the threshold to trade speed against interruptions.

Common misconceptions

• A turn ends when the speaker goes silent — people pause within turns and often start the next turn before or right at the end of the previous one, so silence alone is a poor signal
• Turn-taking is solved by a fast speech recognizer — recognition speed helps, but deciding when to speak requires predicting turn ends from content, prosody and other cues
• Overlaps are always errors — brief overlaps and backchannels are normal in human conversation and do not necessarily mean someone has taken the turn

Who should know this

• Conversational robot and dialogue system engineers
• HRI researchers studying spoken interaction
• Speech and audio engineers integrating ASR and TTS on robots

Related technologies

• Voice Activity Detection
• Voice Activity Projection
• Automatic Speech Recognition
• Spoken Dialogue Systems
• Large Language Models

Related terms

• Backchannel
• Transformer
• Foundation Model
• Humanoid Robot

Related projects

• Reachy Mini: Open-source desktop robot from Pollen and Hugging Face with an expressive 6-DoF head
• RAI: Vendor-agnostic agent framework for ROS 2 robots and physical AI
• Assistive Gym: Physics simulation framework for training assistive robots in caregiving tasks

Videos

• Lecture 2.3.3 - Turn Taking: NPTEL-NOC IITM