Engage
Move
Intone
DEAN
Our interaction research has shown that although augmented speakers communicate at varying rates, delays in message composition consistently create challenges during conversation. These delays can lead to mishearings, misunderstandings, disruptions in shared attention, and breakdowns in mutual understanding.
Many of these difficulties stem from the time required to compose linguistic messages using AAC systems. Communication delays place significant demands on attention and can disrupt the expected timing and flow of everyday interaction. For example, our research has shown that when communication partners speak simultaneously, the relevance and interpretation of an augmented speaker’s contribution can shift or become disrupted.
Our work has also examined how individuals using AAC navigate conversational repair, including both self-initiated repair and repair prompted by others. This research has identified important differences between spoken interaction and AAC-mediated interaction, particularly regarding the effort and time required to clarify misunderstandings and repair communication breakdowns.
Together, these findings provide an empirical foundation for evaluating and guiding our recent AI-AAC design efforts. They also establish measurable interaction benchmarks that can help assess whether emerging communication technologies meaningfully improve conversational participation and understanding.
This line of research has contributed to the development of methods for creating more authentic conversational environments in experimental settings, as well as advanced tracking and interaction analysis techniques that enable faster and more detailed insights from our interaction studies.
Current AAC designs often create barriers to sustained social engagement by concealing the augmented speaker’s message composition process from communication partners. While providing partners with visual indicators of conversational progress—such as the ability to view ongoing message composition—can increase engagement and support conversational flow, it can also introduce new challenges. In particular, partner comments or reactions to partially composed messages may interrupt, delay, or unintentionally redirect the augmented speaker’s intended contribution.
Project Move is based on the idea that enabling quick production of key communicative actions—such as requests, responses, agreements, and disagreements—can significantly reduce message composition delays and support more natural conversation. This approach aligns closely with the goals of DEAN.
The Move interface uses a library of word- and phrase-level conversational contributions, or “moves,” that can be selected individually or combined to create effective utterances in real time. Each selection immediately produces speech output, while additional panels and a QWERTY keyboard support more flexible communication.
In earlier studies, an augmented speaker participated in approximately ten conversations with two communication partners. With training, the participant learned to use the Move interface to communicate fluidly and in real time. Our findings showed that turn-taking times using Move closely resembled those found in typical spoken conversation, while turn transitions using a traditional speech-generating device with an alphabet keyboard were approximately four times longer.
These results suggest that a Move-based vocabulary combined with a touch interface can dramatically reduce conversational delays and support more dynamic interaction.
During the original study, we chose not to use synthesized speech because we could not yet adequately control prosody. Instead, the participant spoke each selected word or phrase aloud while using the interface. Analysis of these interactions revealed rich prosodic patterns that are now helping inform the development of a prosody interface within DEAN.
We are currently integrating the Move interface into DEAN and adapting it specifically for Todd, including a training phase to support his learning of the Move vocabulary system. Todd has expressed strong interest in the interface and has requested a standalone version for use on his PRC device.
Our Move research demonstrated that reducing delays in conversational turn-taking allows spoken intonation to become more functionally integrated into social interaction. As conversational timing becomes more natural, intonation can support important interactional functions such as emphasis, referencing, responsiveness, and conversational coordination.
Although these findings are still preliminary, they highlight a promising direction for improving the pragmatic effectiveness of synthesized speech in AAC systems. Emerging speech technologies, including advanced speech engines such as those developed by Microsoft Azure, may provide new opportunities to pair interactionally meaningful intonation with AAC-generated linguistic output.