Project Converse is dedicated to improving face-to-face expressive communication for individuals with complex communication needs (CCN) who use augmentative and alternative communication (AAC). Although AAC technologies have transformed access to communication, many individuals still encounter significant barriers when participating in one of the most fundamental human experiences: conversation.
Our mission is to challenge outdated assumptions about AAC-mediated communication and address the institutional barriers that have limited innovation in the field. By building a highly interdisciplinary research center, we aim to advance the study, design, and development of conversational AAC technologies that better support real-world human interaction.
Using a research-informed, interaction-centered approach, we will design and develop technologies that support children learning AAC and help bridge the communication gap experienced by adults who rely on AAC. Our work is grounded in the belief that effective communication technologies must be shaped by the lived experiences of AAC users and informed by evidence from social interaction research.
Project Converse brings together researchers, engineers, designers, artists, clinicians, and AAC users to rethink how AAC can support dynamic, in-person conversation. Our work focuses on fostering social engagement, mutual understanding, and personal empowerment through a range of interconnected initiatives, including:
Core Values
Timeline
Project Converse: Five-Year Progress Summary
Project Converse began as a high-risk, interdisciplinary effort to reinvent AAC for face-to-face conversation. From the outset, the project was organized as a sandbox: our goals were guided by a clear long-term mission, but the specific research and development pathways were allowed to emerge from the work itself. That mission was to develop technical approaches that help augmented speakers participate more fully in real-time conversation. Each year’s accomplishments reshaped the next year’s priorities, allowing the project to move from broad exploration, to focused research strands, to integrated AI-AAC probe development.
Year 2
Year 3
Year 4
Year 5
Summary
During the first year, much of the work focused on building the technical, personnel, and conceptual infrastructure needed to support this kind of research. We acquired and prepared laboratory space, purchased equipment and materials, developed the project website, wrote IRB protocols, hired and trained staff, and initiated transcription and video-analysis workflows. This period also marked the formation of the core project team, including research staff and collaborators who would later lead work in interaction analysis, interface design, clinical research, speech synthesis, engineering, and conversational AI. At the same time, we began several exploratory lines of inquiry, including pragmatic interface design, eye-tracking methods, AAC engagement, speech output, interaction analysis, and early thinking about conversational AI. These early strands were not yet tightly integrated, but they shared a common concern: current AAC technologies do not adequately support the timing, visibility, partner coordination, and social action demands of face-to-face conversation.
By the second year, these exploratory efforts had begun to take shape as distinct but connected research and development projects. Enchrony focused on identifying interaction problems, strategies, and design-relevant solutions in AAC-mediated conversation, with early emphasis on composition delay, repair, and grounded-unit analysis. This work was supported by the growing ELAN transcription database and by literature reviews that connected conversation analysis, AAC design, and pragmatic action. Move, initially developed as the Pragmatically Effective Phrase project, translated social-interaction research into an AAC interface built around pragmatically powerful words and phrases that could help users exert greater conversational control. Engage examined how AAC interfaces support shared attention and co-construction, while also advancing OS-DPI logging and analysis tools. Intone emerged from the recognition that conversationally appropriate synthetic speech would be essential for short, pragmatic AAC utterances. DEAN, then developing out of Project Agency, began as an effort to test conversational AI agents that could generate contextually responsive utterance options for augmented speakers.
Year 2 culminated in the 2023 Project Converse Workshop held in Buffalo, New York, which brought together an international, interdisciplinary group of researchers, clinicians, educators, designers, and technology developers to examine the future of AAC and communication technologies. Through presentations, demonstrations, discussions, and collaborative sessions, participants explored how emerging technologies can better support communication access, language learning, partner engagement, social participation, autonomy, and agency for individuals with complex communication needs. The workshop created a forum for sharing perspectives across communication sciences, human-computer interaction, education, accessibility research, artificial intelligence, and user-centered design, while identifying shared priorities for future research and innovation. A central outcome was the recognition that AAC technologies must be designed not only to support effective message production, but also to promote meaningful participation, quality of life, and full engagement in everyday social interaction.
Year 3 marked the midpoint of the project and a transition from capacity-building toward coordinated research and prototype development. Large-scale transcription began winding down as the team shifted toward accuracy checking and multi-participant analysis. By this point, we had developed a large video database for future research use, including more than 150 transcribed videos from 60 dyads involving adults with ALS, primary lateral sclerosis, cerebral palsy, and head injury. Grounded-unit analysis became a central method for segmenting AAC-mediated conversation into comparable stretches organized around the augmented speaker’s composition activity and the surrounding actions of both participants. The team also developed a continuous time-sampling technique that substantially reduced coding time for focused questions about interactional performance.
Enchrony provided the empirical and analytic foundation for this phase. Studies of composition delay, concurrent partner talk, repair, and intersubjectivity clarified how AAC-mediated interaction becomes vulnerable when message construction takes longer than the conversational moment allows. This work showed that delay is not simply slow message production; it can alter the sequential placement, intelligibility, and social uptake of an augmented speaker’s contribution. Repair studies further demonstrated that timely repair initiation was largely unavailable to many augmented speakers except through single selections, while graduate research projects provided empirical data on both other-initiated and self-initiated repair. These findings began to function as performance benchmarks for the design of future AAC systems.
At the same time, Move, Engage, and Intone matured as design-oriented research projects. Move advanced from paper-based interface design toward a digital OS-DPI implementation, supported by external modules, device-training resources, interaction-task catalogs, social-interaction training protocols, expressive speech work, and video analyses used to refine interface design. Engage tested interface configurations designed to support mutual gaze, shared attention, and partner access to message construction. Pilot results suggested that visual access to the AAC display supported more orderly and engaged interaction, leading to a larger study comparing multiple communication-mode conditions. Intone developed the expressive speech strand through collaboration with Eva Szekely at KTH Royal Institute of Technology, including recording and testing the Kaylie voice model, evaluating its feasibility through OS-DPI.
Year 3 also marked a major step in the emergence of DEAN as the project’s future integration point. The team installed a server for AI development and built a raspberry pi–based hardware/software architecture with cellular connectivity to manage input and output among the augmented speaker, partner, cloud-based ASR, language-model components, OS-DPI, and TTS. Todd Hutchinson continued as the first prototype evaluator and augmented speaker, and his 60-chapter memoir was used to train and personalize the language model. Initial LLM testing demonstrated both promise and limitation: the system could generate candidate responses, but a trained model alone was insufficient for producing relevant, contextualized, speaker-aligned utterances in conversation. This finding helped redirect the project toward a more layered account of AI-AAC development, in which infrastructure, language generation, interface design, timing, and social fit would all need to be developed together.
Year 4 became the project’s major integration year. Enchrony continued to provide the empirical foundation by documenting how composition delays create recurring problems in AAC-mediated conversation, including mishearings, misunderstandings, breakdowns in shared attention, and shifts in the sequential relevance of augmented-speaker contributions. Antara Satchidanand completed and defended her dissertation on other-initiated repair in SGD-mediated interaction, identifying significant asymmetries in repair initiation and repair types compared with typical spoken conversation. Gabrielle Martino completed and defended her master’s thesis on self-editing with SGDs, showing that users rely heavily on less efficient character deletion options during naturalistic interaction, even though simulation pointed to word deletion as more efficient. Work by Koroschetz, Possemato, and Higginbotham continued to examine concurrent partner talk during augmented-speaker composition as a major source of sequential misalignment affecting mutual understanding and social alignment.
Move and Engage reached major research milestones in Year 4. Move demonstrated that, after systematic training, an augmented speaker could use a Move-based AAC system to produce real-time turns with little or no temporal gap, including interruptions, insults, disagreements, repairs, and collaborative contributions. The project transitioned from a paper-based version to a dynamic OS-DPI interface, completed a replicated single-case pilot study, and prepared findings for publication. Engage documented engagement practices across AAC- and SGD-mediated interactions, using communication boards and natural speech as comparison points. Its results strongly favored shared visual access to the communication display, in sharp contrast to traditional SGD configurations that restrict partners’ access to message construction only. Together, Move and Engage showed that AAC design must support both rapid action by the augmented speaker and visible coordination with the partner during message construction.
Intone also reached an important decision point in Year 4. The collaboration with Eva Szekely produced valuable work on expressive speech synthesis, including experimental materials, a publication, and a conference presentation in Kyoto. However, the synthesis process proved difficult to control in real time for use with SGD. As a result, the team discontinued that line of interface development and shifted TTS work toward Microsoft Azure, which provided more practical control over prosody, loudness, and speech-act design for Move and DEAN.
By the end of Year 4, DEAN had become the field-ready AI-AAC probe through which the project’s major strands were brought together. DEAN was organized as a layered system involving hardware and communications, language, social participation, and interface design. On the hardware side, we refined the communication architecture, migrated from Google Cloud to a UB server, integrated Hutchinson’s own device into the DEAN environment, and developed a remote observation and testing system. On the language and social side, we developed processes for conversational memory, personal information, history/persona modeling, and action-sequence prompting. From an interface perspective, we built and tested early interactive interfaces for Hutchinson and participants with ALS, integrating AI-generated responses, Move utterances, and tone-of-voice controls.
Testing during Year 4 clarified the limits of generic GPT-generated AAC responses. Hutchinson’s evaluations revealed major problems in interpretation, response generation, personalization, and speaker alignment. These findings showed that a fluent AI response is not necessarily an interactionally useful AAC response. The system needed to produce utterances that were timely, contextually relevant, personally acceptable, and aligned with the speaker’s stance and self-presentation. This work led to more fine-grained frameworks for assessing response quality and helped define the next phase of DEAN development. It also produced a major scholarly milestone: the team wrote and submitted a paper on conversational AI, AAC, and the early development and testing of DEAN, which was published in ATIA’s Assistive Technology Outcomes and Benefits in 2025.
Year 5 focused on refining and field testing DEAN with two augmented speakers who represented different interaction ecologies: Todd, a lifelong direct-select MinSpeak communicator with cerebral palsy, and Linda, an oral speaker prior to ALS who has used eye-tracking access with a Dynavox SGD for the past 12 years. This comparison allowed us to examine DEAN not as a single interface, but as an adaptable AI-AAC system that had to be customized for different bodies, devices, access methods, communication histories, and timing constraints.
Major technical progress occurred in DEAN’s hardware and communication layer. We reduced LLM response delays from approximately 10 seconds in 2023 to about 1 second, substantially improving the system’s potential to support near-time conversational participation. We also integrated a web-based version of DEAN directly into the communication devices our participants use daily, eliminating the additional hardware previously required for testing. This shift made DEAN more practical for field use and allowed the project to focus more directly on interactional performance rather than system assembly.
Year 5 also brought advances in DEAN’s language, interaction, interface, and speech-output layers. We integrated RAG-based memory, persona information, and action-sequence prompting to improve the relevance, continuity, and speaker alignment of AI-generated utterances. We combined AI-generated responses with Move-based pragmatic utterances, creating a hybrid system that supported both context-specific response generation and rapid conversational action. Each participant’s interface was customized to their access needs, and we began using ElevenLabs speech synthesis to provide more flexible, conversationally responsive speech output during testing.
Field testing became the central activity of the final grant year. We designed and implemented a multi-session Move training program for each participant, with the number of sessions adjusted according to each person’s improvement in selection accuracy and speed relative to baseline. We then designed a 10-session conversation field-test protocol with two tasks per session to compare each participant’s use of the DEAN conversational probe with their everyday AAC system. User training and testing began in February 2026 and were completed in May 2026. These field trials allowed us to evaluate DEAN under real interactional conditions, including timing, access demands, utterance selection, partner engagement, speech output, and the fit between generated responses and each participant’s communication style.
Across Year 5, Project Converse also moved decisively into scholarly dissemination. We devoted substantial effort to analyzing data and preparing manuscripts across the project’s major strands. During the year, we published three papers, had two manuscripts under review, and prepared three additional manuscripts for submission. We also gave nine presentations at national and international conferences, including five national and four international presentations. Higginbotham delivered two keynote addresses at an international conference, with two additional presentations planned for Fall 2026.
Over the course of the grant, Project Converse evolved from a set of exploratory sandbox projects into an integrated research program centered on DEAN. Enchrony provided the temporal and sequential foundation; Move provided rapid pragmatic action; Engage showed the importance of partner visibility and shared attention; Intone addressed expressive speech and prosody; and DEAN brought these strands together in a layered AI-AAC probe.
The project’s central contribution is not simply the development of a new prototype, but the creation of a research framework for designing and evaluating AAC technologies as tools for real-time social interaction. Project Converse has shown that the future of AAC innovation depends on more than faster typing, better prediction, or more fluent text generation. It requires technologies that help augmented speakers remain in time, in control, visible to their partners, and recognizably themselves within face-to-face conversation.