AI-Enhanced AAC: DEAN

DEAN as a User-Controlled Engagement Support

The qualitative findings were drawn from post-session questionnaire comments, end-of-study debrief interviews, and stimulated-recall interviews in which Todd and Linda reviewed selected video excerpts and reflected on their use of DEAN and their personal AAC systems. In this analysis, Todd and Linda are not treated simply as study participants. They were expert AAC users and co-researchers whose lived experience with AAC was central to understanding what DEAN did well, where it failed, and what the next version needs to support.

Todd and Linda came to the study with very different AAC histories. Todd is a lifelong AAC user with quadriplegic cerebral palsy who relies on a highly learned PRC-Saltillo Minspeak system with manual direct selection and keyguard support. His Minspeak system is not just a device. It is a deeply practiced language system tied to his motor patterns, vocabulary knowledge, and communication style. Linda began using AAC after losing functional speech due to ALS. She uses eye-tracking access with a Tobii Dynavox I-15 and Communicator software. Her system supports precisely authored messages through a QWERTY keyboard and word/phrase prediction, but message construction requires sustained visual attention and time.

These differences shaped how each participant evaluated DEAN. Todd evaluated DEAN in relation to Minspeak, symbol sequences, familiar word retrieval, and whether the system sounded like him. Linda evaluated DEAN in relation to eye-gaze demands, typing efficiency, timing, and her ability to enter the conversation before the relevant moment had passed.

The clearest finding is that DEAN should not be framed as a replacement for the user’s personal AAC system. That is not what Todd or Linda wanted. DEAN was useful when it helped them get into the conversation faster, hold their place, reduce the burden of constructing every message from scratch, or keep the partner oriented to their participation. It became problematic when it generated utterances that were too generic, too agreeable, too polite, or misaligned with what the user was actually trying to do in the conversation.

Note: AAC = augmentative and alternative communication; AI = artificial intelligence; DEAN = Dynamic, Expressive, Augmented Narrator; Move = pragmatically organized conversational vocabulary interface.

Theme 1: DEAN Helped With Increasing the Proportion of Utterances Within the Enchronic Frame

A major question driving our research with DEAN was, can the system respond quickly enough to enable augmented speakers to use their devices in NOW and NEAR TIME? Based on their responses in the debriefing, both participants identified timing as one of DEAN’s most important strengths. Todd said that DEAN made him feel more like he was “talking” because it was more immediate and because Kendra was more engaged while listening. When DEAN worked, Todd was not simply preparing a delayed contribution; he experienced himself as an active speaker in the interaction.

Linda described a related but different timing function. For her, DEAN could help “earmark the place” in the conversation. This is especially important for people who use AAC. A typed message may be accurate and fully authored, but if it is produced too late, the moment it responds to may already be gone. In that sense, DEAN’s value was not only in generating complete messages. It was also in providing a fast initial move that signaled, “I am responding here. Hold this place while I compose more.”

The point is straightforward: timing is part of conversational competence. In conversation, a response that arrives too late can lose its relevance. DEAN’s strongest contribution was its potential to help augmented speakers stay in the temporal flow of interaction.

The effect of reducing turn-taking gaps and keeping the augmented communicator within the temporal flow is illustrated in the following 8-minute video clip, in which Todd uses DEAN to discuss where Kendra should go on vacation. Only 6 of his 35 turn transitions are delayed (10 seconds or greater). Todd takes 30 aided and 5 unaided turns. As expected, all of his unaided turns were within Now time, but importantly, of his aided turns, 80% (24) were within Now or Near time. It is clear that both Todd and Kendra are highly engaged in their conversation, making frequent eye contact, and that, based on the direction of her gaze, Kendra’s attention does not appear to drift away from Todd or his AAC device during message construction.

In this clip, we see Todd using DEAN efficiently in the greeting sequence. In addition to indicating how he is doing at 0.12 seconds, he quickly selects a turn-around, “Got any fun plans?” at 0.14 seconds. At 0.29 seconds, Todd comments on Kendra’s previous turn, “That sounds awesome”, then does another quick turnaround to ask, “Where are you thinking of going?”, at 0.31 seconds, further moving into topical talk about Kendra’s vacation plans. At 0.53 seconds, Todd suggests that Kendra consider taking her vacation in Florida. It should be noted that his background of having family in Florida and making frequent trips there is part of Todd’s persona in the DEAN probe. HIs Person also includes that he lives in Buffalo and things he likes to do there.  When Kendra rejects Todd’s suggestion, he looks but apparently doesn’t find a satisfactory AI prediction so he uses a Move, “Please” at 01.05 seconds, urging her to consider his suggestion. When Kendra asks him why he is being so insistent, he is able to follow with a relevant AI prediction, “I think you’ll love it there”, at 01.23 seconds. So far, the conversation is going well, however, Todd does experience issues with some of the AI predicted messages as we will see below. 

In this clip, Teagan begins the interaction by asking to turn off the country music. Within about 5 seconds, at 0.15 seconds, Linda uses a Move, “No” to reject Teagan’s suggestion. In her next turn, beginning at 0.16 seconds, Teagan asks “Why,” and complains, “I listen to it all day with you,” when Linda interrupts her (0.21 seconds) with another rebuttal, “Are you kidding!”. Near the end of the clip (0.47 seconds), Teagan suggests, “Maybe we could put on some, I don’t know, Taylor Swift or something like that.” Then, at 0.56 seconds, Linda interrupts her (again) with a hedge Move, “I don’t know.” This clip demonstrates the effectiveness of Move to help Linda stay within the enchronic time frame. 

Theme 2: AI Must Support the User’s Agenda, Not Merely Continue the Conversation

The interviews showed that DEAN sometimes produced utterances that were topically relevant but pragmatically off. For example, in Todd’s store-return scenario, DEAN generated responses that fit the general topic of clothing, returns, and garment care. But Todd was not trying to be a generically helpful store employee. He wanted to take a stronger position: the customer had washed the shirt incorrectly, and the problem was not the store’s responsibility.

DEAN did not give him that stance. Todd found the AI too friendly, too accommodating, and unresponsive to his goal. The system kept the interaction moving, but it did not help him direct it the way he was trying to.

Linda identified a similar problem in scenarios requiring disagreement or negotiation. She described the AI as offering “only consensus.” When she needed to object, explain why, or pursue a contrary agenda, DEAN often offered agreement, compromise, or affiliative follow-up questions. Those responses fit the topic but they were not what she needed to reach her goal in the conversation.

This is a central design finding. Conversational AI for AAC cannot be optimized only for plausible next turns; it must support the user’s agenda. That includes disagreement, refusal, resistance, counterproposal, repair, qualification, humor, escalation, and compromise. An agreement is useful sometimes. But a system that overproduces agreement can weaken the augmented speaker’s agency.

This short clip shows the beginning of a conversation where Linda and Teagan role-play planning a surprise birthday party, and Linda strategically uses the yellow buttons to prompt DEAN to refine responses. After Teagan’s turn, which begins at 0.12 seconds, Linda uses a yellow button to bring up alternative responses and selects, “Sounds like fun, let’s plan.” At 0.21 seconds, Teagan begins a 17- second turn (0.22 – 0.39 seconds) during which Linda selects a yellow button multiple times to refresh AI responses. This facilitates a rapid turn transition (3.3 seconds), and she asks, “Which Saturday works best for Antara?” at 0.42 seconds. After Teagan’s next turn, which begins at 45 seconds, for 7 seconds, Linda uses the yellow buttons to refine other responses, then at 0.56 seconds, Teagan takes another turn to which Linda quickly replies at 1.05 seconds with the response, “Yes, that sounds perfect,” which she was about to say before Teagan took another turn. 

Linda had access to the yellow buttons from the third day of field testing. For the rest of the trials, she actively experimented with using them to find and refine responses that stayed up with the flow of conversation. This was particularly important because Teagan frequently took long turns in which the relevance of AI predictions changed over the course of her turn. Although the yellow buttons, when made available, were useful in helping Linda and Todd probe for more relevant responses, using them, especially after the typical speaker’s turn, increased the turn transition time. The timing of refreshing and access to AI responses will continue to be addressed in future versions of DEAN.

This clip demonstrates how Linda was initially able to keep a negotiation going using Moves and AI-generated responses, but to make her stance, she eventually needed to author a message with the QWERTY keyboard.  In this clip, Linda and Teagan disagree about how many guests to invite to the birthday party they are planning. At 0.05 seconds, Linda uses the soft rejection Move, “I don’t think so,” to object to Teagan’s previous suggestion that they should invite 75 or more guests. Between 0.06 and 0.45 seconds, they go back and forth in their negotiation with Linda using AI-generated responses. At 0.34 seconds, Teagan explains the person they are planning the party for has “…a lot of friends and a pretty big family…so that number of people is not unreasonable.” Linda follows at 0.48 seconds with a Move hedge, “I don’t know.” At 0.53 seconds, Teagan asks a very specific question: “Why don’t you want to invite that many people?” At 1 minute, Linda begins constructing her specific answer on the QWERTY keyboard, which takes 2 minutes 45 seconds, and says, “It’s harder to keep it a surprise.” 

Theme 3: Relevance Must Include Role, Stance, Goal, and Personal Voice

Todd and Linda both made clear that “relevant” AI output is not enough. A response can fit the topic and still fail the user. Todd laughed at some AI responses because they made sense in the situation, but they were not things he would have said. This distinction is important: DEAN sometimes spoke on Todd’s behalf without speaking “as Todd”. 

For Todd, the next version of DEAN needs stronger modeling of role, goal, stance, persona, and conversational partner. In the role-play scenarios, he needed responses that aligned with his assigned role, intended strategy, and preferred interpersonal style. He also raised the possibility that the system should know who he is talking to and adapt based on the relationship and prior interaction history.

Linda’s comments point to the same issue. When the scenario required objection, the system needed to support objection. When the task required negotiation, the system needed to support negotiation. The relevant unit was not just the partner’s previous utterance; it was the participant’s role in the scene, her stance toward the partner’s proposal, and the direction she wanted the interaction to take.

The design conclusion is direct: In addition to meeting the user’s communication goals in conversation, conversational AI should generate user-authorized options that fit the user’s role, stance, action project, and communicative identity.

This short sequence demonstrates how some AI responses can be topically relevant, but not meet the user’s goals in the conversation  and some are just off topic. Things start off well, then at 0.18 seconds, Kendra asks Todd what else he likes about Florida, to which he replies, “Love water. It’s calming”, at 0.22 seconds. Kendra agrees that she also loves water but, at 0.28 seconds, points out, “There’s water elsewhere.” Todd spends about 10 seconds considering AI-predicted responses, then immediately interjects after Kendra’s turn at 0.42, saying, “Buffalo’s water is the best.” They both laugh at this comment because it is clear from when Buffalo water came up earlier that neither of them has high regard for Buffalo’s beaches. As Todd mentioned in the debrief, he sometimes chooses a response just to move the conversation along and not because it best represents his point of view. He also laughed when he spoke it, suggesting it should perhaps be taken as a joke. Kendra laughs, suggesting she knows Todd is joking, then indicates she disagrees at 46 seconds. At this point, apparently continuing to joke, Todd chooses a wildly off-topic response, “I love hotwings,” at 50 seconds and both continue to look at one another and laugh. Although both Todd and Kendra seemed to enjoy the joking, Todd’s frustration during the debriefing interviews, as AI predictions did not reflect his stance, is clearly evident in this segment. 

Theme 4: Personal AAC Systems Remained Central to Authorship

Both participants found useful features in DEAN, but neither treated it as a replacement for their personal AAC system. Todd continued to prefer Minspeak because it is his established language system. It gives him access to words through familiar symbol sequences, reduces spelling burden, and reflects decades of practice. His critique of DEAN was not just that it made errors. It lacked his word system and sometimes produced messages that did not sound like him.

Linda’s Dynavox remained essential for a different reason. It supported precisely authored expression. DEAN could help her engage in the interaction, but when she needed specific details, nuanced explanations, or fully authored turns, she still relied on typing either via the QWERTY keyboard in DEAN-mediated interactions or on her personal Dynavox device. The problem was not authorship. 

These findings suggest that continued development of AI-enhanced AAC applications should work as a support layer coordinated with the user’s personal AAC system or analogous features. For augmented communicators like Todd who use Minspeak or other word-based systems, this suggests integrating AI as an adjunct to their current AAC language system. Additionally, support for reading AI-generated messages, such as wireless earbuds, may be needed to reduce the cognitive burden of reviewing and selecting messages for people like Todd who are not proficient in reading. For highly literate users like Linda, these points toward AI-enhanced systems that provide turn-entry, stance-marking, and floor-holding support that works alongside keyboard-based authored message construction with spelling and word prediction. For both types of augmented communicators, as AI improves, there is potential to build in more of their desired capabilities, making the application of AI in AAC technology an evolving process.

Theme 5: Partner Engagement Increased When Participation Became Visible or Audible

Todd’s stimulated recall interviews revealed something especially important. When using DEAN (Moves or AI-predicted responses), he felt like he was “talking” and noticed that Kendra was listening more attentively. This appeared to be due to the immediacy of his contributions compared with composing messages with his Minspeak system. After using DEAN on the first day of the study, he changed his own Minspeak output mode so that words were spoken as he composed them. This suggests that using DEAN helped him see the value of making message construction more visible or audible to the partner.

This observation connects directly with the broader findings from Project Engage. Partner engagement increases when the communication partner has more frequent access to the message as it is being constructed. Todd seemed to recognize this in practice. Word-by-word output made his participation feel more immediate and harder for the partner to drift away.

But, for Todd, this came at a cost. Word-by-word output made the errors public immediately. If he selected the wrong item, he spoke the wrong word before he could correct it. So incremental output can increase engagement but also increase risk. It lowers the boundary between composing and publicly saying.

Linda’s account raised a parallel issue. If she is composing silently, partners may move on. But if she must constantly monitor AI options, she may lose eye contact and social contact. Dean, therefore, needs to support partner engagement without overloading the augmented speaker’s visual attention or exposing unintended output.

The broader design need is configurable publicness. Users should be able to decide when composition remains private, when partial output becomes visible or audible, when a floor-holding signal is produced, and when a message is formally spoken.

Theme 6: The Most Important Design Direction Is User-Controlled Integration

The next version of DEAN should be designed as a user-controlled conversational support layer, not as an autonomous message generator. Its job should be to help with timing, stance, word retrieval, spelling burden, partner orientation, and sequential placement while preserving the user’s authority over what is said.

Next iterations of AI-enhanced AAC for users like Todd, should consider a parallel Minspeak + AI model. Todd imagined AI operating alongside his own system so that he could begin from a familiar Minspeak organization and then receive AI-supported information or alternatives after he initiates a direction. For users like Linda, the strongest direction might be a sequential AI-plus-spelling-and-word-prediction model. This could help her enter the interaction quickly, hold the floor, and mark the relevant prior turn. Spelling and word prediction, as available on her Dynavox, would remain essential for fuller authored expression, but it should also be able to direct AI responses with short prompts related to the user’s goal in the conversation. 

All of this makes clear that a single conversational AI architecture will not meet the needs of the wide variety of AAC users. The focus should be on making systems configurable around the user’s existing AAC ecology.