Move Project

Director: Higginbotham
Supervising Researchers: Possemato & Bizovi
Associate Researchers: Satchidanand, Golleru, Buckley, Hutchinson
Start Date: January 2022

The purpose of this project is to prototype an AAC interface, which utilizes a variety of pragmatically effective word and phrasal constructions (e.g., discourse markers, strategic questions) for leveraging control and agency in an interaction. The project will culminate in the development and testing a hardware/software platform optimized for pragmatically effective face-to-face interactions. 

This initial R&D effort has let us explore and develop a variety of project components which will be used across many other R&D projects, including expressive speech synthesis, Project Open’s Open Source Design and Programmers Interface (OSDPI), data logging, and the application of our interaction analysis and literature findings to inform device design, assembly/fabricataction of our software/hardware prototype, etc.

Current work is focused on developing paper prototypes of PEP interfaces which we are trialing before committing to a digital interface later this spring. This project was recently presented at the Assistive Technology and Industry Association in early February.

Background

We use words and phrasal chunks all the time in our conversations. Work by Wray and colleagues (1999, 2008, 2000, 2011) has shown that individuals use pragmatically loaded words and phrases as a matter of course during conversation, and that  listeners appear to process phrases as meaningful contributions when between-utterance latencies are short.

We also know that experimental utterance-based systems can produce phrases with very short gaps between turns (Todman, 2000; Todman, et al., 2008, Higginbotham, et al., 2011). The use of phrasal constructions in augmented conversation this lab’s work on the Frametalker project (Higginbotham, et al., 1999, 2003, 2005, Higginbotham and Wilkins, 2006;  File, et al., 2003, Todman, et al., 2008).  Quickfires, a Contact component designed to provide quick contributions to ongoing general  interactions, serves as the basis for the PEP project. The screenshot below shows the quickfire feature on the right of the communication page. It consists of a series of pragmatically powerful words and short phrases which are used and understood as full-fledged contributions (i.e., utterances) when spoken.  In addition to their use of highly functional single word expressions, these items were also chosen for their ability to combine with other Quickfire items to perform novel expressions.

In April, 2022, Francesco Possemato began to examine the literature in conversation analysis and interactional linguistics related to the intonation, pragmatics, interaction structure of a variety of high frequency expressions (e.g., ok, well, oh) to determine how to appropriately represent them in an AAC system.  Below is an example of this work. A more complete report is listed in the Literature project on this site.

Developing Utterance Lists

In order to begin interface development we decided to try out our expressions in the form of lists presented on paper. A major goal of this project was to separate interface development from our content research – an ambition that was destined to failure 😀. As soon as we departed from the spreadsheet we started to develop our word and phrase lists on paper, it became immediately apparent that we were in the process of construction interfaces to implicitly support our ideas about how they would be used in practice.

We first designed the set of primary items that would form the basis of our system in a series of three rows. The first two rows of the Home Vocabulary consist of a set of unchanging items, while the 3rd row could be shifted to set a more negative tone when they are expressed out loud (when this is implemented in software)P

We also developed a number of more specific lists based on social pragmatic functionality of the list items. For example the following graphic depicts a version of the Greeting category separated by interaction position (1st, 2nd), as well as including a general response category:

In anticipation of a dynamic interface, we have been thinking about how individual Home Vocabulary items can be linked to sequentially related item lists:

Testing the interface

By October 2022, we began to print out the interface sheets on large pieces of paper to experiment with how the phrases could be used for interactional purposes: The picture below shows Francesco Possemato and Jeff Higginbotham, each with about 6 large pieces of paper with different word lists, struggling to have a conversation.

Armed with the directive of “no talking” we both attempted to use our paper PEP devices to converse about a pre-arranged topic. Conversations were always  difficult without restoring to spelling as we didn’t include any referents on our boards, which made the act of topic introduction and focus excruciatingly difficult. Talking beforehand about the setting and objects in the environment was helpful. It was also immediately apparent that familiarity with the lists, individual items and their location within a list and on a specific paper interface was critical to establishing and maintaining conversational “flow”. It was also important for the participants to regard their conversations as relevant and to “dig in” to their conversational roles in order to make best use of the list materials.  Finally, we abandoned the dual use of the PEP paper prototype in favor of a single paper prototype user and an oral speaker.

Move to Curated Paper Interface Pages

Based on our first usability testing, we began to make changes on the paper interfaces, engineering several different pages each focused on accomplishing a specific conversation activity. We began experimenting with the interfaces, first by moving our prototypes from Google Sheets to Excalidraw (http://excalidraw.com), then cutting up and arranging the individual lists into different configurations, as shown below:

We have now developed about a half-dozen paper interfaces, each focused on a specific conversation activity (e.g., general, greet, leave, reference, repair, evaluate). Here are a few examples:

Phase 2 Usability Work

In January 2023, we began to develop PEP through a series of interactions – to date – between Antara Satchidanand and Jeff Higginbotham, with only one of us using the communication boards.  To overcome some of the referential constraints faced in the first series of interactions, we started to specify the setting, personalities and motivations of the participants, and to provide each of them goals, though not always in a cooperative fashion.  Through these interactions we have begun do develop a set of task specifications that we intend to employ for training others and contexts which will make use of the PEP pages.  

One major consideration that emerged early on, is that its important that the PEP user has to engage with a highly pragmatic style, responding to their partner turning as fast as possible, without resorting to the use of the letter board.  To accomplish this, learning to use the Home vocabulary as much as possible.

We also engage in frequent breaks in PEP interaction to discuss what just transpired, perceived problems, observed strategies and anything else that could make aid in the development of a more effective system.

Future Plans (3/15/2023)

Currently, we are in the final rounds of content and paper interface development for this version of PEP, and will soon begin to test it with participants outside of the development team. To do that, we are beginning to develop a training device using ChatGPT to drill and assess our participants’ knowledge of their system, before using it with human partners.  We are also developing a set of interaction tasks to study and refine the usability of PEP during conversation. 

We will also start to develop a web-based version of PEP using the OSDPI provided by Project Open, a companion research and development project. Our intention is to have an initial prototype by the end of April, 2023.

Bibliography 

Wray, A. 2011. Formulaic language as a barrier to effective communication with people with Alzheimer’s Disease. Canadian Modern Languages Review 67(4), pp. 429-458.

Wray, A. 2008. Formulaic language: Pushing the boundaries. Oxford Applied Linguistics. Oxford: Oxford University Press

Wray, A. and Perkins, M. R. 2000. The functions of formulaic language: an integrated model.

Wray, A. 1999. Formulaic language in learners and native speakers. Language Teaching 32(4), pp. 213-231.

Higginbotham, D. J., & Wilkins, D. P. (2006). The short story of Frametalker: An interactive AAC device. Perspectives on Augmentative and Alternative Communication, 15, 18–21.

Higginbotham, D. J., Lesher, G. W., Moulton, B. J., Adams, K., & Wilkins, D. P. (2005). The Frametalker Project: Building an utterance-based communication device. In Proceedings of CSUN Technology and Persons with Disabilities Conference.

Higginbotham, D. J., Wilkins, D. P., Lesher, G. W., & Moulton, B. J. (1999, June). Frametalker: A communication frame and utterance-based augmentative communication device. In Proceedings RESNA (Vol. 99, pp. 40-42).

File, P., Todman, J., Higginbotham, J., Lesher, G. W., Moulton, B. J., & Aim, N. (2003). CONTACT: a communication aid based on pre-prepared phrases. In G. Chadwick (Ed.), Assistive Technology-Shaping the Future (pp. 304–308). IOS Press.

Todman, J., Alm, N., Higginbotham, J., & File, P. (2008). Whole Utterance Approaches in AAC. Augmentative and Alternative Communication , 24(3), 235–254.

Todman, J., Portia, F., & Higginbotham, J. T. (2003). Video extracts of effective social conversation using a whole-utterance VOCA. Assistive Technology-Shaping the Future.

Todman, J., & Rzepecka, H. (1 2003). Effect of pre-utterance pause length on perceptions of communicative competence in AAC-Aided Social Conversations. Augmentative and Alternative Communication , 19(4), 222–234.

Blog Posts