Building Real-time Intelligent Grounding in Deaf Education (BRIDGE)

Project Overview

BRIDGE is a groundbreaking Artificial Intelligence (AI) tool designed to revolutionize collaborative learning for deaf and hard-of-hearing (DHH) students in university STEM (science, technology, engineering, and mathematics) classes using American Sign Language (ASL).

Background

Deaf and signing communities represent a largely untapped reservoir of potential talent in STEM disciplines. DHH people in America have been historically marginalized, leading to their inclusion in the “missing millions” of STEM professionals in the U.S. workforce (Blatecky et al., 2021). A major challenge in deaf science education is the lack of standard signs in American Sign Language (ASL) for many scientific concepts. For example, one student might fingerspell a term, another might use a sign they created, and a third might use a different sign altogether. This variation can make it difficult for students to engage effectively in class without a shared understanding of scientific terminology. Moreover, most learning technologies are designed for hearing learners and overlook the unique communicative behaviors and needs of DHH students, particularly those who rely on a sign language, such as ASL. This oversight contributes to the significant underrepresentation of DHH individuals in STEM fields, as increasing DHH students’ understanding of foundational STEM concepts will increase their chances of entering a STEM occupation, and technology can play a critical role in accelerating conceptual knowledge acquisition.

In an attempt to address these issues, we are developing a new AI tool designed to revolutionize collaborative learning for deaf students in science, expanding upon the ways they currently communicate to support their convergence on STEM concepts. The tool will use augmented reality, signed animations, and sign recognition to provide real-time information about the signs used in classroom conversations. Ultimately, we aim to provide empirical evidence of the effectiveness of the BRIDGE tool in a STEM educational context.

Two people wearing AVP headsets while learning about photosynthesis

“Technology plays a crucial role in bridging communication gaps for many, especially the Deaf community...By enabling effective communication, these innovations foster connections and understanding between deaf and hearing individuals. Social inclusion is essential for building strong, diverse communities, and technology is a powerful tool for achieving this goal.”

-Dennis Valansi, 2025

Project Objectives:

  • Design and prototype the BRIDGE tool

    Our goals are to co-design and prototype a tool that provides lexical support with an avatar (a virtual human) signing relevant STEM signs to promote a shared understanding of key concepts and provide both just-in-time support to students. We plan to explore what kind of support BRIDGE should deliver and when, including varying where information is placed and the best way to represent information using a combination of signing avatars and English captions. We anticipate exploring both just-in-time support (e.g., presenting a STEM sign to students that they may have forgotten) and dashboard-based support (e.g., visualizing how students in a group have used a particular STEM sign over time).

  • Develop AI-powered recognition and generation of STEM signs

    We will concurrently build AI technologies to recognize when students sign a STEM term from undergraduate-level biology content to keep the lexicon limited in scope. This will be done by capturing both the conceptually-aligned signs and other varied signs that DHH signers use to describe the key concepts. The variability will lend authenticity to our AI models and allow diverse signers and diverse signs to be well-captured by BRIDGE.

  • Evaluate strengths and weaknesses of BRIDGE

    We will conduct user experience testing focused on the ease of use, perceived helpfulness, and both qualitative and quantitative assessments of the BRIDGE prototype's strengths and weaknesses. This feedback will be instrumental in refining the tool and enhancing its usability and effectiveness, as well as informing design principles for similar technologies. Data will also be analyzed to determine whether students’ conceptual understanding of STEM topics improved after using the BRIDGE tool, and whether the BRIDGE tool facilitated their collaboration.