Remarkable new AI tech translates American Sign Language in real-time

For millions of deaf and hard-of-hearing people, using sign language to communicate in a world built around spoken words can be exhausting.

Whether it’s ordering food, asking for directions, or taking part in a classroom discussion, barriers show up everywhere.

While interpreters and captioning services can help, they’re often limited, costly, or unavailable when most needed.

As daily life becomes more digital, the need for smart tools that can translate sign language in real time is more urgent than ever.

That’s why researchers from Florida Atlantic University’s College of Engineering and Computer Science have created a new system that could change the way we think about accessibility.

They’ve developed a real-time American Sign Language (ASL) interpreter powered by artificial intelligence.

This system uses deep learning and hand-tracking to convert ASL gestures into written text, all using a regular webcam and off-the-shelf hardware.

Recognizing sign language is tricky for machines

ASL relies on precise hand shapes and movements to represent letters, words, and phrases. But current recognition tools often miss the mark – especially when signs look alike.

For example, “A” and “T” or “M” and “N” can be hard to tell apart for machines. These tools also struggle in poor lighting, with motion blur, and with differences in hand shape or skin tone, all of which affect how accurate the machine interpretations are.

To solve these issues, the FAU team combined two powerful tools: YOLOv11 for object detection and MediaPipe for detailed hand tracking.

Together, they allow the system to detect and classify ASL alphabet letters with an accuracy of 98.2% (mean Average Precision at 0.5). The entire process works in real time and with very little delay.

“What makes this system especially notable is that the entire recognition pipeline – from capturing the gesture to classifying it operates seamlessly in real time, regardless of varying lighting conditions or backgrounds,” said Bader Alsharif, lead author on the study.

“And all of this is achieved using standard, off-the-shelf hardware. This underscores the system’s practical potential as a highly accessible and scalable assistive technology, making it a viable solution for real-world applications.”

Seeing and understanding sign language

At the center of the system is a basic webcam, which captures live video and turns it into digital frames.

MediaPipe then pinpoints 21 key spots on each hand – fingertips, knuckles, and the wrist – to build a kind of skeleton map. These points help the system understand hand structure and motion.

YOLOv11 uses this skeletal data to match hand gestures accurately to ASL letters.

The researchers also built a massive dataset – 130,000 images strong – to train the model. These images include hands in a variety of lighting conditions, backgrounds, and angles.

This diversity helps the system learn how to generalize across different people and environments, thus reducing the chance of bias.

“This project is a great example of how cutting-edge AI can be applied to serve humanity,” said Imad Mahgoub, a co-author on the publication.

“By fusing deep learning with hand landmark detection, our team created a system that not only achieves high accuracy but also remains accessible and practical for everyday use. It’s a strong step toward inclusive communication technologies.”

A growing need

The deaf and hard-of-hearing community is large and diverse.

In the U.S. alone, about 11 million people – roughly 3.6% of the population – are deaf or have significant hearing loss. Around 37.5 million adults experience some level of hearing difficulty.

That’s a lot of people who could benefit from better communication tools.

“The significance of this research lies in its potential to transform communication for the deaf community by providing an AI-driven tool that translates American Sign Language gestures into text, enabling smoother interactions across education, workplaces, health care and social settings,” commented Mohammad Ilyas, co-author of the research study.

“By developing a robust and accessible ASL interpretation system, our study contributes to the advancement of assistive technologies to break down barriers for the deaf and hard-of-hearing population.”

Helping the deaf community say more

While this new tool already shows strong results for recognizing the ASL alphabet, the team isn’t stopping there. They’re now working to expand the system to understand full ASL sentences.

This would make communication even more natural and fluent, moving from spelling out words to sharing entire ideas.

“This research highlights the transformative power of AI-driven assistive technologies in empowering the deaf community,” said Stella Batalama, Dean of the Department of Electrical Engineering at FAU.

“By bridging the communication gap through real-time ASL recognition, this system plays a key role in fostering a more inclusive society.”

“It allows individuals with hearing impairments to interact more seamlessly with the world around them, whether they are introducing themselves, navigating their environment, or simply engaging in everyday conversations.”

This technology enhances accessibility and supports greater social integration, helping create a more connected and empathetic community for everyone.

With continued development, this AI-powered tool may soon become part of daily life, helping millions to communicate more freely – one gesture at a time.

The full study was published in the journal Sensors.

—–

Like what you read? Subscribe to our newsletter for engaging articles, exclusive content, and the latest updates. 

Check us out on EarthSnap, a free app brought to you by Eric Ralls and Earth.com.

—–