Driven by a desire to help people with disabilities integrate into society, a high school student in Dong Nai has developed an application that supports the visually and hearing impaired in communication and safe mobility.

Moved by the daily struggles faced by people with visual and hearing impairments, Phan Hai Dang, a student at Luong The Vinh High School for the Gifted in Dong Nai province, researched and developed a project titled “A multi-functional support system for the deaf and the blind using deep learning technology.”

Although not enrolled in the school’s IT-specialized class, Dang has nurtured a particular passion for programming, especially deep learning. Unlike many peers who choose to build entertainment or study-support applications, he opted for a more demanding path - creating technology to help the deaf and the blind communicate and integrate into the community.

The application he developed can convert sign language into text and speech, while also transforming spoken language into written text or corresponding sign language, enabling seamless interaction between both sides. The system integrates various AI technologies, including the YOLOv8 algorithm, ArcFace facial recognition and OCR text recognition.

With a simple tap on the microphone icon labeled “Please speak,” the application records speech and then displays a video demonstrating the appropriate sign language. Hearing-impaired users can watch the video to understand the message visually. Conversely, the camera captures hand gestures in sign language, converts them into text and generates audio output so that non-signers can understand.

“The application can also recognize emotions through eye movements and facial expressions to convey the correct tone and attitude of the speaker, enhancing interaction and social connection,” Dang shared.

For visually impaired users, the system provides navigation assistance through an integrated camera that detects objects ahead, measures distance and issues audio warnings to help avoid collisions. Facial recognition features allow users to identify relatives and make emergency calls via Google Assistant using voice commands.

In addition, the app integrates practical support functions such as traffic light recognition, currency denomination reading, book reading and location tracking.

Behind Dang’s journey to bring technology closer to people with disabilities stands the dedicated support of his teacher, Bui Kim Minh, who directly mentored him, nurtured his passion and opened doors for real-world application of his project.

According to Minh, from the moment Dang first conceived the idea of an AI-powered communication support tool for people with disabilities, he recognized the project’s creative potential. Rather than stopping at theory, he proactively connected with the Dong Nai Center for Nurturing and Educating Children with Disabilities to facilitate real-life testing.

“I believe that if technology is to have real value, it must be tested in real settings where users genuinely need it. As a teacher, I support connections so Dang can apply his creation in practice,” Minh said.

At the Dong Nai Center for Nurturing and Educating Children with Disabilities, teacher and student spent weeks listening, observing, testing and refining the system. Updated versions were continuously improved based on feedback from children with disabilities and those directly experiencing the product.

After multiple rounds of testing and evaluation, the system now achieves approximately 95 percent accuracy in recognizing sign language. For Minh, the true joy lies not in awards or recognition but in witnessing his student mature in both thought and action.

“Teaching a student to master technology is not difficult. Teaching them to use technology to serve the community - that is the deeper mission of education,” Minh emphasized.

With the idea of applying deep learning technology to convert sign language into speech and vice versa, Dang won First Prize at the provincial level and Third Prize at the national level in the Science and Engineering Competition for secondary school students.

Hoang Anh