Development of sign language resources using artificial intelligence for people with hearing and speech impairments
"Breaking Barriers, Connecting Lives."
- Enhanced Communication Accessibility
- Social Inclusion & Equal Opportunities
- Improved Access to Education & Services
- Advanced AI & Machine Learning Solutions
- Economic & Professional Benefits
- Scalability & Technological Advancement
Aigerim Yerimbetova
Project & Manager
Call us anytime
+7 776 237 1777
Features
Here are three detailed main features of the project:
AI-Powered Sign Language Recognition and Translation
- The system utilizes deep learning models (such as Transformer, EfficientNet, ResNet) and computer vision algorithms to accurately recognize and interpret gestures from Kazakh-Russian sign language.
- It converts sign language gestures into text and audio with high precision, making communication easier for individuals with hearing impairments.
- The recognition system is trained using MediaPipe for keypoint detection and machine learning models for classification, ensuring real-time and accurate gesture recognition.
Bidirectional Speech-to-Sign and Sign-to-Speech Conversion
- The system enables real-time translation of spoken language into sign language and vice versa.
- It features an interactive avatar that visually represents sign language gestures, allowing hearing users to communicate effectively with deaf individuals.
- Speech recognition is powered by Conformer and Wave2Vec2 models, ensuring high accuracy in converting spoken words into text and sign language.
- The system also includes speech synthesis technology (FastSpeech2, Tacotron2) to convert recognized text into natural-sounding speech, making it easier for hearing-impaired users to interact with the world.
Integration with Educational and Digital Platforms
- The technology is designed to be implemented in learning management systems (LMS), mobile applications, and web platforms, expanding access to education for people with hearing impairments.
- It supports real-time transcription and translation of classroom lectures, online courses, and educational videos, ensuring accessibility for deaf students.
- The project includes customizable learning tools, such as an AI-powered sign language dictionary and interactive tutorials, to help users learn and practice sign language efficiently.
As a result of the program it is expected
AI for Sign Language Recognition
Optimized AI enhances Kazakh-Russian sign recognition, improving accessibility for the hearing and speech impaired.
Bridging Communication
AI translates speech to Kazakh-Russian sign language and gestures to text/audio for seamless understanding.
Adaptive AI Translation
A self-learning system enhances translation speed and accuracy through continuous improvement.
Tech for Inclusive Education
Integrating technology into learning platforms improves access to knowledge, promoting social and professional inclusion for those with hearing and speech impairments.
Experiments
a brief description of the most important experiments
Gesture Classification Network (Sign to Text)
Goal: Development and training of a model for recognizing gestures and converting them into text. Data: Using MediaPipe to collect data on key hand points. Process: Prepare data, train machine learning models (e.g. ResNet, YOLOv8) and evaluate their accuracy. Score: Metrics such as precision, recall, accuracy, F1-measure and others. Result: Textual representation of gestures for subsequent processing.
Avatar Text to Sign
Goal: Development of a model for translating text into gestures with visualization on a 3D avatar. Data: Kazakh-KSL Gloss parallel body. Process: Prepare data, train models (eg Transformer, Seq2Seq), and evaluate their accuracy. Evaluation: Metrics such as precision, recall, BLEU, and others. Result: Visualization of gestures on a 3D avatar to display translated text.
Speech Synthesis
Goal: Development of a model for speech synthesis from text. Data: Text data. Process: Data preparation, training of models (eg Tacotron 2, WaveNet), and assessment of their quality. Evaluation: Metrics such as MOS, MCD, and others. Result: Audio format of synthesized speech for voicing text.
Audio to Text
Goal: Develop a model for converting audio to text. Data: Audio recordings of speech data. Process: Prepare data, train models (eg Conformer, Wav2Vec2), and evaluate their accuracy. Evaluation: Metrics such as precision, recall, WER, CER, and others. Result: Text representation of recognized speech.