Development of sign language resources using artificial intelligence for people with hearing and speech impairments
"Breaking Barriers, Connecting Lives."
- Enhanced Communication Accessibility
- Social Inclusion & Equal Opportunities
- Improved Access to Education & Services
- Advanced AI & Machine Learning Solutions
- Economic & Professional Benefits
- Scalability & Technological Advancement
Aigerim Yerimbetova
Project & Manager
Call us anytime
+7 776 237 1777
Features
Here are three detailed main features of the project:
AI-Powered Sign Language Recognition and Translation
- The system utilizes deep learning models (such as Transformer, EfficientNet, ResNet) and computer vision algorithms to accurately recognize and interpret gestures from Kazakh-Russian sign language.
- It converts sign language gestures into text and audio with high precision, making communication easier for individuals with hearing impairments.
- The recognition system is trained using MediaPipe for keypoint detection and machine learning models for classification, ensuring real-time and accurate gesture recognition.
Bidirectional Speech-to-Sign and Sign-to-Speech Conversion
- The system enables real-time translation of spoken language into sign language and vice versa.
- It features an interactive avatar that visually represents sign language gestures, allowing hearing users to communicate effectively with deaf individuals.
- Speech recognition is powered by Conformer and Wave2Vec2 models, ensuring high accuracy in converting spoken words into text and sign language.
- The system also includes speech synthesis technology (FastSpeech2, Tacotron2) to convert recognized text into natural-sounding speech, making it easier for hearing-impaired users to interact with the world.
Integration with Educational and Digital Platforms
- The technology is designed to be implemented in learning management systems (LMS), mobile applications, and web platforms, expanding access to education for people with hearing impairments.
- It supports real-time transcription and translation of classroom lectures, online courses, and educational videos, ensuring accessibility for deaf students.
- The project includes customizable learning tools, such as an AI-powered sign language dictionary and interactive tutorials, to help users learn and practice sign language efficiently.
As a result of the program it is expected
AI for Sign Language Recognition
Optimized AI enhances Kazakh-Russian sign recognition, improving accessibility for the hearing and speech impaired.
Bridging Communication
AI translates speech to Kazakh-Russian sign language and gestures to text/audio for seamless understanding.
Adaptive AI Translation
A self-learning system enhances translation speed and accuracy through continuous improvement.
Tech for Inclusive Education
Integrating technology into learning platforms improves access to knowledge, promoting social and professional inclusion for those with hearing and speech impairments.
Experiments
a brief description of the most important experiments
Gesture Classification Network (Sign to Text)
Goal: Development and training of a model for recognizing gestures and converting them into text. Data: Using MediaPipe to collect data on key hand points. Process: Prepare data, train machine learning models (e.g. ResNet, YOLOv8) and evaluate their accuracy. Score: Metrics such as precision, recall, accuracy, F1-measure and others. Result: Textual representation of gestures for subsequent processing.
Avatar Text to Sign
Goal: Development of a model for translating text into gestures with visualization on a 3D avatar. Data: Kazakh-KSL Gloss parallel body. Process: Prepare data, train models (eg Transformer, Seq2Seq), and evaluate their accuracy. Evaluation: Metrics such as precision, recall, BLEU, and others. Result: Visualization of gestures on a 3D avatar to display translated text.
Speech Synthesis
Goal: Development of a model for speech synthesis from text. Data: Text data. Process: Data preparation, training of models (eg Tacotron 2, WaveNet), and assessment of their quality. Evaluation: Metrics such as MOS, MCD, and others. Result: Audio format of synthesized speech for voicing text.
Audio to Text
Goal: Develop a model for converting audio to text. Data: Audio recordings of speech data. Process: Prepare data, train models (eg Conformer, Wav2Vec2), and evaluate their accuracy. Evaluation: Metrics such as precision, recall, WER, CER, and others. Result: Text representation of recognized speech.
Have a question? Check out the FAQ
You can find your questions and answers to them here
Q1
Feugiat pretium nibh ipsum consequat. Tempus iaculis urna id volutpat lacus laoreet non curabitur gravida. Venenatis lectus magna fringilla urna porttitor rhoncus dolor purus non.
Q2
Dolor sit amet consectetur adipiscing elit pellentesque habitant morbi. Id interdum velit laoreet id donec ultrices. Fringilla phasellus faucibus scelerisque eleifend donec pretium. Est pellentesque elit ullamcorper dignissim. Mauris ultrices eros in cursus turpis massa tincidunt dui.
Q3
Eleifend mi in nulla posuere sollicitudin aliquam ultrices sagittis orci. Faucibus pulvinar elementum integer enim. Sem nulla pharetra diam sit amet nisl suscipit. Rutrum tellus pellentesque eu tincidunt. Lectus urna duis convallis convallis tellus. Urna molestie at elementum eu facilisis sed odio morbi quis
Q4
Dolor sit amet consectetur adipiscing elit pellentesque habitant morbi. Id interdum velit laoreet id donec ultrices. Fringilla phasellus faucibus scelerisque eleifend donec pretium. Est pellentesque elit ullamcorper dignissim. Mauris ultrices eros in cursus turpis massa tincidunt dui.
Q5
Molestie a iaculis at erat pellentesque adipiscing commodo. Dignissim suspendisse in est ante in. Nunc vel risus commodo viverra maecenas accumsan. Sit amet nisl suscipit adipiscing bibendum est. Purus gravida quis blandit turpis cursus in
Q6
Enim ea facilis quaerat voluptas quidem et dolorem. Quis et consequatur non sed in suscipit sequi. Distinctio ipsam dolore et.