This was my team’s final project for the Fall 2025 Artificial Intelligence class. It’s a real-time classification system that recognizes 36 ASL alphabetical and numeral gestures (26 letters, 10 digits) using deep learning and computer vision.
I have always been fascinated by how people communicate through sign languages using hand shapes, movements, facial expressions, and body postures, instead of doing so verbally. While I understand that not using words to convey a message or ask a question can sometimes be difficult, I personally believe it’s quite beautiful to listen with our eyes. We cannot look away and keep listening, which I think creates a deeper, more focused level of engagement between people.
Returning to the project, our goal was intentionally grounded: not to capture the full richness of ASL, but rather to build something that works reliably as a stepping stone towards a more inclusive and accessible communication technology.
The most interesting part of this project wasn’t the end model itself, but a preprocessing pipeline that helped boost the performance and accuracy of the training. Instead of an in-depth overview, on this page, I will briefly discuss how we did it.
1. The Dataset
The training dataset was constructed by merging two publicly available datasets: Kaggle’s ASL dataset and Roboflow’s ASL Letter Dataset. These two datasets yielded a total of 7,200 RGB images, with 200 samples per class across 36 classes. This gave us a comprehensive collection of hand gesture images with clear hand appearance and different backgrounds. However, one of the most persistent challenges in sign language classification is dealing with the visually similar gestures. For instance, A, E, and S; M and N; or V and 2. This often leads to misclassification.
Manual ASL alphabets and numerals. Image: Wikipedia Commons Free Media RepositoryTherefore, it is necessary to be careful when applying data augmentation. For example, a horizontal flip can turn a typical directional sign into another or even a non-existing one, potentially confusing the model. Moreover, despite its diversity, this curated dataset presented some significant obstacles, including poor image quality in some samples, inconsistent lighting, different skin tones, motion blurs, hand sizes, and noisy backgrounds. All of these did pose significant issues for our CNN-based classifier.
2. MediaPipe-Based Dataset Re-annotation
To address these challenges, a landmark-based preprocessing pipeline was developed to re-annotate the entire dataset using MediaPipe’s hand landmark-based bounding box detection framework. So, unlike traditional bounding box that entirely encloses the hand without precisely identifying its internal structure, this approach detects the 21 anatomical key points on each hand.
Coordinates of the 21 hand joints. Image: ResearchGateEvery dot represents a unique position, and provides a robust structural representation of hand geometry independent of background context. A bounding box then fully encloses all these 21 predicted landmarks with a little extra padding before the image is cropped. Through this pipeline, all images were processed and the isolated hand regions were extracted. This approach offers several advantages, including removing irrelevant contextual information from the background, ensuring consistent centering of the subject, and preserving finger articulation while maintaining gestural completedness.
The dataset was then partitioned using stratified sampling for a balanced class distribution across split: 80% for training (5,760 images), 10% for validation (720 images), and another 10% for testing (720 images).
3. Model Architecture
Our model is designed as a custom Convolutional Neural Network (CNN) that accepts 64×64×3 RGB images as input. The network is structured around three convolutional blocks of increasing depth, each progressively shrinking spatial resolution while extracting richer features.
(1) The first block stacks two 32-filter convolutional layers before a 2×2 max pooling step halves the resolution to 32×32, followed by a 0.25 dropout.
(2) The second block repeats this pattern at 64 filters, pooling down to 16×16 with another 0.25 dropout.
(3) The third block uses a single 128-filter convolution, pools down to 8×8, and applies a heavier 0.4 dropout. Because deeper, more parameter-dense layers are inherently more prone to overfitting, this extra regularization is introduced.
After flattening the resulting 8×8×128 feature map into a vector of 8,192 values, the network passes through two fully connected layers of 256 and then 128 neurons, separated by a 0.5 dropout to further discourage memorization. Finally, a dense output layer maps the features down to 36 classes using a softmax activation, producing a probability distribution over the 36 ASL classes.
In total, the model has roughly 2.27 million parameters, all of which are trainable. The two dense layers account for the vast majority of this count, with the first fully connected layer alone contributing over 2 million parameters.
4. Three Stages
To achieve the overall objective of developing an inclusive ASL interpretation system, the project follows a structured pipeline comprising three interconnected rounds with distinct functional roles across the entire recognition workflow.
Stage 1: Training
We started with two baseline models on the initial dataset: a CNN trained directly on raw images (88.47% test accuracy, 6.86% overfitting gap) and an SVM using HOG features (84.17% test accuracy, 15.15% overfitting gap). Both showed clear signs of overfitting — the SVM badly so. Then, we trained our custom CNN architecture on the newly MediaPipe-annotated dataset using the Adam optimizer, incorporating early stopping and adaptive learning rate reduction.
This approach achieved an impressive 97.92% test accuracy with validation accuracy (98.47%) slightly exceeding training accuracy (97.90%). This negligible gap strongly suggests the model successfully generalized without overfitting.
This stage produced a final trained model artifact which will be used in the next two stages.
Stage 2: Real-time inference
The trained model runs on live webcam input at a resolution of 640×480 at 30 FPS via OpenCV. Each frame goes through the same MediaPipe hand-localization pipeline: The hand region defined by the bounding box is cropped from the RGB frame, resized to 64×64 pixels, and normalized to [0,1]. A batch dimension is added to create a 1×64×64×3 tensor compatible with the CNN input requirements before feeding to the trained model from stage 1.
To handle real-world deployment, raw softmax outputs go through two layers of stability filtering: a confidence threshold of 0.5, and a temporal voting mechanism that requires the same prediction to appear at least twice across three consecutive frames. Only stable predictions are visualized, reducing the visual flicker you’d otherwise get from frame-to-frame noise.
Stage 3: Interactive aggregative application
Here, we had reached our main goal which is to make more practical use of the model by implementing letter accumulation, word formation, and multiple output modalities.
Built on top of Stage 2, this phase introduces a text buffer that accumulates stable predictions into words. A duplicate prevention mechanism rejects the same letter if it is detected twice within 0.5 seconds, which handles the natural tendency to hold a gesture slightly too long. The buffer also tracks active word counts, letter counts, and last added character, with an integrated option to save the output text to a file.
5. An Afterword
This system is far from a finished solution, as it recognizes only static letters and numbers rather than the continuous, fluid motion that makes up real conversational sign language. Transitioning to full Continuous Sign Language Recognition (CSLR) introduces severe challenges, including coarticulation effects, the multi-channel nature of sign language, data scarcity, and high computational complexity. Consequently, while researchers have successfully built continuous models for niche use cases, broader and universally reliable applications remain in their infancy.
I came across a 2025 study that developed a continuous Saudi Sign Language model trained specifically on medical-related sentences. The researchers utilized the KAU-CSSL dataset, a collection of videos representing full sign language sentences, to train a model using a pretrained ResNet-18 architecture. The model ultimately achieved a 99.02% accuracy for known signers. However, its accuracy dropped significantly to 77.71% when evaluating new individuals, illustrating a persistent, industry-wide hurdle that is real-world signer variability.
Despite these algorithmic barriers, the development of assistive technology for individuals with either hearing, visual, or speech impairments remains absolutely crucial. I am optimistic, although progress continues to come incrementally. At the end of the day, projects like this one are profoundly worth building, even when they only close a small part of the gap towards true, universal accessibility.
You may also find the following papers on similar work in sign language classification with deep learning insightful:
- Sign Language Recognition with Convolutional Neural Networks. Gangal, A., Kuppahally, A., & Ravindran, M. Read it here
- DeepASLR: A CNN-based human-computer interface for American Sign Language recognition for hearing-impaired individuals. Kasapbaşı, A., Ahmed Elbushra, A. E., Al-Hardanee, O., & Yılmaz, A. Read it here
P.S. A big shout-out to my incredible teammates for making this work as a whole enjoyable, and to my professor for her guidance.