Fall 2022

I2T-ASL:
AI Hackathon Project

I2T-ASL, aka Imagine-to-Text American Sign Language, is a project created for the 24 hour Artificial Intelligence (AI) days Hackathon, hosted by the University of Florida. The program uses a custom-made trained model to recognize American Sign Language (ASL) numbers from 1 - 9. Then it would show four images, developed with Stable Diffusion using a prompt related to ASL and the number which was identified. The project won 'Honorary Mention'. Our aim for the program is to recognize full ASL sentences and generate original AI art through the text interpretation. This way, we bring an artistic voice to the Deaf Community.

What I Learned

After learning about this AI Hackathon, I teamed up with a peer from my ART-ificial Intelligence class and a CS student to make a project. As we were all beginners in machine learning, we went through a lot of hoops to settle on completable but worthy project. My very first attempt at making a model was by using Google's teachable machine, it gave us a great short-term model for the project. The idea of using Stable Diffusion to create images from hand signs came from our ART-ificial Intelligence class. While we weren't able to incorporate Stable Diffusion itself into the project, we were easily able to import the model, predict the hand number, and display the corresponding images which we created separately by typing a prompt along the lines of "[number] ASL artwork" into Stable Diffusion.

What I Did

Worked with two peers, one from the Art department and another from the Computer Science department. Our collaboration is what lead us to our multi-disciplinary project.
Trained my own ASL hand-numbers model through Google's Teachable Machine, then incorporated it into an HTML website through TensorFlow.
After tweaking the code to show the number which the model believes has the highest prediction, I included space for four images. Each group of four images were generated through Stable Diffusion and displayed with their corresponding number.

Honorable Memory

Our Hackathon awarded us with an 'Honorary Mention' award and $2000 prize split between all three of us for our project!

Media and Links

Devpost Link

Video of our Project:

See the process of logging in, submitting a fortune, receiving a fortune, and viewing your past fortunes and profile page (captions included):

Welcome Page

Welcome page, which is what you see when you first load the page

Back-end View

Visualization of what is happening in the back-end and the database.

Welcome Page

Welcome page, which is what you see when you first load the page

1/8

Generated Example Images

Each time a number was identified, four images would appear corresponding to that number. These images were generated through Stable Diffusion, a prompt-based deep-learning image model.