Camera-Based Facial Expression Instrument

I'm severely disabled without the use of my hands or arms, but in another life I used to be a musician. I would like to create a system that would allow me to turn simple tongue movements into a way to play a musical instrument, using expressions such as those seen in the following video

to trigger MIDI notes. Yes, I know I look ridiculous ;)

I actually have some existing algorithms that can turn just about any stream of data into MIDI notes. All I'd need from you would be a program that sends

• x and y coordinates of the center of any tongue bulge that appears
• some sort of estimate of the size/intensity of the tongue bulge (perhaps number of pixels displaced?)

If your program appeared as a Human Interface Device (HID) that might be ideal, but we can discuss any strategy here that works. Sticking the tongue out completely would ideally also count.

The first idea that comes to mind for me would be using machine learning to create a gesture recognition system to identify the tongue bulges. I've seen demos that only needed to be shown each gesture a couple of times to learn it, so I feel like it should be very possible. Any other strategy could be employed however also. If it works, it works.

When it comes to audio things generally need to be kept below 20 ms of latency or it starts to become compromising. For this reason I think I might need to get a high-speed webcam (at least 50 FPS, but 100 might be better). If you have any suggestions of good hardware I could buy, I would be all ears. I was thinking that it might not even be necessary to send high-resolution images into the computer if it could make things more efficient, but your advice on these things would be much appreciated.

If you think you would like to take this job on, please get a hold of me. All code will need to be clean, organized and very well-commented. Please contact me if you have any questions or would like clarification about anything! If you see a better way to do something than what I'm suggesting, please bring it up!