I'm disabled without the use of my hands and would like to create a system whereby I could use facial gestures captured by my webcam to operate my computer, play midi notes, etc. I discovered the open-source project OpenFace (https://github.com/TadasBaltrusaitis/OpenFace) and it seems very adept at capturing all sorts of useful data. All I need is for the project to be modified to stream out OSC messages to a specified port and I will be able to take it from there.
I want to start out with making this job very simple; if things go well, there are many additional improvements we could continue to work on. For now, all I would like to have done is
- have it automatically stream data from a limited number of data streams to a default port of 6731 on localhost
- the data streams I have in mind would just be the three Orientation and three Pose streams
- the topics for the OSC messages could be
- /OpenFace/orientation_turn
- /OpenFace/orientation_up-down
- /OpenFace/orientation_tilt
- /OpenFace/pose_x
- /OpenFace/pose_y
- /OpenFace/pose_z
- I'm using the Windows GUI for OpenFace (OpenFaceOffline.exe) and I would like the finished product to also be in the form of an executable rather than a commandline tool.
Possible future additions
- instead of these six data streams sending absolute numbers, have them send numbers relative to their initial readings when the webcam stream was begun
- implement a sort of recalibration hotkey (default of F12 most likely) that when pressed replaces those readings with new ones taken from the current value of streams
- implement and send OSC messages for the rest of the "action units"
- see if we can get the frame rate above 30 FPS. My webcam is capable of 100+ FPS on video with small dimensions
- have the IP and port number be user-adjustable in the settings somewhere
- have the topics for the OSC messages also be user-adjustable
- set up the webcam feed and stream of OSC messages to automatically begin when the program loads
- give the recalibration hotkey a dual-purpose by also making it pause/unpause the webcam feed and OSC stream. The recalibration could then take place only upon unpausing
You're the expert here, so I need your opinions. Do you think the approach I describe will be able to achieve my goals? Do you have any better ideas, ways to refine the strategy or a completely different suggestion?
We can discuss an appropriate deadline, but I am most concerned simply that we stay in constant communication about the progress of the project.
All code will need to be clean, organized and very well-commented. Please contact me if you have any questions or would like clarification about anything! If you see a better way to do something than what I'm suggesting, please bring it up!