Head-mouse Prototype Success!

Hey everyone!  I'm going to brag about Kevin's new project!  I understand that I'm writing to a varied audience here; some of you are very tech-y and some of you are more like me and just need the picture painted in broad strokes.  I think you know who you are, but if you need some help deciding, I offer this: When I say "Raspberry Pi, " do you think dessert?  If so, then you're with me.  Go ahead and cut yourself a nice big wedge of pie and read my scoop, the "tech lite" version.  And those of you already drooling for some details can just skip below to Kevin's own words!  

The "tech-lite" scoop...

Kevin had discovered, through his visits to the Perlman Center, that when people with motor disabilities need to be able to use their computers without using their hands (in other words, by using a head-mouse or an eye-gaze tracker), they are often limited in very specific ways.  Products are designed to work with one type of machine (for example, one that runs Microsoft Windows) and they do not work with other machines.  Plus, these accessibility products are pricey (think thousands) and sometimes awkward to use.  If you are a person who wants to use a lot of different kinds of machines, then this situation isn't going to work.

So, Kevin purchased an inexpensive webcam and a very tiny (also inexpensive) computer, and then wrote a bunch of code so that....

TA-DA!
(Raspberry Pi Zero on left, Raspberry Pi 4B on right with attached camera module)

He can plug this little do-dad into any USB port on any device (he thinks--still in testing stages) and control the mouse using head movements.  Doesn't matter if the machine is running Windows or is a Mac, may even work with gaming systems.  And the whole thing costs between $30-$60.  Yay!  He's still fine-tuning the code, and he's reached out to a few places, including Team Gleason, to see if there is any interest in helping him through these final stages.  He'd like to be able to get this easy and inexpensive solution in the hands of people who could benefit.

The nitty-gritty, in Kevin's words...

"I have met with some wonderful speech and occupational therapists (at The Perlman Center at Cincinnati Children’s Hospital) who introduced  me to an array of computer access and AAC devices that use head movement or eye tracking technologies. The meetings quickly turned bi-directional as I showed them how I could completely "remote control" my android phone from my MacBook while using Apple's latest accessibility features for head and facial gestures for mouse control.  All for free and no insurance involved ;-) More about that in a minute.

What I had discovered was that all of the access solutions (from Tobii, EyeGaze, etc...) are Microsoft Windows specific (not counting tablet speech generation devices) and require special software installation. However, I need to control Windows, MacOS, Linux, and Unix systems and sometimes in embedded or appliance form factors. These companies do have a partial solution in the form of a USB "dongle" that presents itself as a keyboard and mouse and receives input via blue tooth from the respective eyegaze system setup. The interface is clumsy due to the dual controlling and controlled screen nature. Having to use one screen to issue mouse movement via button click and then checking the other screen for progress and then back to the controlling screen to stop the movement is awkward and slow. I asked a product representative why they didn’t combine the camera directly with the USB device, but I wasn’t speaking to an engineer and so got an uncertain answer.

I've hacked many devices over the years to impersonate keyboards, mice and other USB devices.  I started to envision a Raspberry Pi (low cost single board computer) combined with a camera module to act as a keyboard and mouse HID device.  I would use the OpenCV and Dlib open source libraries for computer vision and machine learning to perform face detection.  Then I could translate head movement and facial gestures directly into HID reports to the host system for mouse movement and keyboard control (see video below).  I have since built a prototype with a Raspberry Pi 4B ($35) and the Pi Camera V2 Noir module ($25).  I prototyped the code in Python and have successfully been able to manipulate mouse movement on any host computer that accepts a USB keyboard and mouse.  The camera is mounted to the Pi and so only needs to be pointed at my head but does not require a second screen for control.  My attention is only on the system I am controlling.

I hope to optimize the code enough to downsize to a Raspberry Pi Zero ($5) form factor making the low cost solution including camera just $30. All of my code will be free and open source.  In addition, I see potential for converting the head-mouse feature into a true eye-gaze solution utilizing the same open source computer vision libraries along with the companion infra-red camera for performing pupil tracking.  Additionally, it has the potential to interface with common gaming consoles (PlayStation,  Xbox, Switch, etc...) opening up the options for kids and adults who have little access to these platforms.

I shared these ideas and others with the therapists at Perlman along with my consent to connect me with any families they thought I might be able to help. They were very enthusiastic and started introductions to some families open to the idea. As I continued to develop the low-cost head mouse, I started wondering about how to turn the prototype into a product to benefit families.  This is where organizations like Team Gleason came to mind, to help finalize the solution and determine whether this does satisfy a need that families have."

Facial Detection Testing