Update (12/18/2020): I now use a different method to integrate the tracker. See Efficient UI interaction with OCR and Gaze Tracking for the most up-to-date instructions.
You can do a lot just using your voice, but there are still a few times you’ll find yourself reaching for a mouse. It’s often for the silliest little things, like clicking in the empty space within a webpage to change the keyboard context. If you’re serious about not using your hands, you can use an eye tracker to eliminate these last few cases. This post will teach you how to get started. Make sure you’ve read my introductory post on voice coding, since will be building upon that.
Eye trackers used to cost several thousand dollars, but now you can grab a cheap one for less than a couple hundred bucks. I use the Tobii Eye Tracker 4C, which retails right now for $169. I have a full review of it posted here.
The basic idea behind eye tracker interaction is that you look somewhere on the screen and then use some other method for clicking or “activating” the item you’re looking at. It’s generally too distracting to have the pointer follow wherever you’re looking, so usually a keypress instead of a click is used. For our purposes, of course, we’ll want to use a voice command.
The tricky part is integrating it with Dragonfly. It really ought to be easy, except that right now there’s an outstanding bug where their software does not listen for virtual keypresses. There’s a thread in their forums complaining about this, but it sounds like it won’t be fixed until the consumer version, which doesn’t have a release date yet. The workaround is surprisingly elaborate, but the good news is I’ve already done the heavy lifting. The basic idea is that we will call into their C API from Python. The raw API is extremely complicated for our needs, so I wrote a simple wrapper DLL with a few basic functions to connect to the eye tracker, get position data, and activate the current gaze point. You can get the source code and binary distribution of the wrapper from my github repository.
Python makes it a breeze to call into a DLL. Load the DLL with the following lines:
from ctypes import *
eyex_dll = CDLL(DLL_DIRECTORY + "/Tobii.EyeX.Client.dll")
tracker_dll = CDLL(DLL_DIRECTORY + "/Tracker.dll")
Then you can define some simple wrapper functions that call the DLL functions:
def connect():
result = tracker_dll.connect()
print "connect: %d" % result
def disconnect():
result = tracker_dll.disconnect()
print "disconnect: %d" % result
def get_position():
x = c_double()
y = c_double()
tracker_dll.last_position(byref(x), byref(y))
return (x.value, y.value)
def print_position():
print "(%f, %f)" % get_position()
def move_to_position():
position = get_position()
Mouse("[%d, %d]" % (max(0, int(position[0])), max(0, int(position[1])))).execute()
def activate_position():
tracker_dll.activate()
With these in place, it’s easy to bind them to voice commands using the handy dragonfly Function action. It’s useful to have separate commands for moving the pointer and clicking, because the eye tracker accuracy isn’t always perfect.
Foot pedals are another alternative to voice commands. I often use a voice command to move the mouse based on my gaze point, then use my foot to click. I recommend the Omnipedal Quad. These are also great for scrolling, which is pretty awkward with dictation.
There’s a lot more you can do with a tighter integration with the eye tracking API. The major shortcoming of my simple approach is that it doesn’t work well with small click targets. The full API lets the application describe all the click targets, so the closest one will be automatically picked. Of course, this usually requires access to the application source code (or at least an extension), so it’s less generic and harder to get up and running. Please post in the comments if you come up with something!
Hi James,
My name is Tim and I am the user on the tobii forums that started the post concerning voice integration that we both have been updating. I’m very interested in getting this working with VAC but dont have the coding prowess that you do to do so. I read through your blog and DLed the tracker.dll from github and would greatly appreciate any help you could give in getting the above wrapper to listen for keypresses. Can this be done with an elseif statement or a while loop to trigger off the two or three keys that we want with sendinput or something similar? I tried to use AHK to send the virtual key code, the hardware scan code, and key down key up commands but it still wont trigger from Tobii’s prebuilt app. If i use sendplay instead of sendinput it bypasses tobii and still sends the key through voice commands but at the same time doesnt trigger at a low enough level for what im guessing would be tobii’s keyboard hook to see it and trigger the mouse movement.
Thanks so much for any help!
Tim
Hi Tim, welcome to the site! I don’t think I’m familiar with the voice command software you’re referring to, but I’ll do my best to help. First, the key thing to understand is that my wrapper does not make software keypresses suddenly start working. Instead, it offers a few simple functions you can call *instead* of the software keypress functions. You can see the list of these functions in the header file: https://github.com/wolfmanstout/simple-eye-tracker/blob/master/Tracker.h
Hence, it requires that your voice command software be able to call into a DLL. This definitely works in Python using Dragonfly with Dragon NaturallySpeaking, but the process is going to be different if you’re using a different environment (it should be the same in any other Python environment, though). If your voice command software can only send keypresses, but cannot execute arbitrary code, then I’m afraid my solution won’t work. Hopefully that is not the case here!
James,
I checked per your suggestion and confirmed VAC can’t call into a DLL. So I have downloaded python and dragonfly and watched a few tutorials on dragonfly commands in hopes of giving that a try instead.
So far I have copied your commands above into a .py file and modified them with the path of the two DLLs below. I read that since backslash is used in windows path’s that I need a second backslash as a terminator.
————————–
Traceback (most recent call last):
File “C:Python27Libsite-packagespythonwinpywinframeworkscriptutils.py”, line 326, in RunScript
exec codeObject in __main__.__dict__
File “C:Python27Scriptseyetracker.py”, line 2, in
ctypes.CDLL(“c:\Program Files (x86)\Tobii\Tobii EyeX\Tobii.EyeX.Client.dll”)
File “C:Python27libctypes__init__.py”, line 365, in __init__
self._handle = _dlopen(self._name, mode)
WindowsError: [Error 127] The specified procedure could not be found
—————————
import ctypes
ctypes.CDLL(“c:\Program Files (x86)\Tobii\Tobii EyeX\Tobii.EyeX.Client.dll”)
ctypes.CDLL(“c:\Program Files (x86)\Tobii\Tobii EyeX\Tracker.dll”)
def connect():
result = tracker_dll.connect()
print “connect: %d” % result
def disconnect():
result = tracker_dll.disconnect()
print “disconnect: %d” % result
def get_position():
x = c_double()
y = c_double()
tracker_dll.last_position(byref(x), byref(y))
return (x.value, y.value)
def print_position():
print “(%f, %f)” % get_position()
def move_to_position():
position = get_position()
Mouse(“[%d, %d]” % (max(0, int(position[0])), max(0, int(position[1])))).execute()
def activate_position():
tracker_dll.activate()
Then after I can get this working, I assume I need to map the actions. I found a tutorial for mapping keys like this
class KeystrokeRule(MappingRule):
mapping = {
“enter []” : Key (“enter:%(n)d”),
And tried to apply it to the function commands:
class KeystrokeRule(MappingRule):
mapping = {
“connect tracker” :def connect(),
or
“connect tracker” :tracker_dll.connect(),
“disconnect tracker” :
“move” :
“click” :
If you have any suggestions on the dll path error and/or mappings once again your help is greatly appreciated.
Thanks again so much for your help,
Tim
You are really close to getting this to work. The DLL error bit me first too, there are couple things to double check:
1) You are right that backslashes would need to be escaped, but note that in my code I use forward slashes which are perfectly legal in Windows Python and are less prone to escaping errors. Try using that.
2) Try opening the tracker DLL in dependency walker, and it will tell you what dependencies you are missing. I will save you the Google search and tell you you probably need to download this: http://www.microsoft.com/en-us/download/details.aspx?id=30679. I believe you only need the 32-bit version, but it wouldn’t hurt to install both if you have a 64-bit processor.
Next, you need to use a slightly different syntax for declaring these macros. Here’s what it looks like:
“(I|eye) connect”: Function(connect),
“(I|eye) disconnect”: Function(disconnect),
“(I|eye) print position”: Function(print_position),
Also, make sure other dragonfly macros in general are working before trying to get this working.
I’m going to post my dragonfly macros within the next couple weeks (hopefully the next couple days), although they will contain a bunch of stuff you won’t want, so it is still worth figuring this out on your own.
Thanks for powering through this, I promise it’ll be sweet when you get there!
Hello James, thank you for sharing your solution, I was little disappointed when I found out that EyeX doesn’t handle virtual keystrokes at the moment.
I cannot work with Dragonfly, since my native language isn’t English, but I have different software that can do various stuff. One of them is running an arbitrary executable file on a voice command.
Would it be, in principal, possible to create a python script, which would call functions connect() and activate_position() directly?
So far i didn’t have much luck with this – using your example, I end with error messages like
result = tracker_dll.connect()
NameError: global name ‘tracker_dll’ is not defined
OR
NameError: global name ‘c_double’ is not defined
I’m not very skilled in Python, but I think I can get the basics in couple of days – the main question is, whether the solution above can be realized using your DLL.
Hi Michal,
My first snippet of Python code was broken; please try again now that it is fixed. I guess I must’ve mixed up a couple versions of the code.
I’m not sure how well your approach will work exactly as described. The trouble is that as soon as the Python process exits, the eye tracker context will be destroyed and disconnect. There are couple ways of solving this. The quick and dirty way would be to create a script that does everything you want in one go (connect, activate position, then quit). The problem is this is likely to be slow due to reconnecting each time. A cleaner approach would be to create a long-running Python server that you start separately one time and then have your executable file just send a request to your server. You would just need to pick some interprocess communication mechanism to send and receive the request.
Another approach is to make the long-running Python process do the work of listening for virtual keypresses and then triggering actions. It sounds like that’s what most folks really want.
Thank you, James. I think that your last suggestion is the best one. I was able to make it running very fast despite the fact that I’ve never worked with Python before. I also use the function Warp sometimes. Do you think it can be controlled by voice, too?
Glad to hear this worked for you! The warp function is not built into the API, but it shouldn’t be too hard to do this yourself using Python libraries to move the mouse pointer, using the function in my DLL to get the last gaze position. I do this using Dragonfly, which is in turn using Python libraries to handle the mouse movements.
Here is the full list of actions currently built into the API, for reference:
I’m looking forward to the zoom functions. I can add support for this when it is available.
I tried one of the more expensive eye tracking solutions a few years ago, and wasn’t very impressed. The natural flickering of my eyes made it nearly impossible to hone in on my targets. However, I didn’t have control of when the cursor moves like you do, and it never occurred to me to separate the motion of the eye tracker cursor from the motion of the system cursor. I read that in your article (where you have a function for moving the cursor) and thought to myself, what a good design choice. It makes me want to give eye tracking a second try.
Thanks, let me know how it goes! Of course, the one downside of separating these actions is that it is slower. But usually I use eye tracking as a last resort, so I’m just happy to have something (and it’s certainly faster than using “mouse grid”).
Don’t you need some kind of hardware to do eye tracking as well? I assume that isn’t free.
Yes, I’m using the Tobii EyeX eye tracker, which is $140. I’ve got a short comparison of options in the second paragraph of my post.
I’m surprised it’s that cheap; I’d heard I trackers were in the $1000 range.
Do you have to calibrate it each time you sit down?
One time calibration is all that is necessary. The accuracy is so-so, though.
Most trackers are way more expensive, even from the same company. I got a promotional email from Tobii encouraging me to step up to their next best eye tracker… at only 20x the price 🙂
Just wanted to shout out, Thanks! For writing this up and making that DLL wrapper. It took me a while to get it working with my limited experience, but now I can voice warp too!
Hm. This is unrelated maybe, but sometimes I’m able to use the Tobii EyeX – Pointer Interaction – Warp on mouse movement, to move scroll bars and windows when I touch the mouse, when usually it does not. But, I haven’t isolated what causes it to start working… I find it useful actually, and wondered if you’ve run into it? In case it’s something the dll accidently causes.
Thanks for the shout out 🙂
I haven’t ever noticed mouse warping. I’d be really surprised if my DLL caused that; it shouldn’t. If you find it useful though, I suppose you can just enable it in the settings permanently.
Hi James. Thanks alot for making this publicly available.
Did you have to update/change anything when you changed to 4C? I just bought a 4C and trying to make your hack work. but nothing really happens. At the moment I’m just using the precompiled DLL found at your “releases” in your github, and then I found the Tobii.EyeX.Client. I placed the two files alone in a directory which I fed to the “local.py” file.
when I load natlink no tracker errors appears, and the functions seem to work i.e. it recognizes the commands “eye connect”, “eye print position” and “eye move “. However there are no reaction to them, the cursor doesnt move and nothing is printed. Do you have an idea of what could be the cause?
If i have to compile a DLL myself: I cant find a “sample” folder in my Tobii SDK ( i guess that is just the folder called “Tobii” installed per default in the program-files directory?
I think I did have to rebuild this at some point a while back. I’ve released the version that I’m currently using: https://github.com/wolfmanstout/simple-eye-tracker/releases/tag/v0.2
Let me know if you still have trouble.
Splendid, the few commands i tried works flawless. Thanks alot.
Hello James, first of all I wanted to say I really like your work. But I have one problem with it because I use a recent version of the EYEX Engine I get this error TX_CONNECTIONSTATE_SERVERVERSIONTOOHIGH. So could you pleas get in contact with me so I can send you my project and you can take a look at it. Thank you in advance and I hope to hear of you soon.
I’ll take a look this weekend using my own setup. Presumably if I upgrade Tobii EyeX I should be able to reproduce this. I’m guessing I simply need to rebuild the DLL using the latest API from Tobii — it has been a long time.
Thank you so much! Sorry to bother you with this but I tried everything and could not get it to work. What I did was build it with an old version of the EYEX Engine because that one contains a couple of files you need to get the tracker.dll to compile like CopyEyeXDllToOutputDirectory.target, so I could not try it with a new version. Again sorry to bother you and thank you in advance, I will be waiting on your response.
I’m not able to reproduce this on my own machine. I tried updating the drivers, and even uninstalled and reinstalled them. I’m using version 2.16.4. I can try rebuilding the DLL but it would be very helpful if I could reproduce this first. Some questions for you:
1) Which version of my DLL are you using? I did a second release in January 2018: https://github.com/wolfmanstout/simple-eye-tracker/releases
2) What version of Tobii software are you using? What hardware? This is tested on the 4C.
3) Has this ever been working for you? Was this a recent breakage?
4) When do you see this issue?
5) Anything else that is unusual about your setup that might be relevant?
Thanks!
Hello
I’ve been trying to get this to work but I keep getting this error:
connect: 1
(0.000000, 0.000000)
The connection state is now TRYINGTOCONNECT (We are trying to connect to the EyeX Engine)
My code looks like this:
from ctypes import *
eyex_dll = CDLL(“C:\Program Files (x86)\Tobii\Tobii Screentime\Tobii.EyeX.Client.dll”)
tracker_dll = CDLL(“C:\Program Files (x86)\Tobii\Tobii EyeX\Tracker.dll”)
def connect():
result = tracker_dll.connect()
print “connect: %d” % result
def disconnect():
result = tracker_dll.disconnect()
print “disconnect: %d” % result
def get_position():
x = c_double()
y = c_double()
tracker_dll.last_position(byref(x), byref(y))
return (x.value, y.value)
def print_position():
print “(%f, %f)” % get_position()
def move_to_position():
position = get_position()
# Mouse(“[%d, %d]” % (max(0, int(position[0])), max(0, int(position[1])))).execute()
def activate_position():
tracker_dll.activate()
connect()
print_position()
##########
I would be grateful for any help.
I have rewritten my eye tracker integration to leverage newer .NET APIs because the old DLL is no longer supported. I have checked this in and you will find instructions in my readme: https://github.com/wolfmanstout/dragonfly-commands/blob/master/README.md
It’s actually a nicer integration because there’s no intermediate DLL needed; it’s easy enough to work with the APIs directly from Python. Note that if you have trouble with the eye tracking lagging behind, you may want to add this hack to your grammar: https://github.com/wolfmanstout/dragonfly-commands/blob/de7ecb27e9c6d906a81560abdd7e00357efec6dd/_repeat.py#L2263
Thanks for the response. I followed the instructions, copied the contents of _eye_tracker_utils.py into a new Python project and added
tracker = Tracker()
tracker.connect()
tracker.print_position()
The result is:
Tracker connected.
Process finished with exit code 0
So it fails to get the position. The DLLs are unblocked in the same directory. What could be the problem?
Sounds like you are testing this from a standalone script and not from within a NatLink grammar? This may cause problems with the process exiting prematurely — I’ve seen similar issues in the native C# scripts. Do you still see this problem if you execute these functions from within a grammar?
hey, do you know if this still works with Tobii eye tracker 5?
https://help.tobii.com/hc/en-us/articles/360008539058
Hi, I just ordered this new eye tracker so I can’t confirm whether it works or not. I no longer use the integration method described here (I have a deprecation notice at the top of this post). This old API is no longer supported by Tobii, and I’m now using their .NET DLLs directly through Python:
https://github.com/wolfmanstout/gaze-ocr/blob/master/gaze_ocr/eye_tracking.py
I plan to expose this as a PyPI package (and write a blog entry about it) but I’m still finalizing it. If you’d like to try it out before then, here are the installation instructions: https://github.com/wolfmanstout/gaze-ocr
You don’t actually need to install Tesseract if you just plan to use the eye tracking module. I expect that to be compatible with their new eye tracker, but we shall see.
Thank you for your reply.
I would be happy if you would report about the new version. I wanted to buy the Tobii Eye Tracker 4c, but now I am unsure if everything works with the new version.
Just arrived today, and it worked perfectly with my voice setup right after plugging in. Please do bear in mind that I’m using the new integration I described above, not what was originally described in this blog post.
Looks like the accuracy might be slightly better, but it’s hard to say because I also just recalibrated.
That’s great. Just ordered my Tracker. Hopefully I will get it work with your new Integration. Thanks!
The device arrived today. I was able to integrate your components very well into my system. (I only had to install scipy later.) Thank you for that! So far I’ve only tested the move_to_gaze_point functions.
I am using a 15.6 inch notebook screen with a 1080p resolution. I noticed that the accuracy is not the best in some places. How do you deal with it? Small icons in toolbars are particularly difficult to hit. Do you call the command several times until you reach the desired point? In the past, I had a normal mouse under the table and was able to control it very precisely when I reduced the sensitivity. Unfortunately, this type of control is too much for the feet.
Yes, this is expected: eye tracker accuracy is not good enough to use as a mouse. Stay tuned for an upcoming blog post where I discuss how I work around this.
Can you elaborate on needing scipy? I don’t think that’s a dependency of the code I shared — please let me know if I’m missing that somewhere.
It is apparently used by skimage. I have just removed scipy and then I get the following message:
Exception has occurred: ImportError
No module named scipy
File “C: \ Data \ Python \ screen-ocr \ screen_ocr \ _screen_ocr.py”, line 8, in
from skimage import filters, morphology, transform
File “C: \ Data \ Python \ screen-ocr \ screen_ocr \ __ init__.py”, line 1, in
Got it, thanks. Looks like that’s a declared dependency of scikit-image so if you install from PyPI it should pick it up automatically. There are quite a few transitive dependencies for the libraries I’m using.