Hi, I’m James, and I created handsfreecoding.org to share techniques and software that allow me to code and enjoy my computer without using my hands. There are a lot of powerful tools and libraries out there, but it can be overwhelming to learn how to put all the pieces together. I’ve been at it since 2012 and I’m still discovering new ideas, so I hope this blog will be useful to newcomers and experts alike. To read more about my story, check out my first post, Adventures in Hands-Free Coding.
In order to allow the posts on this blog to become usable for open source speech recognition training data, I am using the CC0 license. Please attribute/link to my posts when using them.
To the extent possible under law, the author of this blog (James) has waived all copyright and related or neighboring rights to his posts and pages within this blog, unless noted otherwise. This work is published from the United States.
If you want to send me private feedback, please fill out the following form. If you want to share public feedback that others can comment on, please fill out the form at the bottom of this page.
Hi James,
Thank you so much for this resource.
My little sister is 15 and I started teaching her to code in python last year. However, she has since had chronic pain and cannot type.
I installed dragon for her and tried to use the software you recommend, but ran into issues because I am not familiar with it. She really wants to learn to code because I started teaching her about generative art (she had a successful blog, 6ftaboveground.com but has had to stop due to hand pain).
She’s been trying to use your guide but is having difficulty because she is new to python.
Would you be open to one or two skype sessions with my little sister and/or me to show us the ropes? We would pay you for consulting. If you know anyone else who has emailed you over the years who would be interested in this please let me know.
Thanking you in advance,
Jaan
PhD student, Princeton
jaan.io
Absolutely I would be happy to chat with you and your sister. This is the reason I have the blog, to help people like you and your sister!
Thank you for the offer to pay, but no need for that.
I will reach out to you directly via email on next steps.
Hi,
What would you recommend for only java programming on Eclipse or Intellij? As far as I know it is VoiceCommand?
Has anyone tried a pen tablet eg wacom or handwriting recognition on surface?
I would still personally recommend my own setup to someone who wants to do Java programming. It is designed to work with a wide range of applications. You will just need to set up commands for the various keyboard shortcuts available in Eclipse and IntelliJ.
Lots of folks use pen tablets as an alternative to mouse. It’s not hands-free, though 🙂
Man, your blog is awesome! As a non-programmer thats diving in to dragonfly and subsequently Python, this information is invaluable.
Although trying to understand dragonfly grammars is and has been a daunting task! Right now im trying to set up a grammar for writing the latex markup language using Vim. Looking to do mathematics and because of RSI I’m not able to use a pen for very long. Doing mathematics by voice seems to be a viable alternative.
I’m experimenting with existing dragonfly scripts in some of the github repositories and getting some very inconsistent results. For example i spent a good few hours yesterday trying to get the multiedit grammar in the main dragonfly github repo to work. The format_functions were working fine but the key bindings saved in the comfig.cmd.map were not. And no error was being thrown in the Natlink message window. Anyway I finally got key bindings working using an alternative template script that utilised MappingRule. It’s not ideal though as it seems you need to pause to activate the key binding commands rather than using continuous speech. Anyway I’ve got a lot to learn!
I wonder if you have no of any grammar that would be suitable for working with latex in vim? I’m looking for a template that would make it fairly straightforward to build up my own grammar because as you can guess, my programming skills are limited 🙂 Cheers for the blog!
Thanks for the kind words! I don’t personally have a LaTeX grammar, but I do think you were on the right track starting with multi-edit. My first guess for why your configuration might not be working is that the file is not named correctly. You need your configuration file to have the same name as your grammar file, except with the extension “.txt”. In other words, if your grammar is _multiedit.py, then your configuration should be _multiedit.txt.
If anyone else has a good grammar for LaTeX, please share.
wow you replied! Thanks! I never got a notification of your response and was just checking your blog to research something else when I noticed your reply.
With regard to the multi-edit grammar file indeed you were correct. When the grammar file and the configuration file had the same name it began to work.
I’m still working on progressing with my latex/voice recognition work flow. I have successfully created an app specific grammar file for Virtual box that runs ubuntu. Using ubuntu I can very easily set up Vim with useful plug-ins to make writing latex files by voice much easier e.g. snippets, latex live auto compilation etc.
My Virtual box grammar file is very basic and so far only contains two dictionaries. One dictionary is created using MappingRule class and the other is created using KeyStrokeRule class with MappingRule being the parent class (i hope Im using correct terminology as I just recently learned about classes and object oriented concepts in general to aid my understanding of Dragonfly scripts).
A key difference between the two dictionaries is that with the MappingRule dictionary you need to pause between commands whereas with the KeystrokeRule dictionary you can execute commands continuously with no pauses. This concept of continuous commands without pausing is exactly what I’m looking for while inputting into latex. It will be too inefficient if I have to pause for every command I input into latex.
However, KeystrokeRule is limited that it doesn’t seem to execute commands that output text using the Text class e.g. {“square root” : Text(“sqrt()”)}. It only seems to out put Key commands (hence the name of the class i guess 🙂 )It seems that only MappingRule class can output text but then you lose the continuous command atrributel.
I notice your repeat module has continuous commands for all commands but I wouldnt know were to start to try and implement that as it’s so complex!
Is there an easy way to create a dictionary of commands that output text and can be executed continuously?
I’ve been stuck on this for quite a while now and not sure how to solve it.
Cheers
Mike
… actually, after a few hours of trying I finally got your dragonfly setup up and running. With your instructions it wasn’t too bad! Thank you. However I still cant add full words to the dictionaries in your repeat.py file. They seem to be working for commands that output one or two characters only i.e., symbol_map, numbers_map etc.
The main reason I want to add multiple character strings (words 🙂 ) to a dictionary in a grammar that I can perform continuous commands on is because if I just use normal dictation without a dragonfly grammar it invariably adds spaces before words and capitals. By putting the words into a dragonfly grammar it eliminates the spaces and caps without the need for saying ‘no caps, no space” before dictation all the time.
Do you know of any way in which I can achieve this? Sorry for all the questions! Im just very eager to try and find a way to get this working!
I use Python dicts for several different things in my grammar files. In some cases, I will use them directly as commands, and in other cases I put them into a DictList, and then reference these elsewhere in commands. What you discovered is that DictLists can be modified at any time and they will dynamically be available. On the other hand, actual commands need to be finalized before they are loaded into the grammar. That’s why in some cases you are able to make changes after I loaded the grammar, and in other cases you are not.
My recommendation for your use case is to stuff these words into a DictList, and then set up a grammar that references them however you like. I have some support for that already in my grammar, where my custom words are made available in various places. I don’t allow these to be directly dictated, though, but there’s no reason you couldn’t do that.
Hope that helps!
warning! wall of text…apologies but i found following along to your grammars a little complex and wanted to go through it to help my understanding.
Examining your repeat.py grammar, I see you create a dictionary with the keys as the spoken dictation e.g.”zero” and the values are the desired output “0” and assign it to a variable e.g. numbers_map.
You then create a DictList object passing the name of the DictList you’re creating (“numbers_dict_list”) and the name of the desired dictionary (“numbers_map”).
so far so good but This is where it gets a little complicated for me. You then create a new variable (numbers_element) and assign it an instance of RuleWrap passing the name argument as ‘None’, the element argument as an instance of utils.JoinedRepetition which itself is passed an empty string and a DictListRef object passing ‘None’ and the name of the desired DictList ‘numbers_dict_list’. and the final RuleWrap argument is the default giving it ‘max = 10’ or a limit of 10 consecutive commands.
This numbers_element is then referenced by the character_rule which utilises your utils.create_rule passing a string ‘CharacterRule’, the previously created dictionary character_action_map and the 3rd argument being a dictionary associating the keywords in your character_action_map to the different element objects e.g., ‘numerals : numbers_element’
This character_rule variable is then referenced in the extras argument when initialising the RepeatRule which as you say handles the recognitions through its process_recognition method.
The process of writing out the code in plain english like this has actually been quite helpful but reading over it, it is easy to see why a nonprogrammer like myself would have difficulties writing their own dragonfly scripts. especially if they require continuous command recognition. There are class instances embedded within the arguments of other class instances embedded within other class objects which just gets me totally lost!
but by what you mean by using DictLists is I should create a dictionary with my commands and assign it to a variable. e.g. my_commands_map = {‘exercise’ : Text(‘exe’), ‘theorem’ : Text(‘thm’), etc.}.
note that the output of my commands are typically multiple character strings unlike your numbers_map which only had a single character output. I’m not sure if that difference is important.
I would then create a DictList object ‘my_commands_dict_list’ using my_commands_map dictionary.
my_commands_dict_list = DictList(“my_commands_dict_list”, numbers_map)
Now to allow my commands be executed using continuous command recognition following a similar format I would have to use the RuleWrap class to create a ‘my_commands_element’ as in:
my_commands_element = RuleWrap(None, utils.JoinedRepetition(“”, DictListRef(None, my_commands_dict_list), min=0, max=10)).
I would then create a my_commands_rule that utilises the utils.create_rule method. something like:
my_commands_rule = utils.create_rule(“MyCommandRule”, my_commands_action_map, {“my_commands”: my_commands_element,}).
Then add the my_commands_rule to the RepeatRule variable ‘extras’ list:
Alternative(RuleRef(rule= my_commands_rule), name=”nested_repetitions”),…
I will give this a shot. As you can see though thus far im really just trying to copy patterns in other people’s code and seeing if it works. Thanks a million for your feedback none the less. I feel like I am learning a lot from this.
Sorry for the slow response, I was a little intimidated by the wall of text 🙂
The reason you see me passing in “None” into grammar elements is that I’m setting the name of the element to None, because I overwrite that name later on just before it’s used. I’ve basically implemented my own lightweight layer atop Dragonfly which I think is much more flexible. This is part of that system. From an engineering standpoint I’m pretty happy with it, but it’s not the most beginner-friendly, because some of the underlying Dragonfly quirks still shine through. Despite this, you seemed to basically get a good handle on it! Your plan seemed reasonable to me (I didn’t check all the syntax but it seems like you have the basic idea). You have the JoinedRepetition which is going to let you repeat multiple DictList entries in sequence (and yes, these can be full words, or even phrases). The “Joined” part of that is that it outputs the strings all joined together (you would probably want to join them with spaces, not empty strings). And then you introduce this into the grammar as part of nested_repetitions, which is good because it means that we don’t add another excessive layer of repetition support.
Copy and pasting other people’s grammars is a good way of getting started 🙂
Hey James!
Just wanted to say thank you for the super informative site, we really need more resources around hands free coding.
I have been wanting to get started for years but I work exclusively on linux (Kubuntu) so it’s not as simple to get setup.
So I just deal with the pain on a daily basis.
Quick question, I still have dragon 12.5 which was the golden standard… is that still the case or are the new dragon versions worth looking into?
I have windows 10 with dragon 12.5 running in a virtualbox, I just haven’t gotten as far as getting it to pipe out signals / events to my work machine (which is offline, so no internet-reliant speech engines usable here).
cheers,
Jordan
Thanks Jordan! These days I recommend Dragon 15, the latest version. It does have the downsides that I’ve mentioned in other posts, but there are enough improvements in quality, speed, and Windows 10 integration that it’s worth it.
Definitely have a look at my post on using Linux; solutions do exist. The most straightforward add on for the setup you described would be Aenea which creates a server on Linux and operates it via a Windows client connected to Dragon. None of that requires internet access (in fact I would not recommend exposing your Aenea server to the Internet!).
Hey guys I made a speech recognition software for programming by voice. It’s a one click download.
Check it out at http://SpeechToCommand.com
There’s a demo video on the front page.
Hi James,
Thanks for your great site!
I’m planning to buy DNS 15, working with windows 8.1 at the moment. Do you recommend upgrading to Windows 10 or will 8.1 do fine?
Thanks,
Hans
I think 8.1 will work fine although I recommend upgrading to 10 for other reasons (it’s been so long I don’t remember all the reasons. Windows Subsystem for Linux is very cool and I’ve played around with integrating Dragonfly with the Toast notification system. I remember it felt broadly like a major upgrade to finally being a modern OS.)