Scrolling with voice commands has always felt clunky. If you scroll a little at a time, it’s tiring to issue repeated commands. If you scroll a larger amount, at best it’s hard to visually track the content, and at worst you overshoot and skip some content. Fortunately, there’s a better way. Over the last couple months, I’ve been honing a voice-native scrolling system in talon-gaze-ocr that addresses all of these problems. It introduces scrolling commands that scroll a predictable fraction of the viewport, no matter its size, and it shows where the content shifted to with a subtle indicator. And like other commands in talon-gaze-ocr, it is based on computer vision so it works across all platforms in any app that responds predictably to the scroll wheel, even games and remote desktop windows.

It’s easy to get started after you’ve synced talon-gaze-ocr (either main or beta branch will work). The enhanced scrolling commands are invoked with “eye scroll down” or “eye scroll up”. (I personally rebind these to “scroll down” and “scroll up”, but this conflicts with the talonhub/community commands for paging up and down, which I rename to “screen up” and “screen down”.) These will scroll whichever viewport you are looking at by 80% of its height. They finish by drawing a quickly fading blue line at the bottom (or top) of the content that shifted, to help you quickly find your place.

On macOS, this generally just works because scrolling behavior is consistent across apps. On Windows and Linux, some apps add a smooth scrolling animation that can confuse the scroll detection algorithm if it runs too early. By default, I compensate for this with a long delay before running scroll detection in browsers on Windows and Linux. Fortunately, both Chrome and Firefox make it easy to disable this behavior, so you can follow the instructions in the readme to remove the delay. In general, if you see that scroll detection is not working in a particular app, try increasing the user.ocr_scroll_wait_ms setting.

Let’s break down how these commands work so you can customize the behavior:

1. After a scrolling command is recognized, the cursor moves to where you are looking via the action move_cursor_to_gaze_point. This is slightly offset based on the scroll direction, to avoid accidentally scrolling the wrong region near the edge of the viewport. If you don’t have an eye tracker connected, by default the cursor won’t move. This can be configured to move to the active window center with user.ocr_cursor_behavior_when_no_eye_tracker = "ACTIVE_WINDOW_CENTER".
2. Next, the action enhanced_scroll_down or enhanced_scroll_up is called. These are drop-in replacements for mouse_scroll_down and mouse_scroll_up from talonhub/community, but instead of the input specifying a fixed amount of scrolling, it is proportional to the viewport. By default, an input of 1.0 scrolls 80% of the viewport, 0.5 scrolls 40% of the viewport, and so forth. This can be configured with user.ocr_scroll_viewport_fraction.
3. Once the scrolling is complete, a line is drawn to show where the bottom (or top, if scrolling up) of the content has shifted to. The appearance can be configured with user.ocr_scroll_indicator_color and user.ocr_scroll_indicator_fade_seconds.

Under the hood, here’s how the enhanced scrolling works:

1. A screenshot is taken before scrolling.
2. A small “probe” scroll is issued.
3. A second screenshot is taken and compared with the first to determine the viewport bounds and distance scrolled (see below for algorithm details).
4. A second scroll is issued to finish scrolling the requested amount, calibrated based on the detected probe scroll distance. This approach automatically handles viewports with arbitrary size and differing scroll sensitivity.
5. A final screenshot is taken and scroll distance detection is run again to confirm the scrolled amount. This is necessary when the bottom of the content is reached during scrolling, which can’t be predicted in advance.
6. The scroll indicator animation is drawn relative to the viewport bounds, based on the total detected scroll distance.

The viewport bounds detection and scrolling calibration follows a 3-phase bespoke algorithm implemented in pure NumPy that is fast enough that most of the cost comes from converting the screenshots to grayscale:

1. Viewport estimation: The before and after screenshots are diffed, and then searched for a large region with lots of changed content. This is the initial viewport estimate.
2. Scroll distance detection: The before and after screenshots are cropped based on the viewport estimate. Different potential scrolling offsets are compared to find the distance that results in the highest normalized cross-correlation. Raw pixel matching is too sensitive to small rendering changes, so vertical gradients are extracted first to emphasize structural edges. These are divided into vertical strips and summed horizontally to improve efficiency.
3. Viewport refinement: The viewport estimate is refined by aligning the before and after screenshots using detected scrolling distance and searching for matching pixels.

This enhanced scrolling system has noticeably improved my focus when reading, as it enables me to scroll efficiently without losing my place. Please let me know what you think in the comments!