Out of curiosity, have you looked at Sikuli [1]?

[1]: http://www.sikuli.org/

I hadn't heard of it before, and it does look potentially very powerful.

My only initial concern from reading the quick start (http://www.sikulix.com/quickstart.html) is the apparent requirements to use the "SikuliX IDE" for scripting...

The magic of Sikuli is its use of the OpenCV library under the hood (and Tesseract for OCR). You could skip the Sikuli part and just use OpenCV and Tesseract directly. (Not easy, but theoretically possible.)

I once used sikuli to extract a table from a pdf. Pretty funny way of hacking a fast solution for a problem. It's a pretty powerful tool but not very stable. IDE is also not so exciting but gets the job done. Pretty excited to see how far this project will go.

You actually don't need to use their IDE as the scripts it generates are just python scripts.

I've used it a few times in the past and was able to use an external screenshot program (Apple-Shift-4 on Mac) and a text editor to write the scripts.

Check out Morgan Dixon's awesome research project Prefab: The Pixel-Based Reverse Engineering Toolkit, which would be great to integrate into RobotJS.

http://homes.cs.washington.edu/~mdixon/research/prefab/

Sikuli is awesome, I've used it a bunch. But developing and distributing cross platform commercial applications with it is a mess. Also interacting with other languages, to make a GUI for example, isn't fun.

Autohotkey is a crazy dense macroing tool; I originally thought it was just move the mouse and type, but it can do so, so much more- interface with spreadsheets, web browsers, heck, there's even a tiling window manager out there that's programmed in ahk script. (bug.n)

Yeah AutoHotkey is amazing. I've written many successful apps in AHK. Like this one:

https://github.com/octalmage/mDesktop

With RobotJS and nw.js I've been able to port many of my apps to JavaScript, and ultimately make them cross platform.

As another example, I wrote some scripts (well, hacked together other people's work) to map MIDI CC values to a virtual joystick's Y axis, so I could use my MIDI expression pedal as an accelerator pedal in Euro Truck Simulator 2.

I mostly used AHK for keyboard stuff, so TextExpander would be my Mac equivalent, and within its domain it's a lot nicer and maybe even more powerful than AHK. Never done much mouse-scripting on any platform, though...

http://www.keyboardmaestro.com/main/ Mac Os X option.

I love the features! But the language omg. Every time i want to modify my script it feels like such a chore. Would love some js or lisp that compiles to AHK.