I wrote up some ideas about "aQuery -- Like jQuery for Accessibility", which RobotJS would be very useful for implementing. It refers to the Mac accessibility API but it could work with any platform, and even abstract the differences between platforms just like jQuery does.
aQuery -- like jQuery, but for selecting, querying and manipulating Mac app user interfaces via the Accessibility framework and protocols.
So you can write jQuery-like selectors that search for and select Accessibility objects, and then it provides a convenient high level API for doing all kinds of stuff with them. So you can write higher level plugin widgets with aQuery that use HTML with jQuery, or even other types of user interfaces like voice recognition/synthesis, video tracking, augmented reality, web services, etc!
For example, I want to click on a window and it will dynamically configure jQuery Pie Menus with the commands in the menu of a live Mac app. Or make a hypercard-like user interface builder that lets people drag buttons or commands out of Mac apps into their own stacks, and make special purpose simplified guis for controlling and integrating Mac apps.
[...]
aQuery could apply the DOM tree searching and traversal and data association stuff to the Accesibility Tree, which is similar in a lot of ways to a DOM tree, and describes all the widgets and user accessible affordances and commands in an app, as well as non-tree-like relationships between them (this label describes that widget, this tab represents that panel, this icon represents that view, this editor manipulates that object, etc).
[...]
aQuery should provide ways of registering patterns and calling handlers when user interface items that match them are created and destroyed. jQuery doesn't directly provide a way to do that (handling page onload events and XHR request responses is usually sufficient), but of course there is a jQuery plug-in that does it: https://code.google.com/p/mutation-summary/ .
So when some user interface objects you're interested in controlling come into existence, you can wrap them with your own "widget" to glue them into whatever other user interface you want to provide. (pie menus, hyperlook, ar, speech recognition, etc).
[...]
I think aQuery should be independent of jQuery, but I like to use jQuery as a metaphor for how it works, even though that might suggest that it's tied to jQuery, or even HTML, which it shouldn't be.
I wrote up some ideas about "aQuery -- Like jQuery for Accessibility", which RobotJS would be very useful for implementing. It refers to the Mac accessibility API but it could work with any platform, and even abstract the differences between platforms just like jQuery does.
http://www.donhopkins.com/mediawiki/index.php/AQuery
Also, Morgan Dixon did some wonderful stuff with Prefab: The Pixel-Based Reverse Engineering Toolkit, which would be great to integrate into RobotJS.
http://homes.cs.washington.edu/~mdixon/research/prefab/
aQuery -- like jQuery, but for selecting, querying and manipulating Mac app user interfaces via the Accessibility framework and protocols.
So you can write jQuery-like selectors that search for and select Accessibility objects, and then it provides a convenient high level API for doing all kinds of stuff with them. So you can write higher level plugin widgets with aQuery that use HTML with jQuery, or even other types of user interfaces like voice recognition/synthesis, video tracking, augmented reality, web services, etc!
For example, I want to click on a window and it will dynamically configure jQuery Pie Menus with the commands in the menu of a live Mac app. Or make a hypercard-like user interface builder that lets people drag buttons or commands out of Mac apps into their own stacks, and make special purpose simplified guis for controlling and integrating Mac apps.
[...]
aQuery could apply the DOM tree searching and traversal and data association stuff to the Accesibility Tree, which is similar in a lot of ways to a DOM tree, and describes all the widgets and user accessible affordances and commands in an app, as well as non-tree-like relationships between them (this label describes that widget, this tab represents that panel, this icon represents that view, this editor manipulates that object, etc).
[...]
aQuery should provide ways of registering patterns and calling handlers when user interface items that match them are created and destroyed. jQuery doesn't directly provide a way to do that (handling page onload events and XHR request responses is usually sufficient), but of course there is a jQuery plug-in that does it: https://code.google.com/p/mutation-summary/ .
So when some user interface objects you're interested in controlling come into existence, you can wrap them with your own "widget" to glue them into whatever other user interface you want to provide. (pie menus, hyperlook, ar, speech recognition, etc).
[...]
I think aQuery should be independent of jQuery, but I like to use jQuery as a metaphor for how it works, even though that might suggest that it's tied to jQuery, or even HTML, which it shouldn't be.