Video unavailable. Open the source below.
NewJev Pointer Companion
A Mac pointer experiment uses local OmniParser/CoreML detection and sends text labels to Jev to choose the next click.
Added to Jevfast
How it uses Jev
A local CoreML model segments screen controls and on-device OCR reads labels. Jev receives the text labels, returns probabilities over elements and identifies a click target; the app clicks, detects again, and repeats.
App workflow
- Detect controls
A local CoreML model segments buttons and other UI elements.
- Read labels
On-device OCR extracts text labels, which are sent to Jev.
- Choose target
Jev returns probabilities over elements and identifies the best click.
- Click and repeat
The app clicks, runs detection again, and repeats until the goal is done.
Why it is interesting
The loop separates local visual parsing from text-based target choice, and iterates after each click until the goal is done, according to the post.