A voice-controlled browser built on typed decisions

I’ve been experimenting with TypeSafe’s Jev after getting early access, and ended up building a small voice-controlled browser just to see what it could actually do.

I talk. It navigates. It clicks links, searches sites, fills forms, and answers yes/no and either/or questions about the open page.

The interesting part is what happens underneath.

Jev doesn’t return a paragraph of generated text for my application to interpret. It returns a typed decision from a set of choices I define, along with probabilities.

For every request the browser snapshots the page (links, buttons, form fields, visible text) and asks Jev several questions in parallel: is this a yes/no question, is this an either/or, what should I click etc.

Because the application defines the possible choices, Jev can’t hallucinate.

Ask "was it Kevin or William Jevons?" and the application gets back William Jevons at 88%. Not a paragraph that I then have to parse.

And when confidence is low, the application can ask for clarification instead of blindly taking an action.

The entire testing session cost me just $0.0091.

This was just a small experiment. I’m looking forward to pushing it much further.