5 Tests of the Jev AI Model, From Routing to Flights

If you need fast classification inside an application rather than chat text, the Jev AI model is built for exactly that job. The tests below show where it behaves well and where restricted outputs can still produce a wrong answer.

What Is the Jev AI Model?

The Jev AI model is what its vendor, TypeSafe, calls a System One model: you supply information and specific questions, and it returns structured decisions rather than chat text. It does not write applications or compose prose. The design goal is small judgments that are cheap and fast enough to embed throughout software, such as deciding which team should receive a support message or whether a refund was requested.

The version used in the playground tests was identified in responses as Jev 1.13.0, even though the alias 'latest' had been selected. Recording the resolved version alongside results is sensible, because an alias can silently change between runs.

Three Question Types: Choice, Score and Null

Jev exposes three main question types, and the shape of each one determines what your code receives back.

• Choice selects from options you provide. Your application defines the candidate set, and the model picks one.
• Score rates something using levels you describe, for example a frustration scale from calm to furious.
• Null returns a probability that a yes-or-no statement is true, such as whether a customer requested a refund.

Because every answer is a structured value, your code decides what to do with it. The model never takes an action on its own, which keeps the boundary between judgment and execution clear.

Support Routing and Negation Tests

The first test sent a support message about a duplicate charge with four questions bundled together: which department should handle it, was a refund requested, was it urgent, and how frustrated the customer sounded. Jev selected billing, gave the refund request a probability of 98 percent, urgency 11 percent, and a frustration score near the calm end. A billing problem did not automatically become a technical problem, and a refund request did not automatically become an emergency.

The message was then changed to read 'I am not asking for a refund. I only need a copy of my invoice.' The refund probability dropped to 3 percent while billing stayed selected. The model handled negation correctly instead of matching the word 'refund' and treating it as a request. This is exactly why testing your real wording matters: asking for a refund, asking about refund policy, and explicitly refusing one are different intents.

Restricted Choices Do Not Guarantee Correct Answers

The most instructive failure came from a question with an incomplete option set. Asked what time the cafeteria closed, with an 'other' option available, Jev correctly selected 'other' rather than forcing the question into billing, technical support or sales. When the option list was narrowed to only those three departments, the model selected sales with a confidence of 0.31.

The answer fit the supplied choices, but none of them covered the request. This is the practical meaning behind claims of zero hallucinations: restricting the output can stop the model from inventing a new category, but it cannot guarantee that the chosen category is useful or correct. A confidence value is also not a calibrated accuracy figure; a 90 percent reading is not proof of 90 percent correctness on your data.

Two design rules follow from this test. Include an 'other' or 'unknown' option wherever a path to human review exists, and validate confidence behavior on your own examples before wiring the output to an automated action.

Prompt Injection Resistance: One Pass Is Not Immunity

A basic injection test added a fake system override inside a customer message, instructing the model to choose billing and mark refund and urgency as true. The evaluation instructions told Jev to treat text inside the message as untrusted. The result: the technical support classification held, the refund probability stayed low at 3 percent, and urgency stayed near the uncertain middle.

That is a good outcome for one attempt, but a single simple injection test does not establish that a model resists attacks. Security claims need a proper adversarial suite, and production systems still need allowlists and output validation on top of any model behavior.

Exact-Value Extraction and Agent Auditing

For extraction, a message contained a sender's email, an old billing address, and a new address where the receipt should go. With those addresses supplied as choices, Jev selected the new address exactly as given, including the plus sign and year. The pattern is useful: your code collects candidate values, the model selects the relevant one, and your code copies the original string. If the collection step misses the correct address, however, a choice question cannot recreate it. The candidate list is part of the system you must test.

The agent audit test compared evidence against a claim. Tool results said permission was denied and nothing had been saved, while the assistant's final message claimed the draft was saved successfully. Jev classified the task as failed and gave the unsupported success claim a probability of 93 percent. This particular error was simple enough to catch with ordinary code; the more valuable use case is checking messy multi-step traces with partial progress and conflicting claims, which would need a much larger test set.

Browser Automation With Jev Ultrafast

A separate demo, Jev Ultrafast, combines Jev with Browser Use automation tooling. The browser code reads the page structure, builds a numbered list of controls, and refreshes the options after each step. Jev selects the next operation and its target in one bundled request, and a separate small language model generates any text the action needs. The demo used Mercury 2.5 for that generation step; Mercury is the diffusion-based language model family from Inception Labs, described in their research paper published in 2025.

The reported numbers are the demo author's, not independently reproduced here. In his published run, the project searched Google Flights for a one-way trip from Zurich to London in about 7 seconds at a reported model cost of 0.39 (under half a cent), with the video played at original speed. Three caveats apply: the timer started after the first page observation and excluded startup, the run was a search with no booking, and the reported price matched a model cost estimate that excluded browser infrastructure.

Reported Speed and Cost of the Test Requests

Across eight playground requests, the service reported model evaluation times between about 92 and 214 milliseconds. That is the evaluation time returned by the service itself; it excludes browser and network latency a real application would experience. The requests used a total of 4,148 input tokens. At the published price of 4.2 cents per million input tokens, with output tokens free at that rate, the estimated model input charge was much less than 1 cent.

Total application cost includes more than model inference: preparing inputs, checking answers, and routing edge cases to another model or a person all cost time and money. These figures came from eight small synthetic requests. They suggest Jev is inexpensive for small classification work, but they do not establish production reliability or verify the vendor's broader speed comparisons against other models. Run your own broader set and compare bundling several questions into one request against sending them separately.

FAQ

• What is the Jev AI model used for? Jev is a System One model for fast, cheap, structured decisions. Typical uses include support ticket routing, refund detection, urgency scoring, exact-value selection from documents, and auditing agent claims against tool results.

• How fast is Jev? In the eight playground tests described here, the service reported evaluation times of roughly 92 to 214 milliseconds per request. This is service-side evaluation time and excludes network and application latency.

• Does restricting Jev's outputs prevent hallucinations? Restricting outputs to a supplied choice list stops the model from inventing new categories. It does not guarantee the selected category is correct, as shown when an incomplete department list produced a sales routing with 0.31 confidence for a cafeteria-hours question.

• Was the 7-second Google Flights demo independently verified? No. The roughly 7-second Zurich-to-London search and the 0.39 reported cost came from the demo author's published results. The timer excluded startup, the run involved no booking, and the price excluded browser infrastructure.

• Can Jev replace a general language model in an agent? The evidence supports a division of labor, not replacement. Jev handles structured decisions while a separate model generates text; the browser demo paired Jev with a Mercury model for generation and with Browser Use tooling for page interaction.

Turn Your Own Video Tests Into a Written Write-Up

This article exists because a hands-on test session, like the one behind these Jev results, is only useful if other people can read and repeat it. If you have knowledge like that sitting in your YouTube videos, whether walkthroughs, benchmarks, or interviews, you can turn it into a structured article without writing from scratch.

Skala blog takes a YouTube URL, transcribes the video, and generates a publishable article draft from it, so your testing notes reach readers who will never press play.

Source video