A10X lab · run 2026-09-18
Jev on 3,080 bank support messages
Jev returns a decision and a confidence, never an explanation. So the only question that matters in a deployment is this one: when it says 0.9, how often is it right? We ran it on every message in the Banking77 test set to find out.
The threshold is the deliverable
Pick the confidence above which the decision ships with no human. Everything below it goes to a person. Move it and watch what you trade.
At 0.99 it automates 46.5% of traffic and gets 2.8% of those wrong. At 0.90 it automates 67.8% and gets 7.3% wrong. Which one is right depends on what a wrong route costs this customer, and that is a number you agree with them before launch, not after.
When it says X, how often is it right
Grouped by the confidence Jev reported. If it were perfectly calibrated, the two columns would match.
| Confidence | Messages | Stated (avg) | Actually right | Gap |
|---|---|---|---|---|
| 0.00 to 0.50 | 155 | 41.5% | 36.8% | -4.7 pts |
| 0.50 to 0.70 | 377 | 59.3% | 45.4% | -13.9 pts |
| 0.70 to 0.80 | 196 | 74.5% | 60.7% | -13.8 pts |
| 0.80 to 0.90 | 264 | 85.3% | 67.8% | -17.5 pts |
| 0.90 to 0.95 | 243 | 92.3% | 75.7% | -16.6 pts |
| 0.95 to 0.99 | 413 | 96.8% | 87.2% | -9.6 pts |
| 0.99 to 1.00 | 1,432 | 99.8% | 97.2% | -2.6 pts |
The top bucket holds up: near-certain answers are right 97.2% of the time. The middle does not. Every bucket from 0.50 to 0.99 is overconfident, by 10 to 18 points. On this data the raw number is a good ranking and a bad probability, so measure your own curve on your own traffic before you promise a customer anything.
The confident mistakes
93 messages were wrong at 0.95 confidence or higher. They are not random. They sit on pairs of labels that mean nearly the same thing:
declined_transferread asdeclined_card_payment(8)top_up_by_bank_transfer_chargeread astransfer_fee_charged(5)supported_cards_and_currenciesread astopping_up_by_card(4)order_physical_cardread asget_physical_card(4)wrong_exchange_rate_for_cash_withdrawalread ascash_withdrawal_charge(4)card_linkingread asactivate_my_card(3)
Read some of them yourself:
| Customer message | Jev said | Human label |
|---|---|---|
| I found my card, I would like to reactivate it. | activate_my_card 1.00 | card_linking |
| How can I check the exchange rate applied to my transaction? | exchange_rate 1.00 | card_payment_wrong_exchange_rate |
| My card was declined today when eating and I need to know what's wrong. | declined_card_payment 1.00 | card_not_working |
| I attempted to use my card while I was intoxicated, and I failed to input my PIN, and the machine kept my card. How soon can I have it back? | card_swallowed 1.00 | pin_blocked |
| how long do money transfers take? | transfer_timing 1.00 | transfer_not_received_by_recipient |
| The app wouldn't accept my top up. | top_up_failed 1.00 | top_up_reverted |
| Someone stole my cards! | lost_or_stolen_card 1.00 | lost_or_stolen_phone |
| Good morning. I tried to make a purchase with my credit card last night and again this morning. Both times it was declined. Can you investigate? | declined_card_payment 1.00 | declined_transfer |
In several of these the model looks right and the label looks wrong. "Someone stole my cards!" is labeled as a stolen phone. That is the second lesson: in a real deployment the eval set is part of the system, and a taxonomy with near-duplicate labels will cap any model. The fix is usually a rubric sentence per label, or merging two labels the business never needed to separate. Both are conversations with the customer.
What this does not tell you
- We gave Jev the label names only, no descriptions and no examples. Written criteria per label should move the confused pairs. We have not measured that yet.
- One public dataset, in English, short messages. Your traffic is not Banking77.
- No comparison to other models or to a fine-tuned classifier in this run. The point of Jev is doing it with no training, at this cost and latency; whether that is good enough is the threshold question above.
- Early-access model, no SLA, and it gives no reasons. In claims or credit decisions that is still a blocker.
How we ran it
Banking77 test split (PolyAI), all 3,080 human-labeled messages, 77 intents. One call per message through the Vercel AI Gateway (typesafe-ai/jev, AI SDK experimental_evaluate), one choice question: "Which banking support intent does this customer message express?", options are the 77 label names with underscores turned into spaces. Confidence is the value Jev returns for that question. 0 errors, 9 minutes of wall time at 4 calls in parallel, $0.22 total.
Choosing the threshold, building the eval and walking the customer through the trade-off is the job of a forward deployed engineer. We train engineers for it.