The honest probability

September 2026

On September 15, a company called TypeSafe AI ended two years of stealth with a model that never speaks. Jev, built by a team founded by Diogo Almeida (ex-OpenAI, where he helped build the methods that made language models good at following instructions), does not generate text at all. You hand it a situation and a list of questions; it hands back decisions with probabilities attached. The name is a bet as much as a tribute: it honors William Stanley Jevons, the economist who noticed that making coal cheaper made Britain burn more of it, and it wagers that making machine decisions two orders of magnitude cheaper will make the world consume vastly more of them.

If you read the machinery essay on this site, you walked a transformer's whole pipeline and ended at the sampler: the loop that throws dice once per token, hundreds of times per paragraph. This essay is about what happens when you delete that loop entirely and keep the thing hiding just before it, the probabilities themselves. It turns out the interesting question is not how to get them out. It is whether they can be trusted, and that question runs through a finding in the GPT-4 technical report, a scoring rule from 1950s weather forecasting, and a coin that is unfair in a very precise way.

1 · Decisions, not sentences

TypeSafe calls Jev a System One Model, borrowing Kahneman's split between fast intuitive judgment and slow deliberate reasoning. The interface is a function call, not a chat. You send a state, a short dense paragraph describing your program's situation, plus a list of questions. Per Arcturus Labs' analysis, each question is one of three types: a choice (pick from a list, returning the pick plus a probability for every option), a score (a position on an ordered rubric), or a noul (a bare 0-to-1 probability for a yes/no question). The answers come back typed and schema-guaranteed, all questions resolved in one parallel pass through the model, the "parallel sampler", with no token-by-token loop at all.

Deleting the loop is where the headline numbers come from: TypeSafe claims 40x to 200x faster than frontier LLMs at similar System One task intelligence, 70 to 500 milliseconds end to end, $0.042 per million input tokens with output tokens free ("too cheap to meter"), and a demo agent playing Doom at about ten decisions a second, which TypeSafe puts at roughly $7 an hour. Two honest asterisks they print themselves: the 0% hallucination and 0% type-error numbers are by construction, since schema matching is guaranteed by the output format rather than measured, "not empirical"; and the headline speedups come from workflow evaluations built to resemble production, which TypeSafe itself expects to be "on the higher end of real world gains". The cleanest way to feel the shape of the product is to run one call. Below is a support ticket as a state, three questions against it, and the part that matters most: a threshold slider that turns a probability into an action.

state: Customer on the Pro plan reports checkout failing with a 500 error since the 14:00 deploy. Retried three times. Mentions a demo to their own client in two hours. Attached stack trace points at the payments service.
choice
Route this ticket to?
bugs.81
security.12
billing.07
→ bugs · confidence .81
noul
Is this urgent?
p(yes).83
score
Frustration, 1 to 5?
4.47
5.27
3.18
→ score 4 on the rubric
One Jev-shaped call: a state paragraph, three typed questions, hardcoded but plausible answers. The canvas runs the urgency noul over a batch of 200 synthetic tickets: green tickets are confident enough (probability above the threshold or below one minus it) to handle automatically, orange ones fall in the murky middle and go to a human. Drag the threshold and watch the automation rate trade against escalations; the marker is the ticket above. This slider is the product: code can branch on a number, and never could on prose.

The framing that makes this click is the fuzzy if statement. Ordinary code branches on conditions it can compute: if (retries > 3). It has never been able to branch on conditions it can merely describe: if this customer sounds like they are about to churn. An LLM can answer that question, but it answers in prose, which your code must then parse, validate, and pray over. Jev's answer is already a number in a schema. The if statement gets to stay an if statement; it just got smarter about what it can test. And notice what the threshold slider did in the figure: the boundary between software acting alone and software asking a human became one tunable parameter, auditable, adjustable in production, meaningful only because the probability under it is supposed to mean something. That "supposed to" is the rest of this essay.

2 · The probabilities were always there

Here is the deflating part, and the honest place to start: decision probabilities are not something Jev invented. Every LLM already computes them, at every step, and then throws most of them away. Recall the last station of the machinery essay's pipeline: after the final layer, the model produces a raw score for every token in the vocabulary, the logits, and a softmax turns them into a probability distribution before the dice are thrown. Generation samples one token from that distribution and discards the rest. But nothing forces you to sample. If the next token is the answer to your question, the distribution over it is your answer.

The parody-with-a-point proving it is NobodyWho's "Jev in 25 lines": a spam classifier built on Qwen3-0.6B, a model small enough to run locally. Prompt it so that the next token must be one of three labels, then skip sampling entirely: read the final-token logits for just the three label tokens, softmax them (with the log-sum-exp trick for numerical stability), done. For a phishing email their example returns Phishing 0.885, Spam 0.084, Legitimate 0.031. Local, fast, twenty-five lines. Play with the evidence and watch the same arithmetic move:

Left: the final-token logits for the three label tokens, the raw scores any LLM computes anyway. Right: the same three numbers after softmax. With all three evidence boxes checked, the probabilities are the actual figures from NobodyWho's Qwen3-0.6B run: .885, .084, .031. Untick the evidence and the logits shift plausibly (hardcoded deltas, honest shape) and re-softmax live. Note how the argmax label flips long before the probabilities become certain: that gap between "most likely" and "how likely" is the entire subject of this essay.

So the raw material is free and always was: three logits and a softmax. What TypeSafe is selling cannot be the extraction. The hard question, the one that decides whether you can wire that threshold slider to anything that matters, is whether 0.885 means anything. When this little classifier says 88.5% phishing, is it right about 88.5% of the time? For chat-tuned LLMs the answer, measured carefully, is no, and the reason it is no was documented by the very lab that made them.

3 · What calibration actually means

The property we want has a name: calibration. A model is calibrated when its stated confidence matches its actual hit rate: among all the predictions where it says 70%, about 70% should come true; among the 90s, about ninety in a hundred. Note what this does not require: it does not require being right, or even being confident. A weather forecaster who says 50% rain every single day in a city where it rains half the days is perfectly calibrated and perfectly useless. Calibration is a promise about the numbers, not about the knowledge: whatever confidence the model reports, you can take it at face value.

The standard instrument for checking the promise is the reliability diagram: bucket every prediction by stated confidence, then plot each bucket's stated confidence against its actual frequency of being correct. A calibrated model hugs the diagonal. An overconfident model sags below it (it says 90% and delivers 70%); an underconfident one bows above. The gap, averaged across buckets and weighted by how many predictions land in each, is the expected calibration error, ECE. And the useless-forecaster caveat gets its own axis, sharpness: how far the predictions dare to stray from the base rate. The two axes are independent, and the figure lets you move each without the other:

Synthetic predictions, base rate 50%, each one resolved by an actual simulated coin flip at its true probability. Each dot is a confidence bucket: stated confidence across, observed hit rate up; dot size is how many predictions live there (the gray strip below counts them). Drag character toward overconfident and the curve sags below the diagonal while ECE climbs; drag toward underconfident and it bows above. Slide sharpness to zero: every prediction collapses to the base rate, and the model is calibrated and useless, one dot sitting smugly on the diagonal. Then shrink the number of predictions observed: even a perfectly calibrated model wobbles off the diagonal, because a hit rate measured on fifty predictions is mostly luck. ECE has a noise floor, and a reliability diagram is only as trustworthy as the data under each dot.

Keep both axes in mind, because the next two sections are a story about each. Chat-tuned LLMs tend to fail on the first axis in a specific direction, overconfidence, and the usual explanation is that their training rewards the second axis: sounding sharp. A pretrained base model, it turns out, starts out surprisingly honest. Then we fine-tune the honesty out of it.

4 · How RLHF unlearns honesty

The post-training recipe that turned base models into assistants, RLHF, runs in three beats. First, collect human preference pairs: show annotators two candidate answers, ask which is better. Second, train a reward model to predict those preferences. Third, optimize the policy against that reward model with an algorithm like PPO, held on a KL leash so it cannot drift too far from the base model. Notice what the objective actually is: not truth, not accuracy, but what humans approve of. Those usually correlate. Where they part ways is exactly where things get interesting.

Here is the widely held explanation for where they part. Human raters tend to prefer articulate, assertive answers: a hedge reads as weakness, and a confident wrong answer can read better than a correct maybe. To the extent that holds, the reward model learns to pay out for confidence regardless of correctness, and preference optimization does what optimizers do: it sharpens the policy's distributions toward the rewarded mode, driving entropy down, collapsing the spread that encoded honest uncertainty. The cleanest published evidence of the effect, if not the cause, is in OpenAI's GPT-4 technical report: on a subset of MMLU, the pretrained model was highly calibrated, its confidence in an answer tracking the probability of that answer being correct, and after post-training "the calibration is reduced"; the report's own figure caption says post-training hurts calibration significantly. The report shows the damage without explaining it. The figure below runs the explanation above in miniature: a calibrated distribution under repeated "annotators prefer confident answers" pressure.

Five candidate answers. At zero updates the model is calibrated by construction: each answer's probability equals its actual chance of being correct (the small ticks). Drag the slider and each update applies the annotators-prefer-assertiveness pressure: mass flows to the mode, entropy drains, the approval meter climbs, and the calibration error meter climbs right along with it, because answer A is still only right 42% of the time no matter how loudly the model says it. Nothing about the answers improved; only the tone did.

Arcturus Labs measured the real-world version of that collapse with a beautifully simple probe: a coin described as landing heads 60% of the time. Ask Jev a chat-shaped question about it, a choice between heads and tails, and the answer comes back 99% heads: winner-take-all, the mode with the volume turned up, exactly the end state of the slider above. Ask the same fact as a noul, "probability the next flip is heads?", and it returns 58%, two points off truth. Same model, same coin, and the question type decides whether you get an honest number or a confident one. So chat models sound most sure precisely where being sure is unearned, and even a purpose-built decision model carries scar tissue in some of its answer types. Which raises the real question: what would training for honesty even look like?

5 · RLCD: reward honesty itself

TypeSafe's answer is a training method they call RLCD, Reinforcement Learning for Calibrated Decisions: optimize not for answers humans approve of (RLHF) and not just for verifiably correct answers (RLVR), but for decisions paired with probabilities that are epistemically honest, confidence that correlates with actual accuracy, trained on System One tasks where the outcome is known. The recipe is proprietary; TypeSafe has published the name and the goal, not the loss function. But there is a standard, decades-old mechanism that the community (Arcturus among others) infers is doing the core work, because it is the only known way to make honesty the mathematically optimal move: score the reported probability against the realized outcome with a strictly proper scoring rule.

Start from first principles. The model reports a probability p that some event will happen; then reality answers, 1 or 0; then we pay the model a reward that depends on both. The question is what payment schedule makes truthfulness the best strategy. Pay for accuracy only, a point if your most-likely pick matches the outcome, and honesty is worthless: reporting 51% and reporting 99% earn identical expected reward, so nothing stops the drift to bravado (that is the preference-shaped reward of the last section in miniature). But pay with the Brier score, reward = -(p - outcome)^2, or with the log score, reward = log p of whatever actually happened, and something remarkable becomes provable: your expected reward is maximized at exactly one report, your true belief. Any exaggeration in either direction costs you, in expectation, every time. That is what "strictly proper" means, and rather than prove it with algebra, the figure lets you feel it. You are the model. Reality flips a coin with a hidden bias. Report a probability and try to beat honesty:

The curve is your expected reward per prediction as a function of what you report, given the true rate. Under accuracy-only reward the optimum is a flat green plateau: an honest 0.60 and a shameless 0.99 pay identically, so overconfidence is free. Switch to Brier and the curve turns strictly concave with its peak exactly on the true rate (the star; move the truth slider and the peak follows it). Switch to log score: same peak, far crueler tails, the cliff toward negative infinity at 0 and 1 clamped here at the chart floor. Under a strictly proper rule, every report except the truth leaves reward on the table, and the readout shows exactly how much yours does.

Now scale the game up: thousands of decision tasks with known outcomes, the reported probability scored by a strictly proper rule, reinforcement learning climbing the expected-reward hill. The policy that hill-climbing finds is not a model that grudgingly tolerates uncertainty; it is a model for which honest uncertainty is the cheapest possible behavior. That inversion is the whole RLCD idea: in RLHF, calibration fights the objective and loses; here, calibration is the objective's own maximum. It also explains why supervised fine-tuning alone cannot get there: SFT can teach a model which answers to give, but it copies labels, and no label can tell a model how reliable it is on a question; only being scored against real outcomes, many times, lets it learn its own error rate. Two honesty notes belong right here. First, TypeSafe has not published the recipe, so proper-scoring-rule RL is the inferred core, not a confirmed one. Second, calibration evidently is not free even inside Jev: Arcturus swept 401 coin biases through noul questions and found tracking that was close but systematically low, the largest miss 9 points, a 40% prediction for a 49% coin, while choice questions on the same facts collapsed to 99%. Calibration is a per-question-type property that has to be earned, not a switch. (One scholarly footnote: the acronym RLCD already existed. a 2023 paper by Yang and colleagues at Berkeley and Meta, "RLCD: Reinforcement Learning from Contrastive Distillation", is an unrelated alignment method; TypeSafe's RLCD is a different thing wearing the same letters.)

6 · The trade, honestly

So state what Jev is by what it gives up. No text, so no explanations, no reasoning traces, no chain of thought to audit; System 2 work, the deliberate multi-step kind, stays with LLMs, and TypeSafe says so plainly. What you get for the sacrifice: claimed speedups of 40 to 200 times and a price close to free (with the asterisks from section 1 still attached), outputs whose types are guaranteed by construction rather than parsed on faith, and probabilities intended to be load-bearing, so that the interface between AI judgment and ordinary code becomes a threshold in a config file. The natural habitat is everywhere a fuzzy if statement wants to exist: routing and triage like the ticket in section 1, guardrails that score a request before an expensive model sees it, filtering retrieved documents in a RAG pipeline, game AI making ten decisions a second in Doom, any loop too hot and too cheap to justify a sentence.

The trade at a glance, using TypeSafe's own claimed numbers where numbers exist. The right column's failure mode deserves the emphasis: a system that returns only probabilities can no longer hallucinate a sentence, but it can absolutely return a miscalibrated number, which is why the training method matters more than the interface.

And the honest unknowns, stated as unknowns: the training details are proprietary, so the proper-scoring-rule account in section 5 is the community's best inference, not a spec; the coin experiment shows calibration quality differs by question type inside the same model, so every deployment needs its own reliability diagram, not a vibe; and nobody yet knows how the calibration holds up under domain shift, on states unlike anything in the training distribution, which is precisely where an overconfident number does the most damage. What remains is the name on the box. Jevons' paradox was that efficiency does not reduce consumption; it explodes it. TypeSafe's bet is that decisions at $0.042 per million tokens and 70 milliseconds stop being something you ration and start being something you sprinkle through every branch of every program, the way cheap coal found its way into everything. Whether that future arrives depends less on the speed than on the property this essay has been circling the whole time: that when the model says 83%, it has earned the right to be believed 83% of the time.

Sources: TypeSafe's announcement of System One Models and Jev, Arcturus Labs' independent analysis and coin experiments, NobodyWho's "Jev in 25 lines", and the calibration figures in the GPT-4 technical report.