Last updated: September 2026
How to set a Jev confidence threshold
Pick the threshold from how hard the action is to undo, then measure it on a labelled sample of your own. Auto-hiding a post is one click to reverse, so it can run at a low bar. Publishing to a brand account cannot be reversed once people have seen it, so it needs a high bar and a person.
One correction first, because it decides whether your code even compiles into something meaningful: a Noul answer has no confidence field. The float it returns is the probability.
Which answers even have a confidence
Two of the three primitives return a confidence. The third does not, and the third is the one people reach for first.
A Noul returns noul and nothing else, per TypeSafe's Noul docs. Code that reads answers.spam.confidence on a Noul gets undefined, and undefined > 0.8 is false, so the gate quietly never fires. It does not throw. You find out in production.
Where confidence does exist, it is derived, not measured separately. TypeSafe's docs describe it as a statistic computed from the probability distribution the answer already gives you. A concentrated distribution produces a high confidence, a flat one a low confidence. It is a summary of the spread, not a second opinion.
What calibration actually means
A calibrated 0.7 means that across a large number of cases where the model said 0.7, roughly 70 percent of them turn out to be true. Not "the model is fairly sure". Seventy out of a hundred.
That property is what makes thresholding meaningful at all. If the number is calibrated, moving your bar from 0.7 to 0.9 has a predictable effect: you act on fewer cases, and a larger share of the ones you act on are right. If the number is not calibrated, moving the bar just moves the bar.
Calibrated probabilities are the centre of TypeSafe's pitch. It says it trains for them with a method it calls RLCD, reinforcement learning for calibrated decisions, optimising for honest probabilities rather than for text a human likes. What it has not published is any evidence: no calibration curves, no Brier scores, no reliability diagrams, no paper. Its own docs decline to give a universal threshold and tell you to start conservative, test on your own data and adjust.
Calibration is a claim until you check it
Check it the cheap way. Take a few hundred labelled cases, bucket them by the returned probability, and compare the bucket midpoint against the share that were actually true. If the 0.7 bucket comes back at 70 percent, the number is doing its job on your data. If it comes back at 40, no threshold you pick from theory will save you.Set the bar by reversibility
This is the part that turns a threshold from a guess into an engineering decision. Ask what the true branch does, then ask what it costs to take it back. TypeSafe's own confidence docs say the same thing in one sentence: different actions in the same system should be gated at different levels depending on the consequences of getting it wrong.
Notice that the bottom two rows are not threshold problems. Some actions do not get a threshold at all, they get a person. Raising a number from 0.9 to 0.95 does not make an unsendable DM sendable.
There is a design move hiding in the top half of that table. If your bar has to be uncomfortably high, change the action instead of the number. "Publish" needs 0.9 and a human. "Hold as a draft for review" needs almost nothing, because the worst case is that a person reads one extra draft. Most autonomous posting pipelines are safer with a low bar on a reversible action than a high bar on an irreversible one.
Read the threshold off a labelled sample
Six steps, and the first one is not about the model.
- Name the action and its undo cost. Write both down. If you cannot describe the undo, you are not ready to automate it.
- Label a couple of hundred real cases. From your own traffic, not from examples you invented. Label before you see any model output, otherwise you are grading the model with the model.
- Score the sample against a pinned model id. Use
jev-1.13.0, notjev-latest. Store every answer field next to your label. - Sweep every candidate cut. Not one. At each cut, count what passed, how many of those you had labelled true, and how many true cases you lost.
- Pick the cut that fits the error you can afford. This is a business decision dressed as a number. See the asymmetry below.
- Freeze it, log it, re-check it. Questions and thresholds in one reviewable file. The pinned model id logged with every decision.
The sweep is about fifteen lines:
// sweep.mjs
// Run once over a labelled sample. Print precision and recall at every cut.
import { readFileSync } from "node:fs";
// Each row: { text, label: true|false, score, confidence }
const rows = JSON.parse(readFileSync("scored-sample.json", "utf8"));
const positives = rows.filter((r) => r.label).length;
for (const cut of [1.0, 1.5, 2.0, 2.5, 3.0]) {
const passed = rows.filter((r) => r.score >= cut);
const right = passed.filter((r) => r.label).length;
const precision = passed.length ? right / passed.length : 1;
const recall = right / positives;
console.log(
cut.toFixed(1),
"passed", String(passed.length).padStart(4),
"right", String(right).padStart(4),
"precision", (precision * 100).toFixed(0) + "%",
"recall", (recall * 100).toFixed(0) + "%"
);
}What the output looks like
A worked example: 200 labelled drafts, 60 of which you decided were worth publishing, scored on a five level Score question.
Illustrative output from a 200 case sample with 60 positives. These numbers are the shape of the exercise, not a measurement of Jev. Run it on your own labels and your table will look different.
Now the table is doing the arguing. At 2.0 you publish 62 drafts and 21 of them should not have gone out. At 2.5 you publish 28 and only 4 are wrong, but you left 36 good drafts on the floor. At 3.0 you are never wrong and you publish almost nothing.
Nothing about the model tells you which of those rows is correct. Only the undo cost does.
False positives and false negatives are not the same size
Every threshold trades one for the other. The mistake is treating the trade as symmetric.
- A false positive on publish is a bad post on a real account, seen by real people, quotable after you delete it. It costs reputation and it does not stay deleted.
- A false negative on publish is a good post that sat in a drafts folder. It costs one post's worth of reach, and a person skimming the queue recovers it.
- A false positive on hide is one post blurred in a feed. The reader clicks it. Cost is a second of annoyance.
- A false negative on hide is the thing you were filtering, still in the feed. Which is exactly where it was before you built the filter.
Those two actions deserve opposite thresholds, and for the same reason. Push the errors toward the cheap side. For publishing, the cheap side is holding things back, so set a high bar. For filtering, the cheap side is over-hiding, so set a low one.
When you write the gate, the two numbers on a Score mean genuinely different things, and conflating them is the second most common mistake after the Noul one:
const PUBLISH_AT = 2.5; // from the sweep, not from a feeling
const CONFIDENT_AT = 0.6; // only applies to Choice and Score
const SLOP_CEILING = 0.5; // a noul is a probability, threshold it directly
function route(answers) {
// Noul: the float IS the probability. There is no answers.slop.confidence.
if (answers.toxicity.noul > 0.2) return "block";
if (answers.slop.noul > SLOP_CEILING) return "review";
// Score: two separate bars, because they mean different things.
// A low score means the draft is weak. Low confidence means Jev is unsure
// whether the draft is weak, which is a different problem with a different fix.
if (answers.strength.confidence < CONFIDENT_AT) return "review";
if (answers.strength.score < PUBLISH_AT) return "rewrite";
return "publish";
}A low score says the draft is weak, and the fix is to rewrite it. A low confidence says Jev is unsure whether the draft is weak, and the fix is a person or a second opinion. Sending both down the same branch throws away the distinction that made the confidence worth returning.
Three ways a working threshold quietly stops working
- The alias moves.
jev-latestpoints at whatever is current. When it changes, your calibration was done against a model that is no longer answering. Pin the version and upgrade on purpose. - Someone edits the criteria. The number comes out of the distribution over the levels you wrote. Change a level description and the distribution moves, so the threshold you measured no longer applies. Version the questions like code, because they are.
- The inputs drift. You calibrated on text posts and now half the traffic is link posts. Nothing errors. The threshold just means something different. Re-run the sweep on fresh labels on a schedule.
One more thing worth knowing before you transpose a number. TypeSafe's own limitations page warns that there are no structural invariants across questions: do not assume a relationship holds between two separate answers, and do not carry a threshold from one question type to another. A 0.8 on a Noul and a 0.8 confidence on a Score are not the same measurement. More of that on what Jev is bad at.
A Score is also not a ruler. TypeSafe documents that score values are not linearly interpolatable in meaning, so a 1.5 does not mean halfway between level 1 and level 2. Use it to rank and to threshold, not to measure how much better one draft is than another.
A threshold running in production
The free post scorer is built on Jev. It asks seven questions per draft: one Score for reach strength across five rubric levels, three Scores for hook, clarity and specificity, and three Nouls for AI slop, engagement bait and toxicity. It asks all seven questions in one request.
It shows the confidence band rather than a bare number, because the distribution is what Jev actually returns and collapsing it to one digit throws away the part that tells you whether to trust it. It needs no signup, so it is the cheapest way to see what these numbers look like on your own writing before you wire a threshold to anything.
Frequently asked questions
Does a Noul answer have a confidence field?
No. A Noul returns only noul, a float between 0 and 1, and nothing else. The float is the probability, so it is both the answer and the confidence signal. Only Choice and Score return a separate confidence field. This is the single most common mistake in third-party Jev code, and reading answers.x.confidence on a Noul gives you undefined, which then fails a comparison silently.
What confidence threshold should I use?
There is no universal number, and TypeSafe declines to give one. Its docs suggest above roughly 0.9 for acting automatically on high-stakes decisions, verification in the middle, and routing to a human below roughly 0.5, then say to start conservative, test on your own data and adjust. The useful rule is to pick the bar from how hard the action is to undo.
What does calibrated actually mean?
It means the number behaves like a real probability. If a calibrated model says 0.7 across a thousand cases, roughly seven hundred of those cases should turn out to be true. Calibration is a property you verify on labelled data, not a property you can read off a single answer.
Is Jev calibrated?
TypeSafe says it trains for calibrated probabilities with a method it calls RLCD, and that is the core of its pitch. It has not published calibration curves, Brier scores or reliability diagrams, and there is no paper. So treat calibration as a claim to verify on your own labelled sample rather than as a guarantee you can build on.
How many labelled examples do I need to set a threshold?
Enough that each threshold bucket has real cases in it, which in practice means a couple of hundred rather than twenty. Twenty examples will tell you whether the question is worded sensibly. They will not tell you where the bar goes, because the interesting region is the tail and twenty examples have almost no tail.
Should the threshold be the same for every action in my system?
No. TypeSafe’s own confidence docs say different actions within the same system should be gated at different levels depending on the consequences of getting it wrong. Hiding a post is one click to undo, so it can run at a low bar. Publishing to a brand account cannot be undone once it has been seen, so it needs a high bar and a person.
What makes a threshold stop working?
Three things, all of them silent. Moving from a pinned model id to an alias, so the model changes under you. Editing the instructions or criteria, which changes the distribution the number comes from. And a shift in the inputs themselves, such as a new content type your sample never contained. Pin the model, version the questions, and re-check against fresh labels on a schedule.
Keep exploring
The primitives the numbers come from, and the places the threshold gets enforced.
Choice, Score and Noul
The three question types, the exact fields each answer carries, and when to reach for which.
What Jev is bad at
Counting, dates, indirection and untrusted state. All sourced to TypeSafe’s own documentation.
Jev and MCP: judgment plus action
Where the threshold sits when an agent is the thing acting on it.
Jev in n8n
The same two comparisons, as an IF node on a canvas.
Using Jev in Claude Code
A script whose exit code enforces the threshold instead of a prompt that asks for it.
Will it go viral
A live Jev scorer that shows the confidence band, because the distribution is what Jev returns.
Put the threshold in front of something that ships
A gate is only worth building if the true branch does something. OpenTweet's API publishes and schedules to X, Bluesky and LinkedIn from one call, and the same key works in the MCP server, the CLI and the n8n node.
7-day free trial. No X developer account needed.