Building a post scorer on Jev

Writing a rubric that scores the same way twice

The interesting work is not the API call. It is writing rubric levels concrete enough that the same draft lands on the same rung tomorrow. The reach rubric below spells out what each level means in terms a person could apply:

const STRENGTH_LEVELS = [
  'Flops. Gets less attention than a typical post from the same account.',
  'Typical. Performs about the same as the account usually does.',
  'Above typical. Noticeably more reach and replies than usual.',
  'Strong. Several times the usual reach. Spreads past the existing audience.',
  'Breakout. Orders of magnitude beyond usual. Reaches people who have never heard of the author.',
] as const;

Note what the levels are anchored to: a typical post from the same account. Without that anchor the model scores fame rather than the writing, and every draft from a small account scores low for reasons the writer cannot act on.

One rubric needed more than prose. Hook quality is vague enough that it takes worked examples alongside each level, which the Score primitive supports:

hook: score('How well does the opening line stop someone mid scroll?', [
  { level: 'No hook. Opens with context, throat clearing, or a greeting.',
    examples: ['So I was thinking about', 'Here is a thread about'] },
  { level: 'Functional. States the subject plainly without creating a reason to continue.',
    examples: ['3 things I learned shipping a side project'] },
  { level: 'Strong. Creates tension, a gap, or a surprise in the first line.',
    examples: ['I deleted 90% of our code and revenue went up.'] },
]),

This matters because the UI names the level a draft reached using the model's own words instead of copy invented afterwards. When the rubric changes, the page changes with it, and there is no second source of truth to drift.

Measuring what each line is worth, instead of guessing

Telling someone their draft scores 2.4 is not useful on its own. What helps is knowing which part is carrying it. Because a score is cheap enough to run repeatedly, you can answer that by measurement rather than heuristics: strip one element, re-score, and report the difference.

export async function measureAblations(text: string, baseline: number): Promise<Ablation[]> {
  const candidates = ABLATIONS.map((a) => ({ ...a, variant: tidy(a.apply(text)) })).filter(
    (a) => a.variant !== tidy(text) && a.variant.length >= MIN_SCOREABLE
  );

  const scores = await Promise.all(
    candidates.map((c) => scoreStrengthOnly(c.variant).catch(() => null))
  );

  return candidates
    .map((c, i) => ({ key: c.key, label: c.label,
      delta: scores[i] === null ? 0 : baseline - (scores[i] as number) }))
    .filter((a) => Math.abs(a.delta) >= 0.05)
    .sort((a, b) => Math.abs(b.delta) - Math.abs(a.delta));
}

Two details worth copying. Only elements that are actually present and that leave a scoreable draft behind get measured, so you never report a number for something the post did not contain. And deltas under 0.05 are dropped, because below that you are showing the reader noise and asking them to act on it.

Ablations are four extra model calls, so they only run when a draft settles rather than on every keystroke. The strength-only variant exists for exactly this: asking the other six questions would be paying for answers nobody reads.

What Jev is not doing here

Worth stating plainly, because the category is new enough that people assume more than is there.

• Jev writes nothing. It cannot. It never emits text, only typed answers. Every word a user sees is either their own draft or our rubric copy.
• Jev does not write anything in AI Studio either. Generation there runs on a large language model. Jev only scores what comes back, which is work a System One model can do and a writing model cannot do cheaply.
• A score is a prior, not a result. It is a model's read of a draft before anyone has seen it. Published performance is the real number, and that lives in analytics.
• Calibrated does not mean correct. Jev always returns a valid answer inside your schema. It can still be confidently wrong, which is a different claim from the one the launch coverage made. See can Jev hallucinate.