Small decisions. Interesting possibilities.Submit contentSubmit

Jevcal confidence-threshold calibration

Everyone picks Jev confidence thresholds by vibes. 0.95? 0.5? I built jevcal. Give it your data and say "I need 99% accuracy". You get the exact threshold, how much Jev can handle, and how much still needs an LLM. Open source, one command. @typesafeai

Added to Jevfast

How it uses Jev

jevcal measures typed questions on labeled examples, selects confidence thresholds against an accuracy target, and estimates how much traffic those thresholds handle versus escalate.

What Jev decides

Illustrative Jev requestExample answers · not a recorded Jev response · Source ↗
Question 1 · is fraud
YOUR APP
INSTRUCTION

Is this email a fraud attempt?

STATE

An email example and the task-specific fraud question.

JEV · NOUL
YesNo

App workflow

  1. Define questions

    Describe the Noul or Choice questions and target acceptance criteria.

  2. Prepare examples

    Supply labeled JSONL data, optionally using an LLM teacher to draft labels and then reviewing them.

  3. Lint and measure

    Check question wording, run Jev on the dataset, and select thresholds on one split.

  4. Validate and gate

    Check thresholds on held-out data, inspect the HTML report and lock file, and enforce targets in CI.

Why it is interesting

It turns threshold tuning into a held-out calibration workflow and produces artifacts that can be checked in CI when model behavior changes.