NewJev as a Judge with Opik
Evaluate support answers with typed Jev rubrics and record the results in Opik.
Added to Jevfast
NewHow it uses Jev
The evaluator passes a support case containing the customer's request, policy, tool results, and final answer to Jev. Jev answers three Noul questions about factual grounding, request coverage, and honest action claims, plus a Score question about how actionable the next step is. The application maps those typed answers to metrics, then applies its own illustrative thresholds to mark the case pass, fail, or review. The sample evaluates synthetic frozen support traces; it is not a production support guardrail.
What Jev decides
Illustrative support-answer rubricExample answers · not a recorded Jev response · Source ↗
Question 1 · grounded
YOUR APP
INSTRUCTION
Judge whether every factual claim in the answer is supported by the policy and tool results.
STATE
The source-defined state contains a customer request, the applicable policy, structured tool-call results, and the final support answer. The project fixture is synthetic; this chart summarizes the documented request fields rather than a captured runtime case.
JEV · NOUL
YesNo
Question 2 · addresses request
YOUR APP
INSTRUCTION
Judge whether the answer addresses the request, including a needed limitation or clarification.
STATE
The source-defined state contains a customer request, the applicable policy, structured tool-call results, and the final support answer. The project fixture is synthetic; this chart summarizes the documented request fields rather than a captured runtime case.
JEV · NOUL
YesNo
Question 3 · action honest
YOUR APP
INSTRUCTION
Judge whether any completed-action claim has confirmation in successful tool results.
STATE
The source-defined state contains a customer request, the applicable policy, structured tool-call results, and the final support answer. The project fixture is synthetic; this chart summarizes the documented request fields rather than a captured runtime case.
JEV · NOUL
YesNo
Question 4 · helpfulness
YOUR APP
INSTRUCTION
Rate how actionable the next step is using the rubric's three levels.
STATE
The source-defined state contains a customer request, the applicable policy, structured tool-call results, and the final support answer. The project fixture is synthetic; this chart summarizes the documented request fields rather than a captured runtime case.
JEV · SCORE
0
App workflow
- Load a frozen case
The CLI reads synthetic support cases with the request, policy, tool calls, and final answer; gold labels and demo answers remain outside the Jev state.
- Run mechanical checks
The evaluator rejects an empty answer or one over the length limit before making a Jev call.
- Ask the rubric
JevClient sends the allowlisted state and the four versioned typed questions, then validates answer keys and types.
- Map answers to policy
The code maps Noul values and normalizes the helpfulness Score, then applies the tutorial's illustrative pass, fail, or review thresholds.
- Record the evaluation
The CLI can write local JSONL results. The optional Opik adapter records metrics and response metadata in a uniquely named experiment.
Why it is interesting
The example separates a model's typed judgments from deterministic validation and acceptance policy. It also excludes gold labels and demo answers from the judged state, which makes the evaluation boundary easier to audit.