LangWatch Instant Evals
Runs typed Jev judgments over selected traces and spans in a user-triggered Instant Eval job.
Added to Jevfast
How it uses Jev
An Instant Eval job selects traces or spans and passes their text to Jev as state. Configured classifier questions produce Boolean, integer Score, or Category Choice judgments, and the run service stores those verdicts with the evaluated traces.
What Jev decides
Illustrative classifier requestExample answers · not a recorded Jev response · Source ↗
Question 1 · configured boolean question
YOUR APP
INSTRUCTION
Judge whether the configured criterion holds for the supplied trace text.
STATE
Text extracted from a selected trace or span, plus configured evaluator questions.
JEV · BOOLEAN
YesNo
App workflow
- Start Instant Eval
A user or API requests an evaluation job over selected traces or spans.
- Page through traces
The run service loads a bounded page and extracts text for the configured evaluator.
- Ask Jev
Build typed classifier questions and send them with the trace text.
- Store verdicts
Validate and persist results against the evaluated traces or spans.
Why it is interesting
LangWatch connects typed evaluation criteria to observability data: a user can define an evaluator once, run it over selected traces, and inspect stored verdicts beside the original activity.