NewConvex Decision Evals
Compare decision models on versioned multiple-choice questions about Convex development.
Added to Jevfast
NewHow it uses Jev
Convex Coding Evals has a separate multiple-choice benchmark that routes questions through OpenRouter's native decision endpoint for Jev, and through structured chat completion for comparison models. For Jev, the harness sends a question context and text with four displayed options; it validates the returned option and probability distribution before scoring against a hidden answer key.
What Jev decides
Convex function visibility question (illustrative)Example answers · not a recorded Jev response · Source ↗
Question 1 · decision
YOUR APP
INSTRUCTION
Choose which of the four presented answers correctly identifies the unauthenticated ordinary-client callable functions. The displayed A–D labels are generated from the source option permutation.
STATE
Representative accepted-bank question q1 from 000-fundamentals/000-empty_functions. The source context shows six empty public and internal handlers in Convex 1.44.0. The question asks which can an unauthenticated ordinary client invoke directly. The four source options are permuted by seed and repetition before presentation, so their displayed letter-to-answer mapping is dynamic.
JEV · CHOICE
- A
- B
- C
- D
App workflow
- Load the benchmark bank
Load curated questions with context, four options, a hidden correct option ID, and source-eval identity. The representative fixture is saved in the review evidence; display order depends on seed and repetition.
- Present a provider request
The harness deterministically permutes the four options and labels them A through D. Jev receives one native Choice question through OpenRouter; other models use structured chat completion.
- Validate and score
The runner validates the selected displayed ID and, for Jev, probabilities and confidence. It maps the displayed choice back to the canonical option and scores the answer against the answer key.
Why it is interesting
The project isolates specific Convex knowledge from open-ended coding performance and maintains a versioned, answer-keyed question bank with archived request evidence. It explicitly cautions that a correct multiple-choice answer does not establish the ability to build or debug an application.