Tasksource Jev Audit
Tasksource publishes rows with state, question, kind, options, and target.
Added to Jevfast
How it uses Jev
Tasksource publishes rows with state, question, kind, options, and target. In the pinned Jev audit path, related rows sharing a source and state are grouped into one request, each question is mapped from its row kind to Choice, Score, or Noul, and the state is shared. The supplied dataset card includes a concrete COPA Choice row.
What Jev decides
Illustrative source-derived request: COPA criterion choice from the Jev audit exampleExample answers · not a recorded Jev response · Source ↗
Question 1 · q0
YOUR APP
INSTRUCTION
Choose the criterion that best answers the question.
STATE
State: “My body cast a shadow over the grass. What was the cause of this?” The question comes from the dataset row; the audit wrapper places a question over this state.
JEV · CHOICE
- The sun was rising.
- The grass was cut.
App workflow
- Load or build typed decision rows
The runbook uses build_jev_dataset.py and the release card documents loading the tasksource-jev-typed-decisions dataset.
- Group rows by source and state
The audit script assembles related rows into a shared-state request and gives them q0, q1, and later keys.
- Map row kinds to Jev questions
The adapter maps choice options to Choice criteria, ordered score options to Score criteria, and noul rows to Noul questions.
- Audit returned answers
The tool validates returned answers, records annotations, and caches completed state/question requests for reuse.
Why it is interesting
The corpus preserves task-specific state, wording, and options while mapping many datasets into a small set of typed decision primitives. Its audit tool groups related decisions over one state into one request.