Jsort
Rank text on a natural-language criterion using pairwise Jev judgments and a fitted statistical scale.
Added to Jevfast
How it uses Jev
jsort compares texts in pairs. For each pair it puts the texts in state as A and B, then asks a Noul question created from the user's criterion: whether Text A ranks higher than Text B on that criterion. jsort fits the pairwise probabilities into a scale, and can save selected texts as anchors for placing later texts on that scale.
What Jev decides
Illustrative pairwise text comparisonExample answers · not a recorded Jev response · Source ↗
Question 1 · q
YOUR APP
INSTRUCTION
Determine whether Text A ranks higher than Text B on the user's stated criterion.
STATE
State contains the two compared texts as A and B, truncated according to the run's max_chars setting. The criterion is supplied by the user. This example shows the request form only; the selected answer is illustrative and was not observed from Jev.
JEV · NOUL
YesNo
App workflow
- Prepare texts and criterion
jsort truncates texts to the configured character limit, removes blank entries from comparison, and creates a Noul question from the supplied description.
- Choose and ask pairs
It starts with a ring schedule, then refits the scale and selects neighboring or uncertain pairs. Each request places the pair in state as A and B and asks question q.
- Fit and report the ranking
Returned probabilities feed a fitted scale, with scores, standard errors, and comparison counts aligned to the input texts.
- Optionally save and reuse a scale
A completed run can keep anchor texts with its question and model identity; later placement runs compare new texts against those anchors.
Why it is interesting
A natural-language criterion can define a ranking dimension without a hand-written scoring rubric. Adaptive pair selection focuses later comparisons on nearby or uncertain texts, and a saved scale lets new texts be compared with anchors from an earlier run.