RAG-Fusion Jev Evaluation
Tests Jev as a typed relevance judge and query router inside a RAG-Fusion retrieval experiment.
Added to Jevfast
How it uses Jev
The RAG-Fusion evaluation script sends a query and candidate documents to Jev. It asks one Noul relevance question per document and uses the returned yes probabilities; a separate Choice request classifies query kind for routing.
What Jev decides
Illustrative document relevance requestExample answers · not a recorded Jev response · Source ↗
Question 1 · D00
YOUR APP
INSTRUCTION
Is document D00 relevant to the query?
STATE
A query and documents keyed D00, D01 and so on.
JEV · NOUL
YesNo
Illustrative query routerExample answers · not a recorded Jev response · Source ↗
Question 1 · kind
YOUR APP
INSTRUCTION
Choose the query kind for retrieval routing.
STATE
The incoming query text.
JEV · CHOICE
- navigational
- specific
- broad
- multi_faceted
App workflow
- Run an evaluation
A caller starts the retrieval experiment with queries and candidate documents.
- Judge relevance
Jev returns one Noul relevance probability per document.
- Route query kind
A separate Choice classifies the incoming query.
- Compare retrieval
Evaluation scripts consume the signals to assess or adjust ranking and routing.
Why it is interesting
The experiment separates retrieval into two small typed decisions: what kind of query arrived and which candidate documents are relevant. That makes ranking and routing signals independently inspectable.