Jev as a reranker: a tie on quality, nine times cheaper

Last week I said I'd try Jev for reranking in RAG pipelines. Here's what happened.

The setup: an internal dataset of a few thousand documents and policies, 50 labeled queries, BM25 pulling the top 20 candidates per query. Then four setups ranked them:

• BM25 only: nDCG@10 = 0.69
• Embedding reranker: 0.72
• LLM reranker: ~0.78
• Jev: ~0.78

On quality, it was a tie between the LLM and Jev but Jev was much faster and cheaper: ~0.5s/query and $0.59 per 1,000 queries vs. ~4.5s and $2.60.

Last week, judging verbose content against a complex rulebook, most of Jev's answers sat in the coin-flip range. Today, across 1,000 pairs, 88% were decisive (below 0.2 or above 0.8). Same model but opposite behavior.

It makes one thing clear that relevance is a classification question and judgment is not.

Meanwhile, on Tuesday, OpenAI announced its own Decisions API at DevDay. No dedicated pricing or schema yet, so nothing to test as of yet.

Things are moving too fast for me to try everything. How are you all coping with the pace?