ek.agents.judge#

LLM-as-judge: a reference-free Signal, a reference-based Metric, and the validation that must precede either.

When a task has no checkable outcome, an LLM judge is the fallback. It is also a **liability unless validated and calibrated**, because its known failure modes have double-digit effect sizes: position bias (which answer came first), verbosity bias (longer looks better), self-preference (a model prefers its own text). misc/docs/ek_09 has the numbers.

Three seams, deliberately kept apart (they are not interchangeable):

  • JudgeSignal – reference-free, criteria-only. This is the only judge that is a Signal: it scores an output against criteria, with no gold, so it plugs into ek.estimate_quality()’s signal -> calibrate -> validate -> decide cascade. Its

    cost_tier is the highest in the framework, so an escalation policy runs the cheap signals first and pays for the judge only on residual uncertainty.

  • JudgeMetric – reference-based. A judge that needs the gold answer is not a Signal at all; it is a Metric, and it belongs on the ek.score() side.

  • pairwise_judge() – pairwise comparison needs two outputs and cannot fit Signal.__call__(one_output). It is a helper, and it swaps the order and averages – the standard, necessary correction for position bias.

Hard Rule 1 applies to judges. A raw judge score is not a probability. Calibrate it before any DecisionPolicy reads it – ek.estimate_quality() already refuses to gate an uncalibrated signal, and a judge is no exception (-> misc/docs/ek_03).

Calibrate a judge per signal family. With a single calibrator, ek.estimate_quality() averages all raw signals into one confidence and calibrates that mean. Mixing a judge score with, say, a logprob – different scales, different reliabilities – that way is not statistically sound. Pass calibrator={"judge": judge_cal, "logprob": logprob_cal} (or a SignalFamilyCalibrator, which can fit one calibrator per signal from labelled rows) so each family is calibrated on its own before the calibrated probabilities are fused.

ek never imports an LLM SDK: judge is an injected callable.

Example

>>> stub = lambda output, *, criteria="", reference=None: 0.9 if "meow" in output else 0.1
>>> sig = JudgeSignal(stub, criteria="Is the answer helpful?")
>>> sig("the cat says meow")
0.9
>>> sig.cost_tier                       # the most expensive signal family
5

Module Attributes

JUDGE_COST_TIER

Judges are the most expensive signal family -- run cheap signals first and escalate.

ALPHA_GOOD

>= 0.8 is good agreement, 0.67-0.8 only tentative.

Functions

judge_validation(judge_labels, human_labels, *)

Certify a judge against human labels before trusting it.

pairwise_judge(judge, a, b, *[, criteria, swap])

Pairwise preference for a over b, corrected for position bias.

Classes

JudgeMetric(judge[, criteria])

A reference-based LLM judge as a Metric (the score() side).

JudgeSignal(judge[, criteria, cost_tier, name])

A reference-free, criteria-only LLM judge as a Signal.
ek.agents.judge.ALPHA_GOOD = 0.8#

>= 0.8 is good agreement, 0.67-0.8 only tentative.

Type:

Krippendorff conventions

ek.agents.judge.JUDGE_COST_TIER = 5#

Judges are the most expensive signal family – run cheap signals first and escalate.

class ek.agents.judge.JudgeMetric(judge, criteria='')[source]#

Bases: object

A reference-based LLM judge as a Metric (the score() side).

A judge that reads the gold answer is not a reference-free signal, so it does not belong in ek.estimate_quality(). It lives here.

Parameters:

  • judge (Callable) – Injected (output, *, criteria, reference) -> float in [0, 1].

  • criteria (str) – The rubric.

aggregate(scores)[source]#

Mean judge score (a judge score is per-item; there is no global accumulation).

Return type:

float

class ek.agents.judge.JudgeSignal(judge, criteria='', cost_tier=5, name='judge')[source]#

Bases: object

A reference-free, criteria-only LLM judge as a Signal.

Parameters:

  • judge (Callable) – An injected callable (output, *, criteria) -> float. ek core never imports an LLM client; you pass your own (or an ek[judge] adapter).

  • criteria (str) – The rubric the judge scores against, bound at construction (exactly as LogprobSignal binds its aggregator), so the object still satisfies Signal.__call__(output) -> float.

  • cost_tier (int) – JUDGE_COST_TIER – the most expensive tier.

finding(target, rationale)[source]#

Surface a judge’s rationale as a Finding (not just a scalar).

A judge’s reasoning is auditable evidence; a bare number throws it away.

Return type:

Finding

ek.agents.judge.judge_validation(judge_labels, human_labels, *, level='nominal')[source]#

Certify a judge against human labels before trusting it. No new code – reuse the IAA.

A judge is just another annotator, so ek’s existing inter-annotator-agreement machinery (which already tells you the ceiling of a gold standard) is exactly the right instrument: Krippendorff’s alpha is the default (it alone handles any number of raters, any measurement level, and missing data); Cohen’s kappa applies for exactly two raters on complete nominal data. This reproduces the MT-Bench judge-vs-human protocol.

Return type:

dict

Returns:

alpha, kappa, percent_agreement, n, and a verdict against the conventional thresholds (alpha >= 0.8 good, 0.67-0.8 tentative, below that unreliable).

Example

>>> r = judge_validation([1, 1, 0, 0], [1, 1, 0, 0])
>>> r["alpha"], r["verdict"]
(1.0, 'good')
ek.agents.judge.pairwise_judge(judge, a, b, *, criteria='', swap=True)[source]#

Pairwise preference for a over b, corrected for position bias.

Position bias is not a rounding error – judges materially prefer whichever answer they see first. The standard correction is to ask twice with the order swapped and average, which is why this cannot be a plain Signal (it needs two outputs).

Parameters:

  • judge (Callable) – Injected (a, b, *, criteria) -> float in [0, 1], the preference for a.

  • swap (bool) – Ask again with the arguments swapped and average (default; disable only if your judge is known to be order-invariant).

Return type:

float

Returns:

Preference for a in [0, 1] (0.5 = a tie).

Example

>>> first_wins = lambda x, y, *, criteria="": 1.0     # a maximally position-biased judge
>>> pairwise_judge(first_wins, "A", "B")              # swapping exposes it as a tie
0.5