ek.agents#

ek.agents – evaluating AI agents and assistants, in cost per successful task.

The agent instance of ek, mirroring ek.ocr. Agent evaluation is the same 2x2 the framework already implements – (reference availability) x (granularity) – but the evaluated object is an episode (tool calls + observations ending in a final state) and the objective is cost per successfully completed task, not cost per token.

corner | agent meaning | facade |

|---|—|---| | reference-based, one item | grade an episode against a gold outcome/trajectory | ek.score() | | reference-based, corpus | task-suite benchmark: pass^k + cost, per slice | run_suite() | | reference-free, one item | judge / verifier / the cascade’s confidence gate | ek.estimate_quality() |

What it reuses (rather than rebuilds): the Layer-A GraphGrammar becomes the **task/tool grammar** (FieldSpec.importance = the cost of a wrong argument, so a wrong argument to a destructive tool is not one unit of error); FieldMetric’s TP/FP/FN accounting powers tool-call scoring; the ROVER agreement seam is self-consistency; the IAA helpers validate a judge; and the whole signal -> calibrate -> validate -> decide cascade is the confidence gate.

Quickstart:

>>> from ek.agents import TaskSpec, run_suite, as_agent
>>> tasks = [TaskSpec("t1", input="hi", gold="HI"), TaskSpec("t2", input="yo", gold="YO")]
>>> report = run_suite(as_agent(str.upper), tasks)
>>> report.success_rate
1.0

Research: misc/docs/ek_07 (map) - ek_08 (task success) - ek_09 (judge/QE) - ek_10 (trajectory) - ek_11 (cost economics) - ek_12 (libraries + integration).

Functions

`tool_grammar`(*tools)	Build the Layer-A `GraphGrammar` from `ToolSpec` s.
`suite_grammar`(tasks)	The union grammar over a whole task suite (later tools win on name collision).
`as_call`(x)	Coerce a `Step`, `(tool, args)` tuple, or `{"tool":…, "args":…}` dict.
`is_episode`(x)	Duck-typed episode check (core never imports this module -- it checks the shape).
`match_calls`(pred, gold, *[, canon])	Assign predicted calls to gold calls -- the layer `FieldMetric` does not provide.
`output_match`(episode, gold)	Canonicalized exact match on the final answer (the GAIA-style oracle).
`final_state_match`(episode, gold)	Compare the episode's final state with the goal state (the tau-bench oracle).
`recorded_success`(episode, gold)	Trust a `success` already recorded on the episode (e.g. by an external harness).
`pass_at_k`(*, n, c, k)	Unbiased `pass@k`: probability that at least one of k samples succeeds.
`pass_hat_k`(*, n, c, k)	Unbiased `pass^k`: probability that all k independent trials succeed.
`reliability`(source, *[, k, slices, cost, ...])	Compute `pass@k` / `pass^k` / success-rate over a k-trial run.
`wilson_interval`(successes, n, *[, z])	Wilson score interval for a proportion -- correct in the tails, unlike Wald/CLT.
`bootstrap_ci`(values, statistic, *[, ...])	Percentile bootstrap CI for any `statistic` over `values` (deterministic by seed).
`newcombe_difference`(successes_a, n_a, ...[, z])	Newcombe score interval for the difference of two proportions `p_a - p_b`.
`difference_upper_bound`(value_a, ci_a, ...[, z])	Upper bound of `a - b` for two statistics known only through their CIs.
`per_million`(input, output, *[, ...])	Build a `ModelPrice` from the human-facing USD per 1M tokens figures.
`dollars`(cost, price, *[, prices, batch])	Monetary cost of one `Cost` tally, in USD.
`episode_dollars`(episode, *[, prices, price, ...])	Monetary cost of one episode (0.0 if it carries no `Cost`).
`cost_of_pass`(total_cost, n_success)	Expected monetary cost of one successful task: `total_cost / n_success`.
`cost_report`(episodes, *[, prices, price])	The quality x cost x latency triple, reported together.
`load_prices`(source)	Parse a LiteLLM-format price catalog into a `{model: ModelPrice}` table.
`price_of`(model, *[, prices])	Resolve a model's rates from an injected catalog, or fail actionably.
`judge_validation`(judge_labels, human_labels, *)	Certify a judge against human labels before trusting it.
`pairwise_judge`(judge, a, b, *[, criteria, swap])	Pairwise preference for `a` over `b`, corrected for position bias.
`run_suite`(agent, tasks, *[, k, check, ...])	Run `agent` over a task suite, `k` trials per task, and report reliability + cost.
`save_agent_baseline`(report, name, *[, rootdir])	Freeze a run as a named baseline (including its provenance, so the gate can guard it).
`load_agent_baseline`(name, *[, rootdir])	Load a named agent baseline (or `None`).
`agent_regression_gate`(report, baseline, *[, ...])	Fail only when the evidence says the agent really got worse -- not on noise.
`as_agent`(fn)	Wrap a plain `input -> answer` function into the harness's `TaskSpec -> Episode` shape.
`trajectory_from_messages`(messages)	Parse a chat transcript into a `Trajectory`.
`episode_from_messages`(messages, *[, ...])	Build a full `Episode` from a transcript (+ optional usage/state).
`cost_from_usage`(usage, *[, latency_s])	Build a `Cost` from a provider `usage` object or dict.
`from_inspect_sample`(sample, *[, task_id])	Adapt an Inspect AI `EvalSample`-shaped object into an `Episode` (duck-typed).
`from_deepeval_test_case`(case, *[, task_id])	Adapt a DeepEval `LLMTestCase`-shaped object into an `Episode` (duck-typed).

Classes

`TaskSpec`(task_id[, input, gold, value, ...])	One task in a suite: its input, its oracle, its value, and its allowed tools.
`ToolSpec`(name[, params, importance, destructive])	One tool the agent may call: its argument schema and its error costs.
`Episode`([task_id, trajectory, output, ...])	Layer B: one agent run over one task -- the object agent evaluation scores.
`Trajectory`([steps])	The ordered sequence of `Step` s an agent took.
`Step`(tool[, args, observation, error])	One step of a trajectory: a tool call and what came back.
`Cost`([input_tokens, output_tokens, ...])	What one episode consumed: tokens (by kind), retries, and wall-clock.
`RunProvenance`([seed, temperature, model, ...])	The hidden eval variables that decide whether two runs are even comparable.
`TaskSuccessMetric`([check])	Did the episode complete the task? (`1.0`/`0.0`; aggregate = success rate).
`CostPerSuccessMetric`([check, price, prices])	Cost-of-Pass: dollars per successfully completed task (lower is better).
`ToolCallMetric`([level, canonicalizer, grammar])	Tool-call correctness (BFCL AST match) as a cost-weighted `Metric`.
`TrajectoryMetric`([scheme, grammar, ...])	Cost-weighted trajectory distance as a `Metric` (lower is better).
`JudgeMetric`(judge[, criteria])	A reference-based LLM judge as a `Metric` (the `score()` side).
`Checker`(args, *kwargs)	The success oracle: did this episode leave the world in the required state?
`ReliabilityReport`([k, n_tasks, n_trials, ...])	Corpus-level reliability over a k-trial run, with per-task and per-slice cuts.
`ModelPrice`([input, output, cached_input, ...])	Per-token rates for one model, in USD per token.
`JudgeSignal`(judge[, criteria, cost_tier, name])	A reference-free, criteria-only LLM judge as a `Signal`.
`AgentGateResult`([passed, reasons, ...])	Outcome of a variance-aware agent regression check.

Exceptions

UnknownModelPrice

Raised when a model's rates are not in the injected price catalog.

class ek.agents.AgentGateResult(passed=True, reasons=<factory>, pass_hat_k=None, pass_hat_k_baseline=None, success_rate=None, success_rate_baseline=None, cost_per_success=None, cost_per_success_baseline=None, underpowered=False)[source]#

Bases: object

Outcome of a variance-aware agent regression check.

class ek.agents.Checker(*args, **kwargs)[source]#

Bases: Protocol

The success oracle: did this episode leave the world in the required state?

Injected, never owned by ek – a real suite plugs in a database-state comparison, a hidden-test runner, or a programmatic predicate. It must run isolated from the agent.

class ek.agents.Cost(input_tokens=0, output_tokens=0, cached_input_tokens=0, reasoning_tokens=0, retries=0, latency_s=None)[source]#

Bases: object

What one episode consumed: tokens (by kind), retries, and wall-clock.

Token kinds are tracked separately because they are priced asymmetrically – output typically costs ~5x input, cached input bills at a fraction, and reasoning tokens are billed. This is precisely why the objective is dollars per successful task and never a raw token count (misc/docs/ek_11). Pricing lives in ek.agents.cost; this dataclass is the provider-neutral tally.

Example

>>> a = Cost(input_tokens=100, output_tokens=50)
>>> b = Cost(input_tokens=10, output_tokens=5, retries=1)
>>> c = a + b
>>> c.input_tokens, c.output_tokens, c.retries
(110, 55, 1)

property total_tokens: int#: All billed tokens (a rough size proxy – not a cost; see dollars()).

class ek.agents.CostPerSuccessMetric(check=None, *, price=None, prices=None)[source]#

Bases: object

Cost-of-Pass: dollars per successfully completed task (lower is better).

The per-episode Score.value is that episode’s dollar cost – the real number is the corpus aggregate, sum(dollars) / sum(successes), which is inf when nothing succeeded (infeasibility is the honest answer; a per-token metric reporting “$0.003/call” for an agent that never succeeds is actively misleading).

Parameters:

check (Any) – The success oracle (as in TaskSuccessMetric).
price (Optional[ModelPrice]) – A ModelPrice applied to every episode, or None to resolve each episode’s model against prices.
prices (Optional[Mapping[str, ModelPrice]]) – A {model: ModelPrice} catalog (see ek.agents.load_prices()).

aggregate(scores)[source]#

sum(dollars) / sum(successes) – a ratio of two sums, never a mean of ratios.

Return type:: float

class ek.agents.Episode(task_id='', trajectory=<factory>, output=None, final_state=None, cost=None, success=None, run=None, provenance=None, meta=<factory>)[source]#

Bases: object

Layer B: one agent run over one task – the object agent evaluation scores.

The direct analog of AnnotatedExtraction: it rides alongside the frozen Layer-A grammar and carries the run’s metadata. It is scored offline against gold (ek.score() / ek.evaluate()) and estimated online against a judge or a consensus (ek.estimate_quality()) – the same object, both halves.

Parameters:

task_id (str) – Which task this episode answers (the k-trial grouping key).
trajectory (Trajectory) – The tool calls taken.
output (Any) – The agent’s final user-facing answer.
final_state (Any) – The end state of the world (the DB, the filesystem) – what a state-based oracle actually grades. Success is a final-state check, not a surface-text check.
cost (Optional[Cost]) – The Cost consumed.
success (Optional[bool]) – Filled in by a checker; None until graded.
run (Optional[RunProvenance]) – The RunProvenance (seed, model, simulator, suite version).
provenance (Optional[Provenance]) – Optional core Provenance for click-to-source audit.
meta (dict) – Anything else the bridge captured.

property calls: tuple#: The (tool, args) pairs taken, in order.

graded(success)[source]#

A copy with success set (never mutates – grading stays idempotent).

meta is copied too: dataclasses.replace is a shallow copy, so without this the “copy” would share the original’s mutable meta dict.

Return type:: Episode

class ek.agents.JudgeMetric(judge, criteria='')[source]#

Bases: object

A reference-based LLM judge as a Metric (the score() side).

A judge that reads the gold answer is not a reference-free signal, so it does not belong in ek.estimate_quality(). It lives here.

Parameters:

judge (Callable) – Injected (output, *, criteria, reference) -> float in [0, 1].
criteria (str) – The rubric.

aggregate(scores)[source]#

Mean judge score (a judge score is per-item; there is no global accumulation).

Return type:: float

class ek.agents.JudgeSignal(judge, criteria='', cost_tier=5, name='judge')[source]#

Bases: object

A reference-free, criteria-only LLM judge as a Signal.

Parameters:

judge (Callable) – An injected callable (output, *, criteria) -> float. ek core never imports an LLM client; you pass your own (or an ek[judge] adapter).
criteria (str) – The rubric the judge scores against, bound at construction (exactly as LogprobSignal binds its aggregator), so the object still satisfies Signal.__call__(output) -> float.
cost_tier (int) – JUDGE_COST_TIER – the most expensive tier.

finding(target, rationale)[source]#

Surface a judge’s rationale as a Finding (not just a scalar).

A judge’s reasoning is auditable evidence; a bare number throws it away.

Return type:: Finding

class ek.agents.ModelPrice(input=0.0, output=0.0, cached_input=None, reasoning=None)[source]#

Bases: object

Per-token rates for one model, in USD per token.

Parameters:

input (float) – USD per input (prompt) token.
output (float) – USD per output (completion) token.
cached_input (Optional[float]) – USD per cached-input token; defaults to DEFAULT_CACHE_DISCOUNT * input when not declared.
reasoning (Optional[float]) – USD per reasoning token; defaults to the output rate (reasoning tokens are billed as output by the major providers).

property cached_rate: float#: The cached-input rate (explicit, else the discounted input rate).

property reasoning_rate: float#: The reasoning-token rate (explicit, else the output rate).

class ek.agents.ReliabilityReport(k=1, n_tasks=0, n_trials=0, n_success=0, success_rate=0.0, success_ci=(0.0, 1.0), pass_at_k=None, pass_hat_k=None, pass_hat_k_ci=(0.0, 1.0), stochastic=True, per_task=<factory>, per_slice=<factory>, cost=<factory>, detail=<factory>)[source]#

Bases: object

Corpus-level reliability over a k-trial run, with per-task and per-slice cuts.

success_rate is micro (over all trials); pass_at_k/pass_hat_k are averaged over tasks (each task’s own unbiased estimator), never over trials.

class ek.agents.RunProvenance(seed=None, temperature=None, model='', simulator='', suite_version='', scaffold='')[source]#

Bases: object

The hidden eval variables that decide whether two runs are even comparable.

Agent evaluation is stochastic, and several of its inputs are themselves models. In a tau-bench-style suite the user simulator is an LLM agent, so its model/prompt/version silently changes results run to run. Recording these is what lets the harness refuse to compare a baseline against a run that changed the simulator or the suite – the same discipline ek’s offline harness already applies when it refuses to compare across two different metrics (misc/docs/ek_08, misc/docs/ek_11).

Parameters:

seed (Optional[int]) – RNG seed for the trial (None = unseeded/nondeterministic).
temperature (Optional[float]) – Sampling temperature of the agent under test.
model (str) – The agent’s model id.
simulator (str) – The user-simulator id/version – an eval variable, not a constant.
suite_version (str) – Version of the task suite (contamination and task fixes move scores).
scaffold (str) – The agent scaffold/harness id (the same model moves points on scaffold alone).

comparability_key()[source]#

The fields that must match for two runs to be comparable (seed excluded).

Seed and temperature are expected to vary across trials; the simulator, the suite version and the scaffold are not – changing them invalidates a baseline.

Return type:: tuple

class ek.agents.Step(tool, args=<factory>, observation=None, error=None)[source]#

Bases: object

One step of a trajectory: a tool call and what came back.

error is set when the call itself failed (a bad argument, a tool exception) – error recovery across subsequent steps is itself an evaluable signal.

property call: tuple#: The (tool, args) pair – what tool-call correctness actually compares.

class ek.agents.TaskSpec(task_id, input=None, gold=None, value=1.0, tools=(), slice=None)[source]#

Bases: object

One task in a suite: its input, its oracle, its value, and its allowed tools.

Parameters:

task_id (str) – Stable identifier (the grouping key for k-trial reliability).
input (Any) – Whatever the agent under test is called with.
gold (Any) – The reference the checker grades against (a goal state, an expected answer, a gold trajectory – the checker decides how to read it).
value (float) – Task-level cost weight: how much a completed task is worth. The task-level analog of FieldSpec.importance; feeds value-weighted cost accounting.
tools (Sequence[ToolSpec]) – The tools this task permits (the Layer-A grammar for the task).
slice (Optional[str]) – Optional slice label (domain, difficulty, language) – per-slice cuts are mandatory, not optional (a low pass^k concentrated in one hard slice is a different problem from uniform flakiness).

grammar()[source]#

The Layer-A grammar for this task’s tool set.

Return type:: GraphGrammar

class ek.agents.TaskSuccessMetric(check=None)[source]#

Bases: object

Did the episode complete the task? (1.0/0.0; aggregate = success rate).

Parameters:: check (Any) – A Checker – a registered name ("output", "final_state", "recorded") or any (episode, gold) -> bool callable. Defaults to "output"; inject a state-based or executable oracle for a real suite.

aggregate(scores)[source]#

The success rate over the corpus.

Return type:: float

class ek.agents.ToolCallMetric(level='call', *, canonicalizer=None, grammar=None)[source]#

Bases: object

Tool-call correctness (BFCL AST match) as a cost-weighted Metric.

Parameters:

level (str) – "call" (default) scores whole calls – a call is a TP only on an exact AST match; "arg" gives partial credit at argument granularity.
canonicalizer – str -> str applied to string arguments before comparison (BFCL’s case/whitespace folding). Defaults to ek’s default canonicalizer.
grammar (Optional[GraphGrammar]) – Layer-A grammar supplying per-tool and per-argument cost weights (may also be passed per call).

The Score carries tp/fp/fn in detail so ek.evaluate() aggregates a correct micro-averaged F1 over the corpus rather than averaging per-episode F1s.

aggregate(scores)[source]#

Micro-averaged F1 over the corpus (sum TP/FP/FN, then divide) – never a mean.

Return type:: float

class ek.agents.ToolSpec(name, params=<factory>, importance=None, destructive=False)[source]#

Bases: object

One tool the agent may call: its argument schema and its error costs.

Parameters:

name (str) – The tool/function name (the AST-match ground truth).
params (Mapping[str, FieldSpec]) – Argument name -> FieldSpec. Each spec’s importance is the cost of getting that argument wrong, and its domain feeds validators – the same contract as an extracted field.
importance (Optional[float]) – Cost weight of the call itself (a missing or spurious call). Defaults to DESTRUCTIVE_WEIGHT when destructive is set, else 1.0.
destructive (bool) – Whether calling this tool mutates the world irreversibly (a refund, a delete, a send). Drives the default weight and the safety validators.

to_node_type()[source]#

Render this tool as a Layer-A NodeType (args as fields).

Return type:: NodeType

property weight: float#: The call-level cost weight (explicit, else destructive-aware default).

class ek.agents.Trajectory(steps=())[source]#

Bases: object

The ordered sequence of Step s an agent took.

A trajectory is linear – which is why it is scored with a sequence edit distance, not a graph edit distance (see ek.agents.trajectory).

Example

>>> t = Trajectory([Step("search", {"q": "cat"}), Step("answer", {"a": "meow"})])
>>> len(t), t.tools
(2, ('search', 'answer'))

property calls: tuple#: The (tool, args) pairs in call order.

property tools: tuple#: The tool names in call order.

class ek.agents.TrajectoryMetric(scheme='in_order', *, grammar=None, canonicalizer=None)[source]#

Bases: object

Cost-weighted trajectory distance as a Metric (lower is better).

Parameters:

scheme (str) – One of SCHEMES. in_order (default) – Needleman-Wunsch edit distance; order matters, gaps allowed. exact – 0 iff the call sequences are identical, else the full mass. any_order – multiset distance; order ignored (the calls, not the path). superset – 0 iff every gold call occurs in the prediction; extra steps are free (use when the reference lists required calls rather than the whole path).
grammar (Optional[GraphGrammar]) – Layer-A grammar supplying step/argument cost weights (may be passed per call).
canonicalizer – str -> str applied to string arguments before comparison.

aggregate(scores)[source]#

Corpus distance = total raw distance / total maximum distance (never a mean).

Return type:: float

exception ek.agents.UnknownModelPrice[source]#

Bases: KeyError

Raised when a model’s rates are not in the injected price catalog.

ek.agents.agent_regression_gate(report, baseline, *, tolerance=0.0, cost_tolerance=0.0, rootdir=None)[source]#

Fail only when the evidence says the agent really got worse – not on noise.

Tests the difference current - baseline, with both runs’ uncertainty folded in: a Newcombe score interval on the success rate, and (for k > 1) a combined-SE bound on pass^k. A regression is declared only when the upper bound of that change is below -tolerance – i.e. we are confident the true change is negative.

Two tempting shortcuts are both wrong, and this module has shipped and fixed each:

Comparing our interval to the baseline’s point. The baseline is itself a noisy estimate; a Wilson upper bound only reaches 1.0 when every trial passed, so a single flake against a lucky baseline reads as a “confident regression”.
Asking whether the two 95% intervals overlap. That is not a 5% test but roughly a 0.5% one: it buys a calm gate by going blind (a 15-point drop on a 60-task suite becomes invisible). A gate that cannot see a real regression is as broken as one that cries wolf.

A result that is merely worse but not significantly so is not a clean pass either – it is an underpowered experiment, and the result says so (underpowered, plus a warning).

Raises:: ValueError – if the baseline was produced with a different user simulator, suite version, or scaffold – those runs are not comparable, and silently comparing them would be a category error (the same reason the offline gate refuses to compare two different metrics); if the baseline recorded zero trials (a permanent free pass); or if it is malformed.
Return type:: AgentGateResult

ek.agents.as_agent(fn)[source]#

Wrap a plain input -> answer function into the harness’s TaskSpec -> Episode shape.

Progressive disclosure: the trivial agent should not have to learn the Episode type.

Return type:: Callable

Example

>>> from ek.agents.base import TaskSpec
>>> agent = as_agent(str.upper)
>>> agent(TaskSpec("t1", input="hi")).output
'HI'

ek.agents.as_call(x)[source]#

Coerce a Step, (tool, args) tuple, or {"tool":…, "args":…} dict.

Gold tool calls are usually written as bare pairs or dicts (no observation), while a prediction arrives as a Step; both must compare. Progressive disclosure: the simple literal works, the rich object works.

Return type:: tuple

Example

>>> as_call(Step("search", {"q": "cat"}))
('search', {'q': 'cat'})
>>> as_call(("search", {"q": "cat"}))
('search', {'q': 'cat'})
>>> as_call({"tool": "search", "args": {"q": "cat"}})
('search', {'q': 'cat'})
>>> as_call("search")
('search', {})

ek.agents.bootstrap_ci(values, statistic, *, resamples=2000, alpha=0.05, seed=0)[source]#

Percentile bootstrap CI for any statistic over values (deterministic by seed).

Used for the cost-per-success ratio, whose distribution is skewed and heavy-tailed (a normal interval on it is meaningless).

Return type:: tuple

Example

>>> lo, hi = bootstrap_ci([1.0] * 20, lambda v: sum(v) / len(v), resamples=50)
>>> lo == hi == 1.0
True

ek.agents.cost_from_usage(usage, *, latency_s=None)[source]#

Build a Cost from a provider usage object or dict.

Understands the OpenAI (prompt_tokens / completion_tokens) and Anthropic (input_tokens / output_tokens / cache_read_input_tokens) spellings.

Return type:: Cost

Example

>>> c = cost_from_usage({"prompt_tokens": 100, "completion_tokens": 20})
>>> c.input_tokens, c.output_tokens
(100, 20)

ek.agents.cost_of_pass(total_cost, n_success)[source]#

Expected monetary cost of one successful task: total_cost / n_success.

Returns inf when nothing succeeded – infeasibility is the honest answer, not a cheap-looking zero (this divergence is the point of the metric).

Return type:: float

Example

>>> cost_of_pass(10.0, 4)
2.5
>>> cost_of_pass(10.0, 0)
inf

ek.agents.cost_report(episodes, *, prices=None, price=None)[source]#

The quality x cost x latency triple, reported together.

Scoring accuracy without cost lets an agent chase tiny gains with unbounded API calls, so a report that omits cost is not a report. Returns n, n_success, success_rate, total_dollars, cost_per_success (Cost-of-Pass), mean_latency_s and total_tokens.

With no price/prices, dollar figures are None rather than a fabricated zero: ek will not invent rates, and a silent 0.0 would make a costly agent look free. Tokens and latency are still reported.

Return type:: dict

Example

>>> from ek.agents.base import Cost, Episode
>>> p = per_million(1.0, 1.0)
>>> eps = [Episode(task_id="a", cost=Cost(input_tokens=1_000_000), success=True),
...        Episode(task_id="b", cost=Cost(input_tokens=1_000_000), success=False)]
>>> r = cost_report(eps, price=p)
>>> r["success_rate"], round(r["cost_per_success"], 2)
(0.5, 2.0)
>>> cost_report(eps)["cost_per_success"] is None      # no rates -> no dollars invented
True

ek.agents.difference_upper_bound(value_a, ci_a, value_b, ci_b, *, z=1.96)[source]#

Upper bound of a - b for two statistics known only through their CIs.

Used for pass^k (a mean of per-task estimators, not a raw proportion), where a Newcombe interval does not apply but the same principle does: test the difference, combining both runs’ uncertainty, rather than asking whether two intervals happen to overlap.

Return type:: Optional[float]

ek.agents.dollars(cost, price, *, prices=None, batch=False)[source]#

Monetary cost of one Cost tally, in USD.

Parameters:

cost (Cost) – The token/retry tally.
price (Union[ModelPrice, str]) – A ModelPrice, or a model name to resolve against prices.
prices (Optional[Mapping[str, ModelPrice]]) – The injected price catalog (required when price is a name).
batch (bool) – Apply the batch-API discount (DEFAULT_BATCH_DISCOUNT).

Raises:

UnknownModelPrice – if a model name is given with no rates for it.

Return type:

float

ek.agents.episode_dollars(episode, *, prices=None, price=None, batch=False)[source]#

Monetary cost of one episode (0.0 if it carries no Cost).

Return type:: float

ek.agents.episode_from_messages(messages, *, task_id='', usage=None, output=None, final_state=None, latency_s=None)[source]#

Build a full Episode from a transcript (+ optional usage/state).

output defaults to the last assistant text – the agent’s final answer.

Return type:: Episode

ek.agents.final_state_match(episode, gold)[source]#

Compare the episode’s final state with the goal state (the tau-bench oracle).

Return type:: bool

ek.agents.from_deepeval_test_case(case, *, task_id='')[source]#

Adapt a DeepEval LLMTestCase-shaped object into an Episode (duck-typed).

Reads .actual_output and .tools_called. Note DeepEval phones home by default (Confident-AI telemetry) – disable it before use if that matters to you; ek never enables it.

Return type:: Episode

ek.agents.from_inspect_sample(sample, *, task_id='')[source]#

Adapt an Inspect AI EvalSample-shaped object into an Episode (duck-typed).

Reads .messages, .output and .id – no inspect_ai import required, so scoring an Inspect log never drags the harness into ek’s dependency closure.

Return type:: Episode

ek.agents.is_episode(x)[source]#

Duck-typed episode check (core never imports this module – it checks the shape).

Return type:: bool

ek.agents.judge_validation(judge_labels, human_labels, *, level='nominal')[source]#

Certify a judge against human labels before trusting it. No new code – reuse the IAA.

A judge is just another annotator, so ek’s existing inter-annotator-agreement machinery (which already tells you the ceiling of a gold standard) is exactly the right instrument: Krippendorff’s alpha is the default (it alone handles any number of raters, any measurement level, and missing data); Cohen’s kappa applies for exactly two raters on complete nominal data. This reproduces the MT-Bench judge-vs-human protocol.

Return type:: dict
Returns:: alpha, kappa, percent_agreement, n, and a verdict against the conventional thresholds (alpha >= 0.8 good, 0.67-0.8 tentative, below that unreliable).

Example

>>> r = judge_validation([1, 1, 0, 0], [1, 1, 0, 0])
>>> r["alpha"], r["verdict"]
(1.0, 'good')

ek.agents.load_agent_baseline(name, *, rootdir=None)[source]#

Load a named agent baseline (or None).

Return type:: Optional[dict]

ek.agents.load_prices(source)[source]#

Parse a LiteLLM-format price catalog into a {model: ModelPrice} table.

source is a path to (or an already-parsed mapping of) LiteLLM’s MIT-licensed model_prices_and_context_window.json. We read the data file only and never import the litellm SDK – its enterprise/ subtree is a proprietary carve-out, and the SDK drags a large transitive tree in just to look up a rate.

Entries lacking token rates (embeddings, unpriced models) are skipped.

Return type:: dict

Example

>>> table = load_prices({"m": {"input_cost_per_token": 1e-6,
...                            "output_cost_per_token": 2e-6}})
>>> table["m"].output
2e-06

ek.agents.match_calls(pred, gold, *, canon=None)[source]#

Assign predicted calls to gold calls – the layer FieldMetric does not provide.

Calls are grouped by tool name (a call to the wrong tool is never a match). Within a tool name, candidate pairs are scored and taken globally best-first (exact-argument matches before partial ones, more agreeing arguments before fewer), so identical repeated calls are interchangeable and a reordered pair is still matched – while a strong pairing can never be stolen by a weaker one that merely happened to come first in the prediction.

This is a deterministic greedy approximation to the optimal assignment. It is exact for the common cases (distinct calls, repeats, reorderings); a full min-cost bipartite matching (Hungarian) could differ only on pathological many-to-many partial-overlap sets, and is not worth a dependency here.

Returns (pairs, unmatched_pred, unmatched_gold), pairs being (pred_call, gold_call).

Return type:: tuple

Example

>>> pairs, up, ug = match_calls([("s", {"q": "b"}), ("s", {"q": "a"})],
...                             [("s", {"q": "a"}), ("s", {"q": "b"})])
>>> len(pairs), up, ug          # order-insensitive within a tool name
(2, [], [])

ek.agents.newcombe_difference(successes_a, n_a, successes_b, n_b, *, z=1.96)[source]#

Newcombe score interval for the difference of two proportions p_a - p_b.

This is the right instrument for “did the agent get worse than its baseline?”, and it is strictly better than the tempting shortcut of checking whether two 95% CIs overlap. Non-overlap is not a 5% test – it is roughly a 0.5% one, so it buys its low false-alarm rate by going nearly blind: on a 60-task suite it cannot see a 15-point success-rate drop. A difference interval keeps the same false-alarm rate and recovers most of the power, because the variance of a difference is not the sum of two interval widths.

Newcombe (1998) method 10: build each proportion’s Wilson interval, then

lower = (p_a - p_b) - sqrt((p_a - l_a)^2 + (u_b - p_b)^2) upper = (p_a - p_b) + sqrt((u_a - p_a)^2 + (p_b - l_b)^2)

Returns (lower, upper). The difference is a confident regression when upper < 0.

Example

>>> lo, hi = newcombe_difference(50, 100, 90, 100)   # 50% now vs 90% before
>>> hi < 0                                            # confidently worse
True
>>> lo, hi = newcombe_difference(89, 100, 90, 100)   # 89% vs 90% -- noise
>>> hi > 0
True

Return type:: tuple

ek.agents.output_match(episode, gold)[source]#

Canonicalized exact match on the final answer (the GAIA-style oracle).

The weakest useful oracle, and the default only because it needs nothing. Prefer a state-based or executable checker whenever the task has one.

Return type:: bool

ek.agents.pairwise_judge(judge, a, b, *, criteria='', swap=True)[source]#

Pairwise preference for a over b, corrected for position bias.

Position bias is not a rounding error – judges materially prefer whichever answer they see first. The standard correction is to ask twice with the order swapped and average, which is why this cannot be a plain Signal (it needs two outputs).

Parameters:

judge (Callable) – Injected (a, b, *, criteria) -> float in [0, 1], the preference for a.
swap (bool) – Ask again with the arguments swapped and average (default; disable only if your judge is known to be order-invariant).

Return type:

float

Returns:

Preference for a in [0, 1] (0.5 = a tie).

Example

>>> first_wins = lambda x, y, *, criteria="": 1.0     # a maximally position-biased judge
>>> pairwise_judge(first_wins, "A", "B")              # swapping exposes it as a tie
0.5

ek.agents.pass_at_k(*, n, c, k)[source]#

Unbiased pass@k: probability that at least one of k samples succeeds.

The HumanEval estimator 1 - C(n-c, k) / C(n, k) over n trials with c successes. This is the capability metric – right when a single success is enough (offline candidate generation behind a verifier).

Return type:: float

Example

>>> round(pass_at_k(n=10, c=1, k=1), 3)
0.1
>>> pass_at_k(n=10, c=10, k=5)
1.0

ek.agents.pass_hat_k(*, n, c, k)[source]#

Unbiased pass^k: probability that all k independent trials succeed.

The tau-bench estimator C(c, k) / C(n, k); for a per-trial success probability p it decays to p**k. This is the reliability metric – the one that matters when consistency is the product.

Return type:: float

Example

>>> pass_hat_k(n=8, c=8, k=8)
1.0
>>> pass_hat_k(n=8, c=4, k=2)
0.21428571428571427

ek.agents.per_million(input, output, *, cached_input=None, reasoning=None)[source]#

Build a ModelPrice from the human-facing USD per 1M tokens figures.

Return type:: ModelPrice

Example

>>> p = per_million(3.0, 15.0)          # $3 in / $15 out per 1M tokens
>>> round(p.input, 9), round(p.output, 9)
(3e-06, 1.5e-05)

ek.agents.price_of(model, *, prices=None)[source]#

Resolve a model’s rates from an injected catalog, or fail actionably.

ek deliberately ships no built-in price table (rates go stale monthly and would be magic numbers). Supply one with prices=, or load the LiteLLM catalog with load_prices().

Return type:: ModelPrice

ek.agents.recorded_success(episode, gold)[source]#

Trust a success already recorded on the episode (e.g. by an external harness).

Return type:: bool

ek.agents.reliability(source, *, k=1, slices=None, cost=None, warn_deterministic=True)[source]#

Compute pass@k / pass^k / success-rate over a k-trial run.

Parameters:

source (Any) – Graded Episode s (grouped by task_id), or an already-grouped {task_id: [bool, ...]} mapping.
k (int) – The k of pass@k / pass^k. Tasks with fewer than k trials are skipped (and counted in detail['skipped']) rather than silently mis-estimated.
slices (Optional[Mapping[str, str]]) – Optional {task_id: slice_label} for per-slice cuts.
cost (Optional[Mapping]) – An optional cost report (from ek.agents.cost_report()) to carry along – reliability without cost is only half the picture.
warn_deterministic (bool) – Warn when no task shows mixed outcomes across trials, in which case pass^k degenerates to pass^1 and is not informative.

Return type:

ReliabilityReport

Example

>>> r = reliability({"t1": [True, True, False], "t2": [True, True, True]}, k=2)
>>> r.n_tasks, r.k
(2, 2)
>>> round(r.pass_hat_k, 3)      # t1: C(2,2)/C(3,2)=1/3 ; t2: 1.0
0.667

ek.agents.run_suite(agent, tasks, *, k=1, check=None, metrics=None, price=None, prices=None, run=None, seed=None, persist=False, run_id=None, rootdir=None)[source]#

Run agent over a task suite, k trials per task, and report reliability + cost.

Parameters:

agent (Callable[[TaskSpec], Any]) – The system under test: TaskSpec -> Episode (a bare answer is wrapped). It receives the whole spec – .input, .tools, .gold – not just the input. If it accepts a keyword seed, the per-trial seed is passed to it.
tasks (Any) – TaskSpec s (or a {task_id: spec} mapping).
k (int) – Trials per task. k > 1 is what makes pass^k meaningful – and it only means anything if the agent is genuinely stochastic (the report warns if it is not).
check (Any) – The success oracle – a registered checker name or an (episode, gold) -> bool callable. Defaults to "output"; inject a state-based/executable oracle for a real suite, and run it isolated from the agent.
metrics (Optional[Mapping]) – Optional {name: Metric} scored on every episode against task.gold, with the suite’s Layer-A grammar injected – this is how the tool/argument cost weights reach a tool_call/trajectory metric. Results land in report.detail["metrics"].
price/prices – Rates for costing the episodes (see ek.agents.cost).
run (Optional[RunProvenance]) – RunProvenance for this run (model, simulator, suite version, scaffold). Recorded on every episode and on any baseline saved from it.
seed (Optional[int]) – Base RNG seed. Trial i of each task uses seed + i: it is recorded on the episode’s provenance and passed to the agent when the agent accepts a seed keyword.
persist/run_id/rootdir – Persist the run to the runs/results stores.

Return type:

ReliabilityReport

Returns:

A ReliabilityReport carrying pass@k, pass^k (with a bootstrap CI), the success rate (with a Wilson CI), the per-slice cuts, and the cost report – because reliability without cost is only half the answer.

ek.agents.save_agent_baseline(report, name, *, rootdir=None)[source]#

Freeze a run as a named baseline (including its provenance, so the gate can guard it).

Raises:: ValueError – if the run evaluated nothing. An empty baseline is worse than no baseline – it is a permanent free pass, because every bound in it is the maximally-uncertain default and no later run can be shown to fall below it.
Return type:: dict

ek.agents.suite_grammar(tasks)[source]#

The union grammar over a whole task suite (later tools win on name collision).

Return type:: GraphGrammar

ek.agents.tool_grammar(*tools)[source]#

Build the Layer-A GraphGrammar from ToolSpec s.

The resulting grammar is the frozen cost SSOT every agent metric reads: node cost = the tool’s call weight, field cost = the argument’s importance.

Return type:: GraphGrammar

Example

>>> from ek.base import FieldSpec
>>> g = tool_grammar(ToolSpec("send", params={"to": FieldSpec("to")},
...                           destructive=True))
>>> g.node_cost("send")
10.0

ek.agents.trajectory_from_messages(messages)[source]#

Parse a chat transcript into a Trajectory.

Each assistant tool call becomes a Step, and the next tool/result message becomes that step’s observation. A tool message carrying an error marks the step as errored – error recovery across later steps is itself an evaluable signal.

Return type:: Trajectory

ek.agents.wilson_interval(successes, n, *, z=1.96)[source]#

Wilson score interval for a proportion – correct in the tails, unlike Wald/CLT.

The naive normal (“Wald”) interval is badly wrong at small n and near 0/1 – exactly where agent success rates live. Wilson is the cheap, dependency-free fix.

Return type:: tuple

Example

>>> lo, hi = wilson_interval(0, 10)
>>> lo == 0.0, round(hi, 3)
(True, 0.278)

Modules

`base`	Agent-evaluation data shapes: the episode (Layer B) and the task/tool grammar (Layer A).
`bridge`	One-way adapters: turn whatever your agent already emits into an `ek` `Episode`.
`cost`	The cost model and the price SSOT: dollars per successfully completed task.
`harness`	The agent task-suite harness: k trials, per-slice cuts, and a gate that survives noise.
`judge`	LLM-as-judge: a reference-free `Signal`, a reference-based `Metric`, and the validation that must precede either.
`metrics`	Task-success and cost-per-successful-task: the two metrics that drive the whole thing.
`toolcalls`	Tool-call correctness: BFCL-style AST matching over a multiset of calls.
`trajectory`	Trajectory (process) evaluation: a cost-weighted sequence edit distance over steps.