ek.agents.metrics#

Task-success and cost-per-successful-task: the two metrics that drive the whole thing.

Two reference-based metrics over an Episode, both of which fit ek.evaluate() exactly because their corpus aggregate is a ratio of two sums, never a mean of per-item scores – the same discipline that makes ek’s WER accumulate globally and its F1 micro-average (misc/docs/ek_08, misc/docs/ek_11):

  • TaskSuccessMetric – grade the final state, not the surface text. The oracle is injected: ek never owns an LLM or a sandbox, so a checker is a plugged-in strategy (a database-state comparison, a hidden-test run, a programmatic predicate, a judge). Aggregate = the success rate.

  • CostPerSuccessMetric – Cost-of-Pass: E[cost] / P(success). Aggregate = sum(dollars) / sum(successes), diverging to inf when nothing succeeded. A benchmark that scores accuracy without cost lets an agent chase tiny gains with unbounded API calls, so this metric is the point, not a nicety.

A safety property, not a nicety: the checker must run somewhere the agent’s trajectory cannot reach. An audited agent-benchmark red-team produced a ~10-line conftest.py that pytest auto-loads and that rewrote every test outcome to “passed” – scoring near-perfect without solving a single task. Verifier isolation is a security boundary.

Example

>>> from ek.agents.base import Cost, Episode
>>> from ek.agents.cost import per_million
>>> ep = Episode(task_id="t1", output="42", cost=Cost(input_tokens=1_000_000))
>>> TaskSuccessMetric()(ep, "42").value
1.0
>>> m = CostPerSuccessMetric(price=per_million(2.0, 6.0))
>>> round(m(ep, "42").value, 4)          # this episode's dollars
2.0

Functions

final_state_match(episode, gold)

Compare the episode's final state with the goal state (the tau-bench oracle).

output_match(episode, gold)

Canonicalized exact match on the final answer (the GAIA-style oracle).

recorded_success(episode, gold)

Trust a success already recorded on the episode (e.g. by an external harness).

Classes

Checker(*args, **kwargs)

The success oracle: did this episode leave the world in the required state?

CostPerSuccessMetric([check, price, prices])

Cost-of-Pass: dollars per successfully completed task (lower is better).

TaskSuccessMetric([check])

Did the episode complete the task? (1.0/0.0; aggregate = success rate).
class ek.agents.metrics.Checker(*args, **kwargs)[source]#

Bases: Protocol

The success oracle: did this episode leave the world in the required state?

Injected, never owned by ek – a real suite plugs in a database-state comparison, a hidden-test runner, or a programmatic predicate. It must run isolated from the agent.

class ek.agents.metrics.CostPerSuccessMetric(check=None, *, price=None, prices=None)[source]#

Bases: object

Cost-of-Pass: dollars per successfully completed task (lower is better).

The per-episode Score.value is that episode’s dollar cost – the real number is the corpus aggregate, sum(dollars) / sum(successes), which is inf when nothing succeeded (infeasibility is the honest answer; a per-token metric reporting “$0.003/call” for an agent that never succeeds is actively misleading).

Parameters:
aggregate(scores)[source]#

sum(dollars) / sum(successes) – a ratio of two sums, never a mean of ratios.

Return type:

float

class ek.agents.metrics.TaskSuccessMetric(check=None)[source]#

Bases: object

Did the episode complete the task? (1.0/0.0; aggregate = success rate).

Parameters:

check (Any) – A Checker – a registered name ("output", "final_state", "recorded") or any (episode, gold) -> bool callable. Defaults to "output"; inject a state-based or executable oracle for a real suite.

aggregate(scores)[source]#

The success rate over the corpus.

Return type:

float

ek.agents.metrics.final_state_match(episode, gold)[source]#

Compare the episode’s final state with the goal state (the tau-bench oracle).

Return type:

bool

ek.agents.metrics.output_match(episode, gold)[source]#

Canonicalized exact match on the final answer (the GAIA-style oracle).

The weakest useful oracle, and the default only because it needs nothing. Prefer a state-based or executable checker whenever the task has one.

Return type:

bool

ek.agents.metrics.recorded_success(episode, gold)[source]#

Trust a success already recorded on the episode (e.g. by an external harness).

Return type:

bool