> built 2026-10-14 03:58 UTC from 3c8454e (main) · ek 0.1.11. Details: build_info.json # index.html.md # ek **ek (Evaluation Kit) — a framework for building Knowledge Evaluation systems**, evaluating the outputs of information-extraction systems. OCR is treated as the noisiest *special case* of a general problem, so the core is source-agnostic and the OCR pieces are optional. ```python import ek ek.score("hello wrld", "hello world") # -> Score(value=0.0909..., metric='cer') ek.score("hello wrld", "hello world", metric="wer").value # 0.5 ek.evaluate( [("ct", "cat"), ("dg", "dog")], metric="cer" ).aggregate # 0.333... (global CER) ``` ## What it does Evaluating an extraction splits along two axes — is there a gold answer (*reference-based*) or not (*reference-free*), and are we scoring one item or a whole corpus. `ek` gives you both halves through two facades over one shared typed schema: - **`score()` / `evaluate()`** — *reference-based*: compare against gold, one item or a corpus, the metric chosen by output type (string → CER/WER, record → field-F1), aggregated *correctly* (global error-rate accumulation, micro-F1; never a naive mean) with optional per-slice cuts. - **`estimate_quality()`** — *reference-free*: gather signals → calibrate → validate → decide accept/flag/block, with no gold answer. Everything swappable is a strategy injected with a smart default, so the simple call works out of the box and every layer stays replaceable. ## Evaluate an OCR engine The first concrete instance: measure OCR accuracy over a gold corpus. `ek` consumes [`ocracy`](https://github.com/thorwhalen/ocracy)’s normalized `OcrResult`, so it can benchmark any of its ~16 engines — or any `image -> OcrResult` callable of your own. ```python import ek.ocr gold = { "inv-1": { "image": "scan.png", "reference_text": "INVOICE 2024", "slice": "invoices", } } report = ek.ocr.evaluate_ocr( "ocrmac", gold, metric="cer", normalize=["lower", "collapse_whitespace"], persist=True, ) report.aggregate # corpus CER report.per_slice # CER per document slice report.detail["per_item"] # prediction, reference, score, confidence per document ``` Gold corpora, results, and runs persist to local `dol` stores under `~/.local/share/ek/`. ## Evaluate an AI agent — in cost per successful task The second instance. Agent evaluation is the *same* 2×2, but the evaluated object is an **episode** (tool calls + observations ending in a final state) and the unit that matters is **cost per successfully completed task**, not cost per token — because tokens spent on a failed episode are pure waste. ```python from ek.agents import TaskSpec, run_suite, per_million tasks = [ TaskSpec("t1", input="2+2", gold="4", slice="easy"), TaskSpec("t2", input="17*23", gold="391", slice="hard"), ] report = run_suite(my_agent, tasks, k=8, price=per_million(3.0, 15.0)) report.pass_hat_k # reliability: succeeds on ALL 8 trials (the production number) report.pass_at_k # capability: succeeds on ANY of 8 trials report.success_ci # a Wilson interval — a point estimate is not a result report.cost[ "cost_per_success" ] # Cost-of-Pass: dollars per *successful* task (inf if none) report.per_slice # ...cut by difficulty ``` Two numbers, not one: an agent that “usually works” is not shippable, and `pass^k` is what exposes it (a 90%-reliable agent is only ~43% reliable across 8 tries). Agent scores are *stochastic*, so the regression gate compares **intervals, not points**, and refuses to compare runs whose user-simulator or suite version changed: ```python from ek.agents import agent_regression_gate, save_agent_baseline save_agent_baseline(report, "v1") assert agent_regression_gate( new_report, "v1" ) # fails only on a *real* regression, not noise ``` Also included: BFCL-style tool-call correctness (cost-weighted by the tool grammar, so a wrong argument to a *destructive* tool costs more), an order-sensitive trajectory distance, and an LLM-as-judge signal — with `judge_validation()`, because an unvalidated judge is a liability, not a metric. **None of this needs an extra**: `ek.agents` is pure-python, and the bridges duck-type, so it scores an Inspect/DeepEval run without importing either. ## Install ```bash pip install ek # lean, permissive core (dol, config2py, jiwer, rapidfuzz) pip install "ek[ocr]" # + the ocracy OCR fleet (install engines via ocracy extras) pip install "ek[agents]" # + external agent harnesses to *run* (inspect-ai, deepeval, ragas) pip install "ek[all]" # + the permissive capability tiers (metrics, calibration, ...) ``` Heavier or copyleft/non-commercial libraries are never installed by default, and a CI license gate enforces it; see the extras in `pyproject.toml`. ## CLI ```bash ek cer "hello wrld" "hello world" # character error rate ek wer "hello wrld" "hello world" # word error rate ek pass-k 10 9 --k 8 # pass@k (capability) vs pass^k (reliability) ek cost-per-success 12.50 5 # dollars per successfully completed task ek where # the local data folder ek check tesseract # what an OCR engine needs to run ``` ## For contributors The architecture, conventions, and the research behind the design are documented for agents and humans in **[AGENTS.md]()**, the dev skills under `skills/`, and the research reports under `misc/docs/`.

This documentation as a single file: ek.md (Markdown, for agents).

# _autosummary/ek.agents.base.html.md # ek.agents.base Agent-evaluation data shapes: the episode (Layer B) and the task/tool grammar (Layer A). Evaluating an *agent* is the same two-layer problem `ek` already solves for information extraction – only the evaluated object changes. An extraction is a static payload; an **episode** is a *process*: a sequence of tool calls and observations ending in a final state. So: - **Layer A – the task/tool grammar.** [`ToolSpec`](_autosummary/ek.agents.base.html.md#ek.agents.base.ToolSpec) and [`TaskSpec`](_autosummary/ek.agents.base.html.md#ek.agents.base.TaskSpec) are *builders* over the existing [`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar): a tool becomes a [`NodeType`](_autosummary/ek.base.html.md#ek.base.NodeType) whose fields are its argument [`FieldSpec`](_autosummary/ek.base.html.md#ek.base.FieldSpec) s, > and `importance` carries the **cost of getting that argument wrong**. A wrong > argument to a *destructive* tool is not one unit of error. This reuses the Layer-A > cost-weight SSOT rather than inventing a parallel schema. - **Layer B – the episode.** [`Episode`](_autosummary/ek.agents.base.html.md#ek.agents.base.Episode) carries the [`Trajectory`](_autosummary/ek.agents.base.html.md#ek.agents.base.Trajectory), the [`Cost`](_autosummary/ek.agents.base.html.md#ek.agents.base.Cost), the outcome, and the [`RunProvenance`](_autosummary/ek.agents.base.html.md#ek.agents.base.RunProvenance) – riding *alongside* the > frozen grammar, never mutating it, exactly as `AnnotatedExtraction` does. These shapes live here, **not** in [`ek.base`](_autosummary/ek.base.html.md#module-ek.base): that module is the zero-dependency, IE-schema-only SSOT, and the OCR instance sets the precedent that an instance’s concrete shapes stay out of core (core duck-types them). Nothing here is imported by `ek` core. ### Example ```pycon >>> from ek.base import FieldSpec >>> book = ToolSpec("book_flight", params={"flight": FieldSpec("flight", "string"), ... "seat": FieldSpec("seat", "string")}) >>> refund = ToolSpec("refund", params={"amount": FieldSpec("amount", "number", ... importance=20.0)}, destructive=True) >>> g = tool_grammar(book, refund) >>> g.field_cost("refund", "amount") # a wrong refund amount is expensive 20.0 >>> g.field_cost("book_flight", "seat") # a wrong seat is not 1.0 ``` See `misc/docs/ek_07` (the map) and `misc/docs/ek_12` (the integration report). ### Module Attributes | [`DEFAULT_TASK_VALUE`](_autosummary/ek.agents.base.html.md#ek.agents.base.DEFAULT_TASK_VALUE) | Default value of a completed task (the task-level analog of `FieldSpec.importance`). | |-----------------------------------------------------------------------|----------------------------------------------------------------------------------------| | [`DESTRUCTIVE_WEIGHT`](_autosummary/ek.agents.base.html.md#ek.agents.base.DESTRUCTIVE_WEIGHT) | Multiplier applied to a destructive tool's importance when it is not set explicitly. | ### Functions | [`as_call`](_autosummary/ek.agents.base.html.md#ek.agents.base.as_call)(x) | Coerce a [`Step`](_autosummary/ek.agents.base.html.md#ek.agents.base.Step), `(tool, args)` tuple, or `{"tool":…, "args":…}` dict. | |------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`is_episode`](_autosummary/ek.agents.base.html.md#ek.agents.base.is_episode)(x) | Duck-typed episode check (core never imports this module -- it checks the *shape*). | | [`suite_grammar`](_autosummary/ek.agents.base.html.md#ek.agents.base.suite_grammar)(tasks) | The union grammar over a whole task suite (later tools win on name collision). | | [`tool_grammar`](_autosummary/ek.agents.base.html.md#ek.agents.base.tool_grammar)(\*tools) | Build the Layer-A [`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar) from [`ToolSpec`](_autosummary/ek.agents.base.html.md#ek.agents.base.ToolSpec) s. | ### Classes | [`Cost`](_autosummary/ek.agents.base.html.md#ek.agents.base.Cost)([input_tokens, output_tokens, ...]) | What one episode consumed: tokens (by kind), retries, and wall-clock. | |----------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------| | [`Episode`](_autosummary/ek.agents.base.html.md#ek.agents.base.Episode)([task_id, trajectory, output, ...]) | Layer B: one agent run over one task -- the object agent evaluation scores. | | [`RunProvenance`](_autosummary/ek.agents.base.html.md#ek.agents.base.RunProvenance)([seed, temperature, model, ...]) | The *hidden eval variables* that decide whether two runs are even comparable. | | [`Step`](_autosummary/ek.agents.base.html.md#ek.agents.base.Step)(tool[, args, observation, error]) | One step of a trajectory: a tool call and what came back. | | [`TaskSpec`](_autosummary/ek.agents.base.html.md#ek.agents.base.TaskSpec)(task_id[, input, gold, value, ...]) | One task in a suite: its input, its oracle, its value, and its allowed tools. | | [`ToolSpec`](_autosummary/ek.agents.base.html.md#ek.agents.base.ToolSpec)(name[, params, importance, destructive]) | One tool the agent may call: its argument schema and its error costs. | | [`Trajectory`](_autosummary/ek.agents.base.html.md#ek.agents.base.Trajectory)([steps]) | The ordered sequence of [`Step`](_autosummary/ek.agents.base.html.md#ek.agents.base.Step) s an agent took. | ### *class* ek.agents.base.Cost(input_tokens=0, output_tokens=0, cached_input_tokens=0, reasoning_tokens=0, retries=0, latency_s=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) What one episode consumed: tokens (by kind), retries, and wall-clock. Token *kinds* are tracked separately because they are **priced asymmetrically** – output typically costs ~5x input, cached input bills at a fraction, and reasoning tokens are billed. This is precisely why the objective is *dollars* per successful task and never a raw token count (`misc/docs/ek_11`). Pricing lives in [`ek.agents.cost`](_autosummary/ek.agents.cost.html.md#module-ek.agents.cost); this dataclass is the provider-neutral tally. ### Example ```pycon >>> a = Cost(input_tokens=100, output_tokens=50) >>> b = Cost(input_tokens=10, output_tokens=5, retries=1) >>> c = a + b >>> c.input_tokens, c.output_tokens, c.retries (110, 55, 1) ``` #### *property* total_tokens *: [int](https://docs.python.org/3/builtins/functions.html#int)* All billed tokens (a rough size proxy – *not* a cost; see `dollars()`). ### ek.agents.base.DEFAULT_TASK_VALUE *= 1.0* Default value of a completed task (the task-level analog of `FieldSpec.importance`). ### ek.agents.base.DESTRUCTIVE_WEIGHT *= 10.0* Multiplier applied to a destructive tool’s importance when it is not set explicitly. Calling a destructive tool wrongly is categorically worse than calling a read-only one; a keyword-tunable default, never a magic number buried in a metric. ### *class* ek.agents.base.Episode(task_id='', trajectory=, output=None, final_state=None, cost=None, success=None, run=None, provenance=None, meta=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Layer B: one agent run over one task – the object agent evaluation scores. The direct analog of `AnnotatedExtraction`: it rides alongside the frozen Layer-A grammar and carries the run’s metadata. It is scored *offline* against gold ([`ek.score()`](_autosummary/ek.html.md#ek.score) / [`ek.evaluate()`](_autosummary/ek.html.md#ek.evaluate)) and estimated *online* against a judge or a consensus ([`ek.estimate_quality()`](_autosummary/ek.html.md#ek.estimate_quality)) – the same object, both halves. * **Parameters:** * **task_id** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Which task this episode answers (the k-trial grouping key). * **trajectory** ([`Trajectory`](_autosummary/ek.agents.base.html.md#ek.agents.base.Trajectory)) – The tool calls taken. * **output** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The agent’s final user-facing answer. * **final_state** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The end state of the world (the DB, the filesystem) – what a state-based oracle actually grades. Success is a *final-state* check, not a surface-text check. * **cost** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Cost`](_autosummary/ek.agents.base.html.md#ek.agents.base.Cost)]) – The [`Cost`](_autosummary/ek.agents.base.html.md#ek.agents.base.Cost) consumed. * **success** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`bool`](https://docs.python.org/3/builtins/functions.html#bool)]) – Filled in by a checker; `None` until graded. * **run** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`RunProvenance`](_autosummary/ek.agents.base.html.md#ek.agents.base.RunProvenance)]) – The [`RunProvenance`](_autosummary/ek.agents.base.html.md#ek.agents.base.RunProvenance) (seed, model, simulator, suite version). * **provenance** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Provenance`](_autosummary/ek.base.html.md#ek.base.Provenance)]) – Optional core [`Provenance`](_autosummary/ek.base.html.md#ek.base.Provenance) for click-to-source audit. * **meta** ([`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)) – Anything else the bridge captured. #### *property* calls *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)* The `(tool, args)` pairs taken, in order. #### graded(success) A copy with `success` set (never mutates – grading stays idempotent). `meta` is copied too: `dataclasses.replace` is a *shallow* copy, so without this the “copy” would share the original’s mutable `meta` dict. * **Return type:** [`Episode`](_autosummary/ek.agents.base.html.md#ek.agents.base.Episode) ### *class* ek.agents.base.RunProvenance(seed=None, temperature=None, model='', simulator='', suite_version='', scaffold='') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The *hidden eval variables* that decide whether two runs are even comparable. Agent evaluation is stochastic, and several of its inputs are themselves models. In a tau-bench-style suite the **user simulator is an LLM agent**, so its model/prompt/version silently changes results run to run. Recording these is what lets the harness *refuse* to compare a baseline against a run that changed the simulator or the suite – the same discipline `ek`’s offline harness already applies when it refuses to compare across two different metrics (`misc/docs/ek_08`, `misc/docs/ek_11`). * **Parameters:** * **seed** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`int`](https://docs.python.org/3/builtins/functions.html#int)]) – RNG seed for the trial (`None` = unseeded/nondeterministic). * **temperature** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`float`](https://docs.python.org/3/builtins/functions.html#float)]) – Sampling temperature of the agent under test. * **model** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The agent’s model id. * **simulator** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The **user-simulator** id/version – an eval variable, not a constant. * **suite_version** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Version of the task suite (contamination and task fixes move scores). * **scaffold** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The agent scaffold/harness id (the same model moves points on scaffold alone). #### comparability_key() The fields that must match for two runs to be comparable (seed excluded). Seed and temperature are *expected* to vary across trials; the simulator, the suite version and the scaffold are not – changing them invalidates a baseline. * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple) ### *class* ek.agents.base.Step(tool, args=, observation=None, error=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One step of a trajectory: a tool call and what came back. `error` is set when the call itself failed (a bad argument, a tool exception) – error *recovery* across subsequent steps is itself an evaluable signal. #### *property* call *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)* The `(tool, args)` pair – what tool-call correctness actually compares. ### *class* ek.agents.base.TaskSpec(task_id, input=None, gold=None, value=1.0, tools=(), slice=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One task in a suite: its input, its oracle, its value, and its allowed tools. * **Parameters:** * **task_id** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Stable identifier (the grouping key for k-trial reliability). * **input** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Whatever the agent under test is called with. * **gold** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The reference the checker grades against (a goal state, an expected answer, a gold trajectory – the checker decides how to read it). * **value** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Task-level cost weight: how much a completed task is worth. The task-level analog of `FieldSpec.importance`; feeds value-weighted cost accounting. * **tools** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`ToolSpec`](_autosummary/ek.agents.base.html.md#ek.agents.base.ToolSpec)]) – The tools this task permits (the Layer-A grammar for the task). * **slice** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – Optional slice label (domain, difficulty, language) – per-slice cuts are mandatory, not optional (a low pass^k concentrated in one hard slice is a different problem from uniform flakiness). #### grammar() The Layer-A grammar for this task’s tool set. * **Return type:** [`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar) ### *class* ek.agents.base.ToolSpec(name, params=, importance=None, destructive=False) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One tool the agent may call: its argument schema and its error costs. * **Parameters:** * **name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The tool/function name (the AST-match ground truth). * **params** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`FieldSpec`](_autosummary/ek.base.html.md#ek.base.FieldSpec)]) – Argument name -> [`FieldSpec`](_autosummary/ek.base.html.md#ek.base.FieldSpec). Each spec’s `importance` is the cost of getting *that argument* wrong, and its `domain` feeds validators – the same contract as an extracted field. * **importance** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`float`](https://docs.python.org/3/builtins/functions.html#float)]) – Cost weight of the *call itself* (a missing or spurious call). Defaults to [`DESTRUCTIVE_WEIGHT`](_autosummary/ek.agents.base.html.md#ek.agents.base.DESTRUCTIVE_WEIGHT) when `destructive` is set, else 1.0. * **destructive** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether calling this tool mutates the world irreversibly (a refund, a delete, a send). Drives the default weight and the safety validators. #### to_node_type() Render this tool as a Layer-A [`NodeType`](_autosummary/ek.base.html.md#ek.base.NodeType) (args as fields). * **Return type:** [`NodeType`](_autosummary/ek.base.html.md#ek.base.NodeType) #### *property* weight *: [float](https://docs.python.org/3/builtins/functions.html#float)* The call-level cost weight (explicit, else destructive-aware default). ### *class* ek.agents.base.Trajectory(steps=()) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The ordered sequence of [`Step`](_autosummary/ek.agents.base.html.md#ek.agents.base.Step) s an agent took. A trajectory is *linear* – which is why it is scored with a sequence edit distance, not a graph edit distance (see [`ek.agents.trajectory`](_autosummary/ek.agents.trajectory.html.md#module-ek.agents.trajectory)). ### Example ```pycon >>> t = Trajectory([Step("search", {"q": "cat"}), Step("answer", {"a": "meow"})]) >>> len(t), t.tools (2, ('search', 'answer')) ``` #### *property* calls *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)* The `(tool, args)` pairs in call order. #### *property* tools *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)* The tool names in call order. ### ek.agents.base.as_call(x) Coerce a [`Step`](_autosummary/ek.agents.base.html.md#ek.agents.base.Step), `(tool, args)` tuple, or `{"tool":…, "args":…}` dict. Gold tool calls are usually written as bare pairs or dicts (no observation), while a prediction arrives as a [`Step`](_autosummary/ek.agents.base.html.md#ek.agents.base.Step); both must compare. Progressive disclosure: the simple literal works, the rich object works. * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple) ### Example ```pycon >>> as_call(Step("search", {"q": "cat"})) ('search', {'q': 'cat'}) >>> as_call(("search", {"q": "cat"})) ('search', {'q': 'cat'}) >>> as_call({"tool": "search", "args": {"q": "cat"}}) ('search', {'q': 'cat'}) >>> as_call("search") ('search', {}) ``` ### ek.agents.base.is_episode(x) Duck-typed episode check (core never imports this module – it checks the *shape*). * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.agents.base.suite_grammar(tasks) The union grammar over a whole task suite (later tools win on name collision). * **Return type:** [`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar) ### ek.agents.base.tool_grammar(\*tools) Build the Layer-A [`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar) from [`ToolSpec`](_autosummary/ek.agents.base.html.md#ek.agents.base.ToolSpec) s. The resulting grammar is the frozen cost SSOT every agent metric reads: node cost = the tool’s call weight, field cost = the argument’s importance. * **Return type:** [`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar) ### Example ```pycon >>> from ek.base import FieldSpec >>> g = tool_grammar(ToolSpec("send", params={"to": FieldSpec("to")}, ... destructive=True)) >>> g.node_cost("send") 10.0 ``` # _autosummary/ek.agents.bridge.html.md # ek.agents.bridge One-way adapters: turn whatever your agent already emits into an `ek` `Episode`. The dependency direction is the `ek -> ocracy` rule restated, and it is a **hard rule**: `ek -> inspect_ai / deepeval / ragas` (via the `ek[agents]` extra), **never the reverse**. Note what that buys: `ek` core depends only on the *shape* of what those tools emit, so the adapters here **duck-type** and import nothing. You need the extra to *run* Inspect or DeepEval; you do not need it to *score* what they produced – exactly as `ek` evaluates any `OcrResult`-shaped object without importing an OCR engine. The workhorse is [`trajectory_from_messages()`](_autosummary/ek.agents.bridge.html.md#ek.agents.bridge.trajectory_from_messages): provider-shaped chat transcripts (OpenAI- and Anthropic-style `tool_calls` / `tool_use` blocks) are the universal wire format for a tool-using agent, and parsing them needs no SDK at all. ### Example ```pycon >>> messages = [ ... {"role": "user", "content": "weather in Paris?"}, ... {"role": "assistant", "tool_calls": [ ... {"function": {"name": "get_weather", "arguments": '{"city": "Paris"}'}}]}, ... {"role": "tool", "content": "18C"}, ... {"role": "assistant", "content": "It is 18C in Paris."}, ... ] >>> traj = trajectory_from_messages(messages) >>> traj.tools ('get_weather',) >>> traj.steps[0].args, traj.steps[0].observation ({'city': 'Paris'}, '18C') ``` ### Functions | [`as_agent`](_autosummary/ek.agents.bridge.html.md#ek.agents.bridge.as_agent)(fn) | Wrap a plain `input -> answer` function into the harness's `TaskSpec -> Episode` shape. | |-----------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------| | [`cost_from_usage`](_autosummary/ek.agents.bridge.html.md#ek.agents.bridge.cost_from_usage)(usage, \*[, latency_s]) | Build a [`Cost`](_autosummary/ek.agents.base.html.md#ek.agents.base.Cost) from a provider `usage` object or dict. | | [`episode_from_messages`](_autosummary/ek.agents.bridge.html.md#ek.agents.bridge.episode_from_messages)(messages, \*[, ...]) | Build a full [`Episode`](_autosummary/ek.agents.base.html.md#ek.agents.base.Episode) from a transcript (+ optional usage/state). | | [`from_deepeval_test_case`](_autosummary/ek.agents.bridge.html.md#ek.agents.bridge.from_deepeval_test_case)(case, \*[, task_id]) | Adapt a DeepEval `LLMTestCase`-shaped object into an `Episode` (duck-typed). | | [`from_inspect_sample`](_autosummary/ek.agents.bridge.html.md#ek.agents.bridge.from_inspect_sample)(sample, \*[, task_id]) | Adapt an Inspect AI `EvalSample`-shaped object into an `Episode` (duck-typed). | | [`trajectory_from_messages`](_autosummary/ek.agents.bridge.html.md#ek.agents.bridge.trajectory_from_messages)(messages) | Parse a chat transcript into a [`Trajectory`](_autosummary/ek.agents.base.html.md#ek.agents.base.Trajectory). | ### ek.agents.bridge.as_agent(fn) Wrap a plain `input -> answer` function into the harness’s `TaskSpec -> Episode` shape. Progressive disclosure: the trivial agent should not have to learn the Episode type. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable) ### Example ```pycon >>> from ek.agents.base import TaskSpec >>> agent = as_agent(str.upper) >>> agent(TaskSpec("t1", input="hi")).output 'HI' ``` ### ek.agents.bridge.cost_from_usage(usage, , latency_s=None) Build a [`Cost`](_autosummary/ek.agents.base.html.md#ek.agents.base.Cost) from a provider `usage` object or dict. Understands the OpenAI (`prompt_tokens` / `completion_tokens`) and Anthropic (`input_tokens` / `output_tokens` / `cache_read_input_tokens`) spellings. * **Return type:** [`Cost`](_autosummary/ek.agents.base.html.md#ek.agents.base.Cost) ### Example ```pycon >>> c = cost_from_usage({"prompt_tokens": 100, "completion_tokens": 20}) >>> c.input_tokens, c.output_tokens (100, 20) ``` ### ek.agents.bridge.episode_from_messages(messages, , task_id='', usage=None, output=None, final_state=None, latency_s=None) Build a full [`Episode`](_autosummary/ek.agents.base.html.md#ek.agents.base.Episode) from a transcript (+ optional usage/state). `output` defaults to the last assistant text – the agent’s final answer. * **Return type:** [`Episode`](_autosummary/ek.agents.base.html.md#ek.agents.base.Episode) ### ek.agents.bridge.from_deepeval_test_case(case, , task_id='') Adapt a DeepEval `LLMTestCase`-shaped object into an `Episode` (duck-typed). Reads `.actual_output` and `.tools_called`. Note DeepEval phones home by default (Confident-AI telemetry) – disable it before use if that matters to you; `ek` never enables it. * **Return type:** [`Episode`](_autosummary/ek.agents.base.html.md#ek.agents.base.Episode) ### ek.agents.bridge.from_inspect_sample(sample, , task_id='') Adapt an Inspect AI `EvalSample`-shaped object into an `Episode` (duck-typed). Reads `.messages`, `.output` and `.id` – no `inspect_ai` import required, so scoring an Inspect log never drags the harness into `ek`’s dependency closure. * **Return type:** [`Episode`](_autosummary/ek.agents.base.html.md#ek.agents.base.Episode) ### ek.agents.bridge.trajectory_from_messages(messages) Parse a chat transcript into a [`Trajectory`](_autosummary/ek.agents.base.html.md#ek.agents.base.Trajectory). Each assistant tool call becomes a [`Step`](_autosummary/ek.agents.base.html.md#ek.agents.base.Step), and the *next* tool/result message becomes that step’s `observation`. A tool message carrying an error marks the step as errored – error *recovery* across later steps is itself an evaluable signal. * **Return type:** [`Trajectory`](_autosummary/ek.agents.base.html.md#ek.agents.base.Trajectory) # _autosummary/ek.agents.cost.html.md # ek.agents.cost The cost model and the price SSOT: dollars per successfully completed task. The unit that matters for an agent is **cost per successfully completed task**, not cost per token. Two facts force it (`misc/docs/ek_11`): 1. **Tokens are not dollars.** Input, output, cached-input and reasoning tokens are priced *asymmetrically* (output typically ~5x input; cached input bills at a fraction). A “tokens per success” figure is at best a rough proxy. 2. **Failure is not free.** Tokens spent on an episode that failed are pure waste, so the denominator must be *successes*, not attempts. This is **Cost-of-Pass**: the expected monetary cost of obtaining a *correct* solution, `E[cost] / P(success)` – which diverges to infinity when the agent never succeeds (a model that cannot solve a task at any price is not cheap, it is unusable; a per-token metric that reports it as “$0.003/call” is actively misleading). **No hardcoded prices.** Rates go stale monthly, so this module ships *no* built-in price table: a [`ModelPrice`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.ModelPrice) is either passed directly or resolved from an injected catalog ([`load_prices()`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.load_prices) reads LiteLLM’s MIT-licensed `model_prices_and_context_window.json` – **the data file, never the SDK**, whose `enterprise/` subtree is a proprietary carve-out). Missing rates raise an *actionable* error rather than silently guessing. ### Example ```pycon >>> from ek.agents.base import Cost >>> price = ModelPrice(input=1e-6, output=5e-6) # $1 / $5 per 1M tokens >>> c = Cost(input_tokens=1_000_000, output_tokens=200_000) >>> round(dollars(c, price), 4) 2.0 ``` ### Module Attributes | [`DEFAULT_CACHE_DISCOUNT`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.DEFAULT_CACHE_DISCOUNT) | Cached input is billed at a fraction of the normal input rate (~10% across the major providers). | |---------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------| | [`DEFAULT_BATCH_DISCOUNT`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.DEFAULT_BATCH_DISCOUNT) | Batch/deferred APIs are ~50% off. | ### Functions | [`cost_of_pass`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.cost_of_pass)(total_cost, n_success) | Expected monetary cost of one *successful* task: `total_cost / n_success`. | |-----------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------| | [`cost_report`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.cost_report)(episodes, \*[, prices, price]) | The quality x cost x latency triple, reported **together**. | | [`dollars`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.dollars)(cost, price, \*[, prices, batch]) | Monetary cost of one [`Cost`](_autosummary/ek.agents.base.html.md#ek.agents.base.Cost) tally, in USD. | | [`episode_dollars`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.episode_dollars)(episode, \*[, prices, price, ...]) | Monetary cost of one episode (0.0 if it carries no [`Cost`](_autosummary/ek.agents.base.html.md#ek.agents.base.Cost)). | | [`load_prices`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.load_prices)(source) | Parse a LiteLLM-format price catalog into a `{model: ModelPrice}` table. | | [`per_million`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.per_million)(input, output, \*[, ...]) | Build a [`ModelPrice`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.ModelPrice) from the human-facing **USD per 1M tokens** figures. | | [`price_of`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.price_of)(model, \*[, prices]) | Resolve a model's rates from an injected catalog, or fail *actionably*. | ### Classes | [`ModelPrice`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.ModelPrice)([input, output, cached_input, ...]) | Per-token rates for one model, in **USD per token**. | |---------------------------------------------------------------------------------------------------|--------------------------------------------------------| ### Exceptions | [`UnknownModelPrice`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.UnknownModelPrice) | Raised when a model's rates are not in the injected price catalog. | |----------------------------------------------------------------------|----------------------------------------------------------------------| ### ek.agents.cost.DEFAULT_BATCH_DISCOUNT *= 0.5* Batch/deferred APIs are ~50% off. A keyword multiplier, never assumed. ### ek.agents.cost.DEFAULT_CACHE_DISCOUNT *= 0.1* Cached input is billed at a fraction of the normal input rate (~10% across the major providers). Used only when a model’s price entry declares no explicit cached rate. ### *class* ek.agents.cost.ModelPrice(input=0.0, output=0.0, cached_input=None, reasoning=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Per-token rates for one model, in **USD per token**. * **Parameters:** * **input** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – USD per input (prompt) token. * **output** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – USD per output (completion) token. * **cached_input** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`float`](https://docs.python.org/3/builtins/functions.html#float)]) – USD per cached-input token; defaults to `DEFAULT_CACHE_DISCOUNT * input` when not declared. * **reasoning** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`float`](https://docs.python.org/3/builtins/functions.html#float)]) – USD per reasoning token; defaults to the `output` rate (reasoning tokens are billed as output by the major providers). #### *property* cached_rate *: [float](https://docs.python.org/3/builtins/functions.html#float)* The cached-input rate (explicit, else the discounted input rate). #### *property* reasoning_rate *: [float](https://docs.python.org/3/builtins/functions.html#float)* The reasoning-token rate (explicit, else the output rate). ### *exception* ek.agents.cost.UnknownModelPrice Bases: [`KeyError`](https://docs.python.org/3/builtins/exceptions.html#KeyError) Raised when a model’s rates are not in the injected price catalog. ### ek.agents.cost.cost_of_pass(total_cost, n_success) Expected monetary cost of one *successful* task: `total_cost / n_success`. Returns `inf` when nothing succeeded – infeasibility is the honest answer, not a cheap-looking zero (this divergence is the point of the metric). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### Example ```pycon >>> cost_of_pass(10.0, 4) 2.5 >>> cost_of_pass(10.0, 0) inf ``` ### ek.agents.cost.cost_report(episodes, , prices=None, price=None) The quality x cost x latency triple, reported **together**. Scoring accuracy *without* cost lets an agent chase tiny gains with unbounded API calls, so a report that omits cost is not a report. Returns `n`, `n_success`, `success_rate`, `total_dollars`, `cost_per_success` (Cost-of-Pass), `mean_latency_s` and `total_tokens`. With **no** `price`/`prices`, dollar figures are `None` rather than a fabricated zero: `ek` will not invent rates, and a silent 0.0 would make a costly agent look free. Tokens and latency are still reported. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### Example ```pycon >>> from ek.agents.base import Cost, Episode >>> p = per_million(1.0, 1.0) >>> eps = [Episode(task_id="a", cost=Cost(input_tokens=1_000_000), success=True), ... Episode(task_id="b", cost=Cost(input_tokens=1_000_000), success=False)] >>> r = cost_report(eps, price=p) >>> r["success_rate"], round(r["cost_per_success"], 2) (0.5, 2.0) >>> cost_report(eps)["cost_per_success"] is None # no rates -> no dollars invented True ``` ### ek.agents.cost.dollars(cost, price, , prices=None, batch=False) Monetary cost of one [`Cost`](_autosummary/ek.agents.base.html.md#ek.agents.base.Cost) tally, in USD. * **Parameters:** * **cost** ([`Cost`](_autosummary/ek.agents.base.html.md#ek.agents.base.Cost)) – The token/retry tally. * **price** (`Union`[[`ModelPrice`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.ModelPrice), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – A [`ModelPrice`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.ModelPrice), or a model name to resolve against `prices`. * **prices** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`ModelPrice`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.ModelPrice)]]) – The injected price catalog (required when `price` is a name). * **batch** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Apply the batch-API discount ([`DEFAULT_BATCH_DISCOUNT`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.DEFAULT_BATCH_DISCOUNT)). * **Raises:** [**UnknownModelPrice**](_autosummary/ek.agents.cost.html.md#ek.agents.cost.UnknownModelPrice) – if a model name is given with no rates for it. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.agents.cost.episode_dollars(episode, , prices=None, price=None, batch=False) Monetary cost of one episode (0.0 if it carries no [`Cost`](_autosummary/ek.agents.base.html.md#ek.agents.base.Cost)). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.agents.cost.load_prices(source) Parse a LiteLLM-format price catalog into a `{model: ModelPrice}` table. `source` is a path to (or an already-parsed mapping of) LiteLLM’s MIT-licensed `model_prices_and_context_window.json`. We read the **data file only** and never import the `litellm` SDK – its `enterprise/` subtree is a proprietary carve-out, and the SDK drags a large transitive tree in just to look up a rate. Entries lacking token rates (embeddings, unpriced models) are skipped. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### Example ```pycon >>> table = load_prices({"m": {"input_cost_per_token": 1e-6, ... "output_cost_per_token": 2e-6}}) >>> table["m"].output 2e-06 ``` ### ek.agents.cost.per_million(input, output, , cached_input=None, reasoning=None) Build a [`ModelPrice`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.ModelPrice) from the human-facing **USD per 1M tokens** figures. * **Return type:** [`ModelPrice`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.ModelPrice) ### Example ```pycon >>> p = per_million(3.0, 15.0) # $3 in / $15 out per 1M tokens >>> round(p.input, 9), round(p.output, 9) (3e-06, 1.5e-05) ``` ### ek.agents.cost.price_of(model, , prices=None) Resolve a model’s rates from an injected catalog, or fail *actionably*. `ek` deliberately ships no built-in price table (rates go stale monthly and would be magic numbers). Supply one with `prices=`, or load the LiteLLM catalog with [`load_prices()`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.load_prices). * **Return type:** [`ModelPrice`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.ModelPrice) # _autosummary/ek.agents.harness.html.md # ek.agents.harness The agent task-suite harness: k trials, per-slice cuts, and a gate that survives noise. [`run_suite()`](_autosummary/ek.agents.harness.html.md#ek.agents.harness.run_suite) is [`ek.evaluate_store()`](_autosummary/ek.html.md#ek.evaluate_store) specialized for agents: it runs the system under test over a suite **k times per task** (reliability needs repeats), grades each episode with an *injected* checker, and reports `pass^k` + cost + latency per slice. **Why a separate gate.** `ek`’s offline [`ek.regression_gate()`](_autosummary/ek.html.md#ek.regression_gate) is a scalar point comparison with `tolerance=0.0` – correct for a deterministic OCR benchmark, **unsound** for an agent. Agent metrics are random variables: re-running the same agent moves the number. A point gate therefore (a) flags sampling noise as a regression and (b) misses a real regression that hides inside the tolerance band. [`agent_regression_gate()`](_autosummary/ek.agents.harness.html.md#ek.agents.harness.agent_regression_gate) instead tests the **difference** `current - baseline` with both runs’ uncertainty folded in (a Newcombe score interval on the success rate; a combined-SE bound on `pass^k`), failing only when the upper bound of that change is confidently negative. Two tempting shortcuts are **both wrong**, and this module has shipped and fixed each of them: comparing our interval to the baseline’s *point* (the baseline is itself noisy, so one flake against a lucky baseline reads as a regression), and asking whether two 95% intervals *overlap* (that is a ~0.5% test, not a 5% one – it goes blind to a 15-point drop on a 60-task suite). An empty run, and an empty **baseline**, are both refused: with no trials every bound is the maximally-uncertain default, so nothing could ever be shown to fall below it. **Hidden eval variables.** In a tau-bench-style suite the *user simulator is itself an LLM* – change its model or prompt and the scores move, silently. So a baseline records its [`RunProvenance`](_autosummary/ek.agents.base.html.md#ek.agents.base.RunProvenance) and the gate **refuses** to compare across a changed simulator, suite version, or scaffold, exactly as the offline gate already refuses to compare two different metrics. Comparing across them is not a regression check, it is a category error. **Verifier isolation is a security property.** The checker must run where the agent’s trajectory cannot reach it (a red-team produced a ~10-line `conftest.py` that rewrote every test outcome to “passed”). `ek` cannot enforce your sandbox – it only refuses to pretend the problem does not exist. ### Example ```pycon >>> from ek.agents.base import Cost, Episode, TaskSpec >>> tasks = [TaskSpec("t1", input="2+2", gold="4"), TaskSpec("t2", input="3+3", gold="6")] >>> def agent(task): ... answers = {"2+2": "4", "3+3": "7"} # gets t2 wrong, every time ... return Episode(output=answers[task.input], cost=Cost(input_tokens=10)) >>> report = run_suite(agent, tasks, k=1) >>> report.success_rate, report.pass_hat_k (0.5, 0.5) ``` ### Functions | [`agent_regression_gate`](_autosummary/ek.agents.harness.html.md#ek.agents.harness.agent_regression_gate)(report, baseline, \*[, ...]) | Fail only when the evidence says the agent **really** got worse -- not on noise. | |-------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------| | [`load_agent_baseline`](_autosummary/ek.agents.harness.html.md#ek.agents.harness.load_agent_baseline)(name, \*[, rootdir]) | Load a named agent baseline (or `None`). | | [`run_suite`](_autosummary/ek.agents.harness.html.md#ek.agents.harness.run_suite)(agent, tasks, \*[, k, check, ...]) | Run `agent` over a task suite, `k` trials per task, and report reliability + cost. | | [`save_agent_baseline`](_autosummary/ek.agents.harness.html.md#ek.agents.harness.save_agent_baseline)(report, name, \*[, rootdir]) | Freeze a run as a named baseline (including its provenance, so the gate can guard it). | ### Classes | [`AgentGateResult`](_autosummary/ek.agents.harness.html.md#ek.agents.harness.AgentGateResult)([passed, reasons, ...]) | Outcome of a variance-aware agent regression check. | |--------------------------------------------------------------------------------------------|-------------------------------------------------------| ### *class* ek.agents.harness.AgentGateResult(passed=True, reasons=, pass_hat_k=None, pass_hat_k_baseline=None, success_rate=None, success_rate_baseline=None, cost_per_success=None, cost_per_success_baseline=None, underpowered=False) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Outcome of a variance-aware agent regression check. ### ek.agents.harness.agent_regression_gate(report, baseline, , tolerance=0.0, cost_tolerance=0.0, rootdir=None) Fail only when the evidence says the agent **really** got worse – not on noise. Tests the **difference** `current - baseline`, with *both* runs’ uncertainty folded in: a Newcombe score interval on the success rate, and (for `k > 1`) a combined-SE bound on `pass^k`. A regression is declared only when the upper bound of that change is below `-tolerance` – i.e. we are confident the true change is negative. Two tempting shortcuts are **both wrong**, and this module has shipped and fixed each: - Comparing our interval to the baseline’s **point**. The baseline is *itself* a noisy estimate; a Wilson upper bound only reaches 1.0 when every trial passed, so a single flake against a lucky baseline reads as a “confident regression”. - Asking whether the two 95% intervals **overlap**. That is not a 5% test but roughly a 0.5% one: it buys a calm gate by going blind (a 15-point drop on a 60-task suite becomes invisible). A gate that cannot see a real regression is as broken as one that cries wolf. A result that is merely *worse but not significantly so* is not a clean pass either – it is an underpowered experiment, and the result says so (`underpowered`, plus a warning). * **Raises:** [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if the baseline was produced with a different **user simulator**, suite version, or scaffold – those runs are not comparable, and silently comparing them would be a category error (the same reason the offline gate refuses to compare two different metrics); if the baseline recorded zero trials (a permanent free pass); or if it is malformed. * **Return type:** [`AgentGateResult`](_autosummary/ek.agents.harness.html.md#ek.agents.harness.AgentGateResult) ### ek.agents.harness.load_agent_baseline(name, , rootdir=None) Load a named agent baseline (or `None`). * **Return type:** [`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)] ### ek.agents.harness.run_suite(agent, tasks, , k=1, check=None, metrics=None, price=None, prices=None, run=None, seed=None, persist=False, run_id=None, rootdir=None) Run `agent` over a task suite, `k` trials per task, and report reliability + cost. * **Parameters:** * **agent** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`TaskSpec`](_autosummary/ek.agents.base.html.md#ek.agents.base.TaskSpec)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – The system under test: `TaskSpec -> Episode` (a bare answer is wrapped). It receives the whole spec – `.input`, `.tools`, `.gold` – not just the input. If it accepts a keyword `seed`, the per-trial seed is passed to it. * **tasks** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – [`TaskSpec`](_autosummary/ek.agents.base.html.md#ek.agents.base.TaskSpec) s (or a `{task_id: spec}` mapping). * **k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Trials per task. `k > 1` is what makes `pass^k` meaningful – and it only means anything if the agent is genuinely stochastic (the report warns if it is not). * **check** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The success oracle – a registered checker name or an `(episode, gold) -> bool` callable. Defaults to `"output"`; **inject a state-based/executable oracle** for a real suite, and run it isolated from the agent. * **metrics** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)]) – Optional `{name: Metric}` scored on every episode against `task.gold`, with the **suite’s Layer-A grammar injected** – this is how the tool/argument cost weights reach a `tool_call`/`trajectory` metric. Results land in `report.detail["metrics"]`. * **price/prices** – Rates for costing the episodes (see [`ek.agents.cost`](_autosummary/ek.agents.cost.html.md#module-ek.agents.cost)). * **run** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`RunProvenance`](_autosummary/ek.agents.base.html.md#ek.agents.base.RunProvenance)]) – [`RunProvenance`](_autosummary/ek.agents.base.html.md#ek.agents.base.RunProvenance) for this run (model, simulator, suite version, scaffold). Recorded on every episode and on any baseline saved from it. * **seed** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`int`](https://docs.python.org/3/builtins/functions.html#int)]) – Base RNG seed. Trial *i* of each task uses `seed + i`: it is recorded on the episode’s provenance **and passed to the agent** when the agent accepts a `seed` keyword. * **persist/run_id/rootdir** – Persist the run to the `runs`/`results` stores. * **Return type:** [`ReliabilityReport`](_autosummary/ek.agents.html.md#ek.agents.ReliabilityReport) * **Returns:** A [`ReliabilityReport`](_autosummary/ek.agents.html.md#ek.agents.ReliabilityReport) carrying `pass@k`, `pass^k` (with a bootstrap CI), the success rate (with a Wilson CI), the per-slice cuts, and the cost report – because reliability without cost is only half the answer. ### ek.agents.harness.save_agent_baseline(report, name, , rootdir=None) Freeze a run as a named baseline (including its provenance, so the gate can guard it). * **Raises:** [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if the run evaluated nothing. An empty baseline is worse than no baseline – it is a *permanent free pass*, because every bound in it is the maximally-uncertain default and no later run can be shown to fall below it. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) # _autosummary/ek.agents.html.md # ek.agents `ek.agents` – evaluating AI agents and assistants, in cost per successful task. The agent instance of `ek`, mirroring [`ek.ocr`](_autosummary/ek.ocr.html.md#module-ek.ocr). Agent evaluation is the **same 2x2** the framework already implements – (reference availability) x (granularity) – but the evaluated object is an **episode** (tool calls + observations ending in a final state) and the objective is **cost per successfully completed task**, not cost per token. corner | agent meaning | facade |
``` |---| ``` — ``` |---| ``` | reference-based, one item | grade an episode against a gold outcome/trajectory | [`ek.score()`](_autosummary/ek.html.md#ek.score) | | reference-based, corpus | task-suite benchmark: `pass^k` + cost, per slice | [`run_suite()`](_autosummary/ek.agents.html.md#ek.agents.run_suite) | | reference-free, one item | judge / verifier / the cascade’s confidence gate | [`ek.estimate_quality()`](_autosummary/ek.html.md#ek.estimate_quality) | What it reuses (rather than rebuilds): the Layer-A `GraphGrammar` becomes the \*\*task/tool grammar\*\* (`FieldSpec.importance` = the cost of a wrong argument, so a wrong argument to a *destructive* tool is not one unit of error); `FieldMetric`’s TP/FP/FN accounting powers tool-call scoring; the ROVER agreement seam is self-consistency; the IAA helpers validate a judge; and the whole `signal -> calibrate -> validate -> decide` cascade is the confidence gate. Quickstart: ```pycon >>> from ek.agents import TaskSpec, run_suite, as_agent >>> tasks = [TaskSpec("t1", input="hi", gold="HI"), TaskSpec("t2", input="yo", gold="YO")] >>> report = run_suite(as_agent(str.upper), tasks) >>> report.success_rate 1.0 ``` Research: `misc/docs/ek_07` (map) - `ek_08` (task success) - `ek_09` (judge/QE) - `ek_10` (trajectory) - `ek_11` (cost economics) - `ek_12` (libraries + integration). ### Functions | [`tool_grammar`](_autosummary/ek.agents.html.md#ek.agents.tool_grammar)(\*tools) | Build the Layer-A [`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar) from [`ToolSpec`](_autosummary/ek.agents.html.md#ek.agents.ToolSpec) s. | |-----------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`suite_grammar`](_autosummary/ek.agents.html.md#ek.agents.suite_grammar)(tasks) | The union grammar over a whole task suite (later tools win on name collision). | | [`as_call`](_autosummary/ek.agents.html.md#ek.agents.as_call)(x) | Coerce a [`Step`](_autosummary/ek.agents.html.md#ek.agents.Step), `(tool, args)` tuple, or `{"tool":…, "args":…}` dict. | | [`is_episode`](_autosummary/ek.agents.html.md#ek.agents.is_episode)(x) | Duck-typed episode check (core never imports this module -- it checks the *shape*). | | [`match_calls`](_autosummary/ek.agents.html.md#ek.agents.match_calls)(pred, gold, \*[, canon]) | Assign predicted calls to gold calls -- the layer `FieldMetric` does not provide. | | [`output_match`](_autosummary/ek.agents.html.md#ek.agents.output_match)(episode, gold) | Canonicalized exact match on the final answer (the GAIA-style oracle). | | [`final_state_match`](_autosummary/ek.agents.html.md#ek.agents.final_state_match)(episode, gold) | Compare the episode's **final state** with the goal state (the tau-bench oracle). | | [`recorded_success`](_autosummary/ek.agents.html.md#ek.agents.recorded_success)(episode, gold) | Trust a `success` already recorded on the episode (e.g. by an external harness). | | [`pass_at_k`](_autosummary/ek.agents.html.md#ek.agents.pass_at_k)(\*, n, c, k) | Unbiased `pass@k`: probability that **at least one** of k samples succeeds. | | [`pass_hat_k`](_autosummary/ek.agents.html.md#ek.agents.pass_hat_k)(\*, n, c, k) | Unbiased `pass^k`: probability that **all k** independent trials succeed. | | [`reliability`](_autosummary/ek.agents.html.md#ek.agents.reliability)(source, \*[, k, slices, cost, ...]) | Compute `pass@k` / `pass^k` / success-rate over a k-trial run. | | [`wilson_interval`](_autosummary/ek.agents.html.md#ek.agents.wilson_interval)(successes, n, \*[, z]) | Wilson score interval for a proportion -- correct in the tails, unlike Wald/CLT. | | [`bootstrap_ci`](_autosummary/ek.agents.html.md#ek.agents.bootstrap_ci)(values, statistic, \*[, ...]) | Percentile bootstrap CI for any `statistic` over `values` (deterministic by seed). | | [`newcombe_difference`](_autosummary/ek.agents.html.md#ek.agents.newcombe_difference)(successes_a, n_a, ...[, z]) | Newcombe score interval for the **difference** of two proportions `p_a - p_b`. | | [`difference_upper_bound`](_autosummary/ek.agents.html.md#ek.agents.difference_upper_bound)(value_a, ci_a, ...[, z]) | Upper bound of `a - b` for two statistics known only through their CIs. | | [`per_million`](_autosummary/ek.agents.html.md#ek.agents.per_million)(input, output, \*[, ...]) | Build a [`ModelPrice`](_autosummary/ek.agents.html.md#ek.agents.ModelPrice) from the human-facing **USD per 1M tokens** figures. | | [`dollars`](_autosummary/ek.agents.html.md#ek.agents.dollars)(cost, price, \*[, prices, batch]) | Monetary cost of one [`Cost`](_autosummary/ek.agents.base.html.md#ek.agents.base.Cost) tally, in USD. | | [`episode_dollars`](_autosummary/ek.agents.html.md#ek.agents.episode_dollars)(episode, \*[, prices, price, ...]) | Monetary cost of one episode (0.0 if it carries no [`Cost`](_autosummary/ek.agents.base.html.md#ek.agents.base.Cost)). | | [`cost_of_pass`](_autosummary/ek.agents.html.md#ek.agents.cost_of_pass)(total_cost, n_success) | Expected monetary cost of one *successful* task: `total_cost / n_success`. | | [`cost_report`](_autosummary/ek.agents.html.md#ek.agents.cost_report)(episodes, \*[, prices, price]) | The quality x cost x latency triple, reported **together**. | | [`load_prices`](_autosummary/ek.agents.html.md#ek.agents.load_prices)(source) | Parse a LiteLLM-format price catalog into a `{model: ModelPrice}` table. | | [`price_of`](_autosummary/ek.agents.html.md#ek.agents.price_of)(model, \*[, prices]) | Resolve a model's rates from an injected catalog, or fail *actionably*. | | [`judge_validation`](_autosummary/ek.agents.html.md#ek.agents.judge_validation)(judge_labels, human_labels, \*) | Certify a judge against human labels **before** trusting it. | | [`pairwise_judge`](_autosummary/ek.agents.html.md#ek.agents.pairwise_judge)(judge, a, b, \*[, criteria, swap]) | Pairwise preference for `a` over `b`, **corrected for position bias**. | | [`run_suite`](_autosummary/ek.agents.html.md#ek.agents.run_suite)(agent, tasks, \*[, k, check, ...]) | Run `agent` over a task suite, `k` trials per task, and report reliability + cost. | | [`save_agent_baseline`](_autosummary/ek.agents.html.md#ek.agents.save_agent_baseline)(report, name, \*[, rootdir]) | Freeze a run as a named baseline (including its provenance, so the gate can guard it). | | [`load_agent_baseline`](_autosummary/ek.agents.html.md#ek.agents.load_agent_baseline)(name, \*[, rootdir]) | Load a named agent baseline (or `None`). | | [`agent_regression_gate`](_autosummary/ek.agents.html.md#ek.agents.agent_regression_gate)(report, baseline, \*[, ...]) | Fail only when the evidence says the agent **really** got worse -- not on noise. | | [`as_agent`](_autosummary/ek.agents.html.md#ek.agents.as_agent)(fn) | Wrap a plain `input -> answer` function into the harness's `TaskSpec -> Episode` shape. | | [`trajectory_from_messages`](_autosummary/ek.agents.html.md#ek.agents.trajectory_from_messages)(messages) | Parse a chat transcript into a [`Trajectory`](_autosummary/ek.agents.base.html.md#ek.agents.base.Trajectory). | | [`episode_from_messages`](_autosummary/ek.agents.html.md#ek.agents.episode_from_messages)(messages, \*[, ...]) | Build a full [`Episode`](_autosummary/ek.agents.base.html.md#ek.agents.base.Episode) from a transcript (+ optional usage/state). | | [`cost_from_usage`](_autosummary/ek.agents.html.md#ek.agents.cost_from_usage)(usage, \*[, latency_s]) | Build a [`Cost`](_autosummary/ek.agents.base.html.md#ek.agents.base.Cost) from a provider `usage` object or dict. | | [`from_inspect_sample`](_autosummary/ek.agents.html.md#ek.agents.from_inspect_sample)(sample, \*[, task_id]) | Adapt an Inspect AI `EvalSample`-shaped object into an [`Episode`](_autosummary/ek.agents.html.md#ek.agents.Episode) (duck-typed). | | [`from_deepeval_test_case`](_autosummary/ek.agents.html.md#ek.agents.from_deepeval_test_case)(case, \*[, task_id]) | Adapt a DeepEval `LLMTestCase`-shaped object into an [`Episode`](_autosummary/ek.agents.html.md#ek.agents.Episode) (duck-typed). | ### Classes | [`TaskSpec`](_autosummary/ek.agents.html.md#ek.agents.TaskSpec)(task_id[, input, gold, value, ...]) | One task in a suite: its input, its oracle, its value, and its allowed tools. | |----------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------| | [`ToolSpec`](_autosummary/ek.agents.html.md#ek.agents.ToolSpec)(name[, params, importance, destructive]) | One tool the agent may call: its argument schema and its error costs. | | [`Episode`](_autosummary/ek.agents.html.md#ek.agents.Episode)([task_id, trajectory, output, ...]) | Layer B: one agent run over one task -- the object agent evaluation scores. | | [`Trajectory`](_autosummary/ek.agents.html.md#ek.agents.Trajectory)([steps]) | The ordered sequence of [`Step`](_autosummary/ek.agents.html.md#ek.agents.Step) s an agent took. | | [`Step`](_autosummary/ek.agents.html.md#ek.agents.Step)(tool[, args, observation, error]) | One step of a trajectory: a tool call and what came back. | | [`Cost`](_autosummary/ek.agents.html.md#ek.agents.Cost)([input_tokens, output_tokens, ...]) | What one episode consumed: tokens (by kind), retries, and wall-clock. | | [`RunProvenance`](_autosummary/ek.agents.html.md#ek.agents.RunProvenance)([seed, temperature, model, ...]) | The *hidden eval variables* that decide whether two runs are even comparable. | | [`TaskSuccessMetric`](_autosummary/ek.agents.html.md#ek.agents.TaskSuccessMetric)([check]) | Did the episode complete the task? (`1.0`/`0.0`; aggregate = success rate). | | [`CostPerSuccessMetric`](_autosummary/ek.agents.html.md#ek.agents.CostPerSuccessMetric)([check, price, prices]) | Cost-of-Pass: dollars per **successfully completed** task (**lower is better**). | | [`ToolCallMetric`](_autosummary/ek.agents.html.md#ek.agents.ToolCallMetric)([level, canonicalizer, grammar]) | Tool-call correctness (BFCL AST match) as a cost-weighted [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric). | | [`TrajectoryMetric`](_autosummary/ek.agents.html.md#ek.agents.TrajectoryMetric)([scheme, grammar, ...]) | Cost-weighted trajectory distance as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric) (**lower is better**). | | [`JudgeMetric`](_autosummary/ek.agents.html.md#ek.agents.JudgeMetric)(judge[, criteria]) | A **reference-based** LLM judge as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric) (the `score()` side). | | [`Checker`](_autosummary/ek.agents.html.md#ek.agents.Checker)(\*args, \*\*kwargs) | The success oracle: did this episode leave the world in the required state? | | [`ReliabilityReport`](_autosummary/ek.agents.html.md#ek.agents.ReliabilityReport)([k, n_tasks, n_trials, ...]) | Corpus-level reliability over a k-trial run, with per-task and per-slice cuts. | | [`ModelPrice`](_autosummary/ek.agents.html.md#ek.agents.ModelPrice)([input, output, cached_input, ...]) | Per-token rates for one model, in **USD per token**. | | [`JudgeSignal`](_autosummary/ek.agents.html.md#ek.agents.JudgeSignal)(judge[, criteria, cost_tier, name]) | A reference-free, criteria-only LLM judge as a [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal). | | [`AgentGateResult`](_autosummary/ek.agents.html.md#ek.agents.AgentGateResult)([passed, reasons, ...]) | Outcome of a variance-aware agent regression check. | ### Exceptions | [`UnknownModelPrice`](_autosummary/ek.agents.html.md#ek.agents.UnknownModelPrice) | Raised when a model's rates are not in the injected price catalog. | |----------------------------------------------------------------------|----------------------------------------------------------------------| ### *class* ek.agents.AgentGateResult(passed=True, reasons=, pass_hat_k=None, pass_hat_k_baseline=None, success_rate=None, success_rate_baseline=None, cost_per_success=None, cost_per_success_baseline=None, underpowered=False) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Outcome of a variance-aware agent regression check. ### *class* ek.agents.Checker(\*args, \*\*kwargs) Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol) The success oracle: did this episode leave the world in the required state? Injected, never owned by `ek` – a real suite plugs in a database-state comparison, a hidden-test runner, or a programmatic predicate. It must run **isolated** from the agent. ### *class* ek.agents.Cost(input_tokens=0, output_tokens=0, cached_input_tokens=0, reasoning_tokens=0, retries=0, latency_s=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) What one episode consumed: tokens (by kind), retries, and wall-clock. Token *kinds* are tracked separately because they are **priced asymmetrically** – output typically costs ~5x input, cached input bills at a fraction, and reasoning tokens are billed. This is precisely why the objective is *dollars* per successful task and never a raw token count (`misc/docs/ek_11`). Pricing lives in [`ek.agents.cost`](_autosummary/ek.agents.cost.html.md#module-ek.agents.cost); this dataclass is the provider-neutral tally. ### Example ```pycon >>> a = Cost(input_tokens=100, output_tokens=50) >>> b = Cost(input_tokens=10, output_tokens=5, retries=1) >>> c = a + b >>> c.input_tokens, c.output_tokens, c.retries (110, 55, 1) ``` #### *property* total_tokens *: [int](https://docs.python.org/3/builtins/functions.html#int)* All billed tokens (a rough size proxy – *not* a cost; see `dollars()`). ### *class* ek.agents.CostPerSuccessMetric(check=None, , price=None, prices=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Cost-of-Pass: dollars per **successfully completed** task (**lower is better**). The per-episode `Score.value` is that episode’s dollar cost – the real number is the corpus `aggregate`, `sum(dollars) / sum(successes)`, which is `inf` when nothing succeeded (infeasibility is the honest answer; a per-token metric reporting “$0.003/call” for an agent that never succeeds is actively misleading). * **Parameters:** * **check** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The success oracle (as in [`TaskSuccessMetric`](_autosummary/ek.agents.html.md#ek.agents.TaskSuccessMetric)). * **price** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`ModelPrice`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.ModelPrice)]) – A [`ModelPrice`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.ModelPrice) applied to every episode, or `None` to resolve each episode’s model against `prices`. * **prices** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`ModelPrice`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.ModelPrice)]]) – A `{model: ModelPrice}` catalog (see [`ek.agents.load_prices()`](_autosummary/ek.agents.html.md#ek.agents.load_prices)). #### aggregate(scores) `sum(dollars) / sum(successes)` – a ratio of two sums, never a mean of ratios. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.agents.Episode(task_id='', trajectory=, output=None, final_state=None, cost=None, success=None, run=None, provenance=None, meta=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Layer B: one agent run over one task – the object agent evaluation scores. The direct analog of `AnnotatedExtraction`: it rides alongside the frozen Layer-A grammar and carries the run’s metadata. It is scored *offline* against gold ([`ek.score()`](_autosummary/ek.html.md#ek.score) / [`ek.evaluate()`](_autosummary/ek.html.md#ek.evaluate)) and estimated *online* against a judge or a consensus ([`ek.estimate_quality()`](_autosummary/ek.html.md#ek.estimate_quality)) – the same object, both halves. * **Parameters:** * **task_id** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Which task this episode answers (the k-trial grouping key). * **trajectory** ([`Trajectory`](_autosummary/ek.agents.base.html.md#ek.agents.base.Trajectory)) – The tool calls taken. * **output** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The agent’s final user-facing answer. * **final_state** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The end state of the world (the DB, the filesystem) – what a state-based oracle actually grades. Success is a *final-state* check, not a surface-text check. * **cost** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Cost`](_autosummary/ek.agents.base.html.md#ek.agents.base.Cost)]) – The [`Cost`](_autosummary/ek.agents.html.md#ek.agents.Cost) consumed. * **success** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`bool`](https://docs.python.org/3/builtins/functions.html#bool)]) – Filled in by a checker; `None` until graded. * **run** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`RunProvenance`](_autosummary/ek.agents.base.html.md#ek.agents.base.RunProvenance)]) – The [`RunProvenance`](_autosummary/ek.agents.html.md#ek.agents.RunProvenance) (seed, model, simulator, suite version). * **provenance** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Provenance`](_autosummary/ek.base.html.md#ek.base.Provenance)]) – Optional core [`Provenance`](_autosummary/ek.base.html.md#ek.base.Provenance) for click-to-source audit. * **meta** ([`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)) – Anything else the bridge captured. #### *property* calls *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)* The `(tool, args)` pairs taken, in order. #### graded(success) A copy with `success` set (never mutates – grading stays idempotent). `meta` is copied too: `dataclasses.replace` is a *shallow* copy, so without this the “copy” would share the original’s mutable `meta` dict. * **Return type:** [`Episode`](_autosummary/ek.agents.base.html.md#ek.agents.base.Episode) ### *class* ek.agents.JudgeMetric(judge, criteria='') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A **reference-based** LLM judge as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric) (the `score()` side). A judge that reads the gold answer is not a reference-free signal, so it does not belong in [`ek.estimate_quality()`](_autosummary/ek.html.md#ek.estimate_quality). It lives here. * **Parameters:** * **judge** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)) – Injected `(output, *, criteria, reference) -> float` in `[0, 1]`. * **criteria** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The rubric. #### aggregate(scores) Mean judge score (a judge score *is* per-item; there is no global accumulation). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.agents.JudgeSignal(judge, criteria='', cost_tier=5, name='judge') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A reference-free, criteria-only LLM judge as a [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal). * **Parameters:** * **judge** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)) – An **injected** callable `(output, *, criteria) -> float`. `ek` core never imports an LLM client; you pass your own (or an `ek[judge]` adapter). * **criteria** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The rubric the judge scores against, bound at construction (exactly as [`LogprobSignal`](_autosummary/ek.qe.signals.html.md#ek.qe.signals.LogprobSignal) binds its aggregator), so the object still satisfies `Signal.__call__(output) -> float`. * **cost_tier** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – `JUDGE_COST_TIER` – the most expensive tier. #### finding(target, rationale) Surface a judge’s *rationale* as a [`Finding`](_autosummary/ek.base.html.md#ek.base.Finding) (not just a scalar). A judge’s reasoning is auditable evidence; a bare number throws it away. * **Return type:** [`Finding`](_autosummary/ek.base.html.md#ek.base.Finding) ### *class* ek.agents.ModelPrice(input=0.0, output=0.0, cached_input=None, reasoning=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Per-token rates for one model, in **USD per token**. * **Parameters:** * **input** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – USD per input (prompt) token. * **output** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – USD per output (completion) token. * **cached_input** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`float`](https://docs.python.org/3/builtins/functions.html#float)]) – USD per cached-input token; defaults to `DEFAULT_CACHE_DISCOUNT * input` when not declared. * **reasoning** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`float`](https://docs.python.org/3/builtins/functions.html#float)]) – USD per reasoning token; defaults to the `output` rate (reasoning tokens are billed as output by the major providers). #### *property* cached_rate *: [float](https://docs.python.org/3/builtins/functions.html#float)* The cached-input rate (explicit, else the discounted input rate). #### *property* reasoning_rate *: [float](https://docs.python.org/3/builtins/functions.html#float)* The reasoning-token rate (explicit, else the output rate). ### *class* ek.agents.ReliabilityReport(k=1, n_tasks=0, n_trials=0, n_success=0, success_rate=0.0, success_ci=(0.0, 1.0), pass_at_k=None, pass_hat_k=None, pass_hat_k_ci=(0.0, 1.0), stochastic=True, per_task=, per_slice=, cost=, detail=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Corpus-level reliability over a k-trial run, with per-task and per-slice cuts. `success_rate` is micro (over all trials); `pass_at_k`/`pass_hat_k` are averaged over *tasks* (each task’s own unbiased estimator), never over trials. ### *class* ek.agents.RunProvenance(seed=None, temperature=None, model='', simulator='', suite_version='', scaffold='') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The *hidden eval variables* that decide whether two runs are even comparable. Agent evaluation is stochastic, and several of its inputs are themselves models. In a tau-bench-style suite the **user simulator is an LLM agent**, so its model/prompt/version silently changes results run to run. Recording these is what lets the harness *refuse* to compare a baseline against a run that changed the simulator or the suite – the same discipline `ek`’s offline harness already applies when it refuses to compare across two different metrics (`misc/docs/ek_08`, `misc/docs/ek_11`). * **Parameters:** * **seed** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`int`](https://docs.python.org/3/builtins/functions.html#int)]) – RNG seed for the trial (`None` = unseeded/nondeterministic). * **temperature** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`float`](https://docs.python.org/3/builtins/functions.html#float)]) – Sampling temperature of the agent under test. * **model** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The agent’s model id. * **simulator** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The **user-simulator** id/version – an eval variable, not a constant. * **suite_version** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Version of the task suite (contamination and task fixes move scores). * **scaffold** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The agent scaffold/harness id (the same model moves points on scaffold alone). #### comparability_key() The fields that must match for two runs to be comparable (seed excluded). Seed and temperature are *expected* to vary across trials; the simulator, the suite version and the scaffold are not – changing them invalidates a baseline. * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple) ### *class* ek.agents.Step(tool, args=, observation=None, error=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One step of a trajectory: a tool call and what came back. `error` is set when the call itself failed (a bad argument, a tool exception) – error *recovery* across subsequent steps is itself an evaluable signal. #### *property* call *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)* The `(tool, args)` pair – what tool-call correctness actually compares. ### *class* ek.agents.TaskSpec(task_id, input=None, gold=None, value=1.0, tools=(), slice=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One task in a suite: its input, its oracle, its value, and its allowed tools. * **Parameters:** * **task_id** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Stable identifier (the grouping key for k-trial reliability). * **input** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Whatever the agent under test is called with. * **gold** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The reference the checker grades against (a goal state, an expected answer, a gold trajectory – the checker decides how to read it). * **value** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Task-level cost weight: how much a completed task is worth. The task-level analog of `FieldSpec.importance`; feeds value-weighted cost accounting. * **tools** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`ToolSpec`](_autosummary/ek.agents.base.html.md#ek.agents.base.ToolSpec)]) – The tools this task permits (the Layer-A grammar for the task). * **slice** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – Optional slice label (domain, difficulty, language) – per-slice cuts are mandatory, not optional (a low pass^k concentrated in one hard slice is a different problem from uniform flakiness). #### grammar() The Layer-A grammar for this task’s tool set. * **Return type:** [`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar) ### *class* ek.agents.TaskSuccessMetric(check=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Did the episode complete the task? (`1.0`/`0.0`; aggregate = success rate). * **Parameters:** **check** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – A [`Checker`](_autosummary/ek.agents.html.md#ek.agents.Checker) – a registered name (`"output"`, `"final_state"`, `"recorded"`) or any `(episode, gold) -> bool` callable. Defaults to `"output"`; **inject a state-based or executable oracle for a real suite.** #### aggregate(scores) The success **rate** over the corpus. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.agents.ToolCallMetric(level='call', , canonicalizer=None, grammar=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Tool-call correctness (BFCL AST match) as a cost-weighted [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric). * **Parameters:** * **level** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – `"call"` (default) scores whole calls – a call is a TP only on an exact AST match; `"arg"` gives partial credit at argument granularity. * **canonicalizer** – `str -> str` applied to string arguments before comparison (BFCL’s case/whitespace folding). Defaults to `ek`’s default canonicalizer. * **grammar** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar)]) – Layer-A grammar supplying per-tool and per-argument cost weights (may also be passed per call). The `Score` carries `tp`/`fp`/`fn` in `detail` so [`ek.evaluate()`](_autosummary/ek.html.md#ek.evaluate) aggregates a correct **micro**-averaged F1 over the corpus rather than averaging per-episode F1s. #### aggregate(scores) Micro-averaged F1 over the corpus (sum TP/FP/FN, then divide) – never a mean. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.agents.ToolSpec(name, params=, importance=None, destructive=False) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One tool the agent may call: its argument schema and its error costs. * **Parameters:** * **name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The tool/function name (the AST-match ground truth). * **params** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`FieldSpec`](_autosummary/ek.base.html.md#ek.base.FieldSpec)]) – Argument name -> [`FieldSpec`](_autosummary/ek.base.html.md#ek.base.FieldSpec). Each spec’s `importance` is the cost of getting *that argument* wrong, and its `domain` feeds validators – the same contract as an extracted field. * **importance** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`float`](https://docs.python.org/3/builtins/functions.html#float)]) – Cost weight of the *call itself* (a missing or spurious call). Defaults to `DESTRUCTIVE_WEIGHT` when `destructive` is set, else 1.0. * **destructive** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether calling this tool mutates the world irreversibly (a refund, a delete, a send). Drives the default weight and the safety validators. #### to_node_type() Render this tool as a Layer-A [`NodeType`](_autosummary/ek.base.html.md#ek.base.NodeType) (args as fields). * **Return type:** [`NodeType`](_autosummary/ek.base.html.md#ek.base.NodeType) #### *property* weight *: [float](https://docs.python.org/3/builtins/functions.html#float)* The call-level cost weight (explicit, else destructive-aware default). ### *class* ek.agents.Trajectory(steps=()) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The ordered sequence of [`Step`](_autosummary/ek.agents.html.md#ek.agents.Step) s an agent took. A trajectory is *linear* – which is why it is scored with a sequence edit distance, not a graph edit distance (see [`ek.agents.trajectory`](_autosummary/ek.agents.trajectory.html.md#module-ek.agents.trajectory)). ### Example ```pycon >>> t = Trajectory([Step("search", {"q": "cat"}), Step("answer", {"a": "meow"})]) >>> len(t), t.tools (2, ('search', 'answer')) ``` #### *property* calls *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)* The `(tool, args)` pairs in call order. #### *property* tools *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)* The tool names in call order. ### *class* ek.agents.TrajectoryMetric(scheme='in_order', , grammar=None, canonicalizer=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Cost-weighted trajectory distance as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric) (**lower is better**). * **Parameters:** * **scheme** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – One of `SCHEMES`. `in_order` (default) – Needleman-Wunsch edit distance; order matters, gaps allowed. `exact` – 0 iff the call sequences are identical, else the full mass. `any_order` – multiset distance; order ignored (the calls, not the path). `superset` – 0 iff every gold call occurs in the prediction; extra steps are free (use when the reference lists *required* calls rather than the whole path). * **grammar** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar)]) – Layer-A grammar supplying step/argument cost weights (may be passed per call). * **canonicalizer** – `str -> str` applied to string arguments before comparison. #### aggregate(scores) Corpus distance = total raw distance / total maximum distance (never a mean). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *exception* ek.agents.UnknownModelPrice Bases: [`KeyError`](https://docs.python.org/3/builtins/exceptions.html#KeyError) Raised when a model’s rates are not in the injected price catalog. ### ek.agents.agent_regression_gate(report, baseline, , tolerance=0.0, cost_tolerance=0.0, rootdir=None) Fail only when the evidence says the agent **really** got worse – not on noise. Tests the **difference** `current - baseline`, with *both* runs’ uncertainty folded in: a Newcombe score interval on the success rate, and (for `k > 1`) a combined-SE bound on `pass^k`. A regression is declared only when the upper bound of that change is below `-tolerance` – i.e. we are confident the true change is negative. Two tempting shortcuts are **both wrong**, and this module has shipped and fixed each: - Comparing our interval to the baseline’s **point**. The baseline is *itself* a noisy estimate; a Wilson upper bound only reaches 1.0 when every trial passed, so a single flake against a lucky baseline reads as a “confident regression”. - Asking whether the two 95% intervals **overlap**. That is not a 5% test but roughly a 0.5% one: it buys a calm gate by going blind (a 15-point drop on a 60-task suite becomes invisible). A gate that cannot see a real regression is as broken as one that cries wolf. A result that is merely *worse but not significantly so* is not a clean pass either – it is an underpowered experiment, and the result says so (`underpowered`, plus a warning). * **Raises:** [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if the baseline was produced with a different **user simulator**, suite version, or scaffold – those runs are not comparable, and silently comparing them would be a category error (the same reason the offline gate refuses to compare two different metrics); if the baseline recorded zero trials (a permanent free pass); or if it is malformed. * **Return type:** [`AgentGateResult`](_autosummary/ek.agents.harness.html.md#ek.agents.harness.AgentGateResult) ### ek.agents.as_agent(fn) Wrap a plain `input -> answer` function into the harness’s `TaskSpec -> Episode` shape. Progressive disclosure: the trivial agent should not have to learn the Episode type. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable) ### Example ```pycon >>> from ek.agents.base import TaskSpec >>> agent = as_agent(str.upper) >>> agent(TaskSpec("t1", input="hi")).output 'HI' ``` ### ek.agents.as_call(x) Coerce a [`Step`](_autosummary/ek.agents.html.md#ek.agents.Step), `(tool, args)` tuple, or `{"tool":…, "args":…}` dict. Gold tool calls are usually written as bare pairs or dicts (no observation), while a prediction arrives as a [`Step`](_autosummary/ek.agents.html.md#ek.agents.Step); both must compare. Progressive disclosure: the simple literal works, the rich object works. * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple) ### Example ```pycon >>> as_call(Step("search", {"q": "cat"})) ('search', {'q': 'cat'}) >>> as_call(("search", {"q": "cat"})) ('search', {'q': 'cat'}) >>> as_call({"tool": "search", "args": {"q": "cat"}}) ('search', {'q': 'cat'}) >>> as_call("search") ('search', {}) ``` ### ek.agents.bootstrap_ci(values, statistic, , resamples=2000, alpha=0.05, seed=0) Percentile bootstrap CI for any `statistic` over `values` (deterministic by seed). Used for the **cost-per-success ratio**, whose distribution is skewed and heavy-tailed (a normal interval on it is meaningless). * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple) ### Example ```pycon >>> lo, hi = bootstrap_ci([1.0] * 20, lambda v: sum(v) / len(v), resamples=50) >>> lo == hi == 1.0 True ``` ### ek.agents.cost_from_usage(usage, , latency_s=None) Build a [`Cost`](_autosummary/ek.agents.base.html.md#ek.agents.base.Cost) from a provider `usage` object or dict. Understands the OpenAI (`prompt_tokens` / `completion_tokens`) and Anthropic (`input_tokens` / `output_tokens` / `cache_read_input_tokens`) spellings. * **Return type:** [`Cost`](_autosummary/ek.agents.base.html.md#ek.agents.base.Cost) ### Example ```pycon >>> c = cost_from_usage({"prompt_tokens": 100, "completion_tokens": 20}) >>> c.input_tokens, c.output_tokens (100, 20) ``` ### ek.agents.cost_of_pass(total_cost, n_success) Expected monetary cost of one *successful* task: `total_cost / n_success`. Returns `inf` when nothing succeeded – infeasibility is the honest answer, not a cheap-looking zero (this divergence is the point of the metric). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### Example ```pycon >>> cost_of_pass(10.0, 4) 2.5 >>> cost_of_pass(10.0, 0) inf ``` ### ek.agents.cost_report(episodes, , prices=None, price=None) The quality x cost x latency triple, reported **together**. Scoring accuracy *without* cost lets an agent chase tiny gains with unbounded API calls, so a report that omits cost is not a report. Returns `n`, `n_success`, `success_rate`, `total_dollars`, `cost_per_success` (Cost-of-Pass), `mean_latency_s` and `total_tokens`. With **no** `price`/`prices`, dollar figures are `None` rather than a fabricated zero: `ek` will not invent rates, and a silent 0.0 would make a costly agent look free. Tokens and latency are still reported. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### Example ```pycon >>> from ek.agents.base import Cost, Episode >>> p = per_million(1.0, 1.0) >>> eps = [Episode(task_id="a", cost=Cost(input_tokens=1_000_000), success=True), ... Episode(task_id="b", cost=Cost(input_tokens=1_000_000), success=False)] >>> r = cost_report(eps, price=p) >>> r["success_rate"], round(r["cost_per_success"], 2) (0.5, 2.0) >>> cost_report(eps)["cost_per_success"] is None # no rates -> no dollars invented True ``` ### ek.agents.difference_upper_bound(value_a, ci_a, value_b, ci_b, , z=1.96) Upper bound of `a - b` for two statistics known only through their CIs. Used for `pass^k` (a mean of per-task estimators, not a raw proportion), where a Newcombe interval does not apply but the same principle does: test the **difference**, combining both runs’ uncertainty, rather than asking whether two intervals happen to overlap. * **Return type:** [`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`float`](https://docs.python.org/3/builtins/functions.html#float)] ### ek.agents.dollars(cost, price, , prices=None, batch=False) Monetary cost of one [`Cost`](_autosummary/ek.agents.base.html.md#ek.agents.base.Cost) tally, in USD. * **Parameters:** * **cost** ([`Cost`](_autosummary/ek.agents.base.html.md#ek.agents.base.Cost)) – The token/retry tally. * **price** (`Union`[[`ModelPrice`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.ModelPrice), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – A [`ModelPrice`](_autosummary/ek.agents.html.md#ek.agents.ModelPrice), or a model name to resolve against `prices`. * **prices** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`ModelPrice`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.ModelPrice)]]) – The injected price catalog (required when `price` is a name). * **batch** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Apply the batch-API discount (`DEFAULT_BATCH_DISCOUNT`). * **Raises:** [**UnknownModelPrice**](_autosummary/ek.agents.html.md#ek.agents.UnknownModelPrice) – if a model name is given with no rates for it. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.agents.episode_dollars(episode, , prices=None, price=None, batch=False) Monetary cost of one episode (0.0 if it carries no [`Cost`](_autosummary/ek.agents.base.html.md#ek.agents.base.Cost)). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.agents.episode_from_messages(messages, , task_id='', usage=None, output=None, final_state=None, latency_s=None) Build a full [`Episode`](_autosummary/ek.agents.base.html.md#ek.agents.base.Episode) from a transcript (+ optional usage/state). `output` defaults to the last assistant text – the agent’s final answer. * **Return type:** [`Episode`](_autosummary/ek.agents.base.html.md#ek.agents.base.Episode) ### ek.agents.final_state_match(episode, gold) Compare the episode’s **final state** with the goal state (the tau-bench oracle). * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.agents.from_deepeval_test_case(case, , task_id='') Adapt a DeepEval `LLMTestCase`-shaped object into an [`Episode`](_autosummary/ek.agents.html.md#ek.agents.Episode) (duck-typed). Reads `.actual_output` and `.tools_called`. Note DeepEval phones home by default (Confident-AI telemetry) – disable it before use if that matters to you; `ek` never enables it. * **Return type:** [`Episode`](_autosummary/ek.agents.base.html.md#ek.agents.base.Episode) ### ek.agents.from_inspect_sample(sample, , task_id='') Adapt an Inspect AI `EvalSample`-shaped object into an [`Episode`](_autosummary/ek.agents.html.md#ek.agents.Episode) (duck-typed). Reads `.messages`, `.output` and `.id` – no `inspect_ai` import required, so scoring an Inspect log never drags the harness into `ek`’s dependency closure. * **Return type:** [`Episode`](_autosummary/ek.agents.base.html.md#ek.agents.base.Episode) ### ek.agents.is_episode(x) Duck-typed episode check (core never imports this module – it checks the *shape*). * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.agents.judge_validation(judge_labels, human_labels, , level='nominal') Certify a judge against human labels **before** trusting it. No new code – reuse the IAA. A judge is just another annotator, so `ek`’s existing inter-annotator-agreement machinery (which already tells you the *ceiling* of a gold standard) is exactly the right instrument: Krippendorff’s alpha is the default (it alone handles any number of raters, any measurement level, and missing data); Cohen’s kappa applies for exactly two raters on complete nominal data. This reproduces the MT-Bench judge-vs-human protocol. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) * **Returns:** `alpha`, `kappa`, `percent_agreement`, `n`, and a `verdict` against the conventional thresholds (alpha >= 0.8 good, 0.67-0.8 tentative, below that unreliable). ### Example ```pycon >>> r = judge_validation([1, 1, 0, 0], [1, 1, 0, 0]) >>> r["alpha"], r["verdict"] (1.0, 'good') ``` ### ek.agents.load_agent_baseline(name, , rootdir=None) Load a named agent baseline (or `None`). * **Return type:** [`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)] ### ek.agents.load_prices(source) Parse a LiteLLM-format price catalog into a `{model: ModelPrice}` table. `source` is a path to (or an already-parsed mapping of) LiteLLM’s MIT-licensed `model_prices_and_context_window.json`. We read the **data file only** and never import the `litellm` SDK – its `enterprise/` subtree is a proprietary carve-out, and the SDK drags a large transitive tree in just to look up a rate. Entries lacking token rates (embeddings, unpriced models) are skipped. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### Example ```pycon >>> table = load_prices({"m": {"input_cost_per_token": 1e-6, ... "output_cost_per_token": 2e-6}}) >>> table["m"].output 2e-06 ``` ### ek.agents.match_calls(pred, gold, , canon=None) Assign predicted calls to gold calls – the layer `FieldMetric` does not provide. Calls are grouped by tool name (a call to the *wrong tool* is never a match). Within a tool name, candidate pairs are scored and taken **globally best-first** (exact-argument matches before partial ones, more agreeing arguments before fewer), so identical repeated calls are interchangeable and a reordered pair is still matched – while a strong pairing can never be stolen by a weaker one that merely happened to come first in the prediction. This is a deterministic greedy approximation to the optimal assignment. It is exact for the common cases (distinct calls, repeats, reorderings); a full min-cost bipartite matching (Hungarian) could differ only on pathological many-to-many partial-overlap sets, and is not worth a dependency here. Returns `(pairs, unmatched_pred, unmatched_gold)`, `pairs` being `(pred_call, gold_call)`. * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple) ### Example ```pycon >>> pairs, up, ug = match_calls([("s", {"q": "b"}), ("s", {"q": "a"})], ... [("s", {"q": "a"}), ("s", {"q": "b"})]) >>> len(pairs), up, ug # order-insensitive within a tool name (2, [], []) ``` ### ek.agents.newcombe_difference(successes_a, n_a, successes_b, n_b, , z=1.96) Newcombe score interval for the **difference** of two proportions `p_a - p_b`. This is the right instrument for “did the agent get worse than its baseline?”, and it is strictly better than the tempting shortcut of *checking whether two 95% CIs overlap*. Non-overlap is not a 5% test – it is roughly a 0.5% one, so it buys its low false-alarm rate by going nearly blind: on a 60-task suite it cannot see a 15-point success-rate drop. A difference interval keeps the same false-alarm rate and recovers most of the power, because the variance of a difference is not the sum of two interval widths. Newcombe (1998) method 10: build each proportion’s Wilson interval, then > lower = (p_a - p_b) - sqrt((p_a - l_a)^2 + (u_b - p_b)^2) > upper = (p_a - p_b) + sqrt((u_a - p_a)^2 + (p_b - l_b)^2) Returns `(lower, upper)`. The difference is a *confident regression* when `upper < 0`. ### Example ```pycon >>> lo, hi = newcombe_difference(50, 100, 90, 100) # 50% now vs 90% before >>> hi < 0 # confidently worse True >>> lo, hi = newcombe_difference(89, 100, 90, 100) # 89% vs 90% -- noise >>> hi > 0 True ``` * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple) ### ek.agents.output_match(episode, gold) Canonicalized exact match on the final answer (the GAIA-style oracle). The weakest useful oracle, and the default only because it needs nothing. Prefer a state-based or executable checker whenever the task has one. * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.agents.pairwise_judge(judge, a, b, , criteria='', swap=True) Pairwise preference for `a` over `b`, **corrected for position bias**. Position bias is not a rounding error – judges materially prefer whichever answer they see first. The standard correction is to ask twice with the order swapped and average, which is why this cannot be a plain `Signal` (it needs two outputs). * **Parameters:** * **judge** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)) – Injected `(a, b, *, criteria) -> float` in `[0, 1]`, the preference for `a`. * **swap** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Ask again with the arguments swapped and average (default; disable only if your judge is known to be order-invariant). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) * **Returns:** Preference for `a` in `[0, 1]` (`0.5` = a tie). ### Example ```pycon >>> first_wins = lambda x, y, *, criteria="": 1.0 # a maximally position-biased judge >>> pairwise_judge(first_wins, "A", "B") # swapping exposes it as a tie 0.5 ``` ### ek.agents.pass_at_k(, n, c, k) Unbiased `pass@k`: probability that **at least one** of k samples succeeds. The HumanEval estimator `1 - C(n-c, k) / C(n, k)` over `n` trials with `c` successes. This is the **capability** metric – right when a single success is enough (offline candidate generation behind a verifier). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### Example ```pycon >>> round(pass_at_k(n=10, c=1, k=1), 3) 0.1 >>> pass_at_k(n=10, c=10, k=5) 1.0 ``` ### ek.agents.pass_hat_k(, n, c, k) Unbiased `pass^k`: probability that **all k** independent trials succeed. The tau-bench estimator `C(c, k) / C(n, k)`; for a per-trial success probability `p` it decays to `p**k`. This is the **reliability** metric – the one that matters when consistency *is* the product. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### Example ```pycon >>> pass_hat_k(n=8, c=8, k=8) 1.0 >>> pass_hat_k(n=8, c=4, k=2) 0.21428571428571427 ``` ### ek.agents.per_million(input, output, , cached_input=None, reasoning=None) Build a [`ModelPrice`](_autosummary/ek.agents.html.md#ek.agents.ModelPrice) from the human-facing **USD per 1M tokens** figures. * **Return type:** [`ModelPrice`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.ModelPrice) ### Example ```pycon >>> p = per_million(3.0, 15.0) # $3 in / $15 out per 1M tokens >>> round(p.input, 9), round(p.output, 9) (3e-06, 1.5e-05) ``` ### ek.agents.price_of(model, , prices=None) Resolve a model’s rates from an injected catalog, or fail *actionably*. `ek` deliberately ships no built-in price table (rates go stale monthly and would be magic numbers). Supply one with `prices=`, or load the LiteLLM catalog with [`load_prices()`](_autosummary/ek.agents.html.md#ek.agents.load_prices). * **Return type:** [`ModelPrice`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.ModelPrice) ### ek.agents.recorded_success(episode, gold) Trust a `success` already recorded on the episode (e.g. by an external harness). * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.agents.reliability(source, , k=1, slices=None, cost=None, warn_deterministic=True) Compute `pass@k` / `pass^k` / success-rate over a k-trial run. * **Parameters:** * **source** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Graded [`Episode`](_autosummary/ek.agents.base.html.md#ek.agents.base.Episode) s (grouped by `task_id`), or an already-grouped `{task_id: [bool, ...]}` mapping. * **k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – The k of pass@k / pass^k. Tasks with fewer than k trials are skipped (and counted in `detail['skipped']`) rather than silently mis-estimated. * **slices** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]]) – Optional `{task_id: slice_label}` for per-slice cuts. * **cost** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)]) – An optional cost report (from [`ek.agents.cost_report()`](_autosummary/ek.agents.html.md#ek.agents.cost_report)) to carry along – reliability without cost is only half the picture. * **warn_deterministic** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Warn when **no** task shows mixed outcomes across trials, in which case `pass^k` degenerates to `pass^1` and is not informative. * **Return type:** [`ReliabilityReport`](_autosummary/ek.agents.html.md#ek.agents.ReliabilityReport) ### Example ```pycon >>> r = reliability({"t1": [True, True, False], "t2": [True, True, True]}, k=2) >>> r.n_tasks, r.k (2, 2) >>> round(r.pass_hat_k, 3) # t1: C(2,2)/C(3,2)=1/3 ; t2: 1.0 0.667 ``` ### ek.agents.run_suite(agent, tasks, , k=1, check=None, metrics=None, price=None, prices=None, run=None, seed=None, persist=False, run_id=None, rootdir=None) Run `agent` over a task suite, `k` trials per task, and report reliability + cost. * **Parameters:** * **agent** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`TaskSpec`](_autosummary/ek.agents.base.html.md#ek.agents.base.TaskSpec)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – The system under test: `TaskSpec -> Episode` (a bare answer is wrapped). It receives the whole spec – `.input`, `.tools`, `.gold` – not just the input. If it accepts a keyword `seed`, the per-trial seed is passed to it. * **tasks** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – [`TaskSpec`](_autosummary/ek.agents.base.html.md#ek.agents.base.TaskSpec) s (or a `{task_id: spec}` mapping). * **k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Trials per task. `k > 1` is what makes `pass^k` meaningful – and it only means anything if the agent is genuinely stochastic (the report warns if it is not). * **check** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The success oracle – a registered checker name or an `(episode, gold) -> bool` callable. Defaults to `"output"`; **inject a state-based/executable oracle** for a real suite, and run it isolated from the agent. * **metrics** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)]) – Optional `{name: Metric}` scored on every episode against `task.gold`, with the **suite’s Layer-A grammar injected** – this is how the tool/argument cost weights reach a `tool_call`/`trajectory` metric. Results land in `report.detail["metrics"]`. * **price/prices** – Rates for costing the episodes (see [`ek.agents.cost`](_autosummary/ek.agents.cost.html.md#module-ek.agents.cost)). * **run** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`RunProvenance`](_autosummary/ek.agents.base.html.md#ek.agents.base.RunProvenance)]) – [`RunProvenance`](_autosummary/ek.agents.base.html.md#ek.agents.base.RunProvenance) for this run (model, simulator, suite version, scaffold). Recorded on every episode and on any baseline saved from it. * **seed** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`int`](https://docs.python.org/3/builtins/functions.html#int)]) – Base RNG seed. Trial *i* of each task uses `seed + i`: it is recorded on the episode’s provenance **and passed to the agent** when the agent accepts a `seed` keyword. * **persist/run_id/rootdir** – Persist the run to the `runs`/`results` stores. * **Return type:** [`ReliabilityReport`](_autosummary/ek.agents.html.md#ek.agents.ReliabilityReport) * **Returns:** A [`ReliabilityReport`](_autosummary/ek.agents.html.md#ek.agents.ReliabilityReport) carrying `pass@k`, `pass^k` (with a bootstrap CI), the success rate (with a Wilson CI), the per-slice cuts, and the cost report – because reliability without cost is only half the answer. ### ek.agents.save_agent_baseline(report, name, , rootdir=None) Freeze a run as a named baseline (including its provenance, so the gate can guard it). * **Raises:** [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if the run evaluated nothing. An empty baseline is worse than no baseline – it is a *permanent free pass*, because every bound in it is the maximally-uncertain default and no later run can be shown to fall below it. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### ek.agents.suite_grammar(tasks) The union grammar over a whole task suite (later tools win on name collision). * **Return type:** [`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar) ### ek.agents.tool_grammar(\*tools) Build the Layer-A [`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar) from [`ToolSpec`](_autosummary/ek.agents.html.md#ek.agents.ToolSpec) s. The resulting grammar is the frozen cost SSOT every agent metric reads: node cost = the tool’s call weight, field cost = the argument’s importance. * **Return type:** [`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar) ### Example ```pycon >>> from ek.base import FieldSpec >>> g = tool_grammar(ToolSpec("send", params={"to": FieldSpec("to")}, ... destructive=True)) >>> g.node_cost("send") 10.0 ``` ### ek.agents.trajectory_from_messages(messages) Parse a chat transcript into a [`Trajectory`](_autosummary/ek.agents.base.html.md#ek.agents.base.Trajectory). Each assistant tool call becomes a [`Step`](_autosummary/ek.agents.base.html.md#ek.agents.base.Step), and the *next* tool/result message becomes that step’s `observation`. A tool message carrying an error marks the step as errored – error *recovery* across later steps is itself an evaluable signal. * **Return type:** [`Trajectory`](_autosummary/ek.agents.base.html.md#ek.agents.base.Trajectory) ### ek.agents.wilson_interval(successes, n, , z=1.96) Wilson score interval for a proportion – correct in the tails, unlike Wald/CLT. The naive normal (“Wald”) interval is badly wrong at small n and near 0/1 – exactly where agent success rates live. Wilson is the cheap, dependency-free fix. * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple) ### Example ```pycon >>> lo, hi = wilson_interval(0, 10) >>> lo == 0.0, round(hi, 3) (True, 0.278) ``` ### Modules | [`base`](_autosummary/ek.agents.base.html.md#module-ek.agents.base) | Agent-evaluation data shapes: the episode (Layer B) and the task/tool grammar (Layer A). | |-----------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`bridge`](_autosummary/ek.agents.bridge.html.md#module-ek.agents.bridge) | One-way adapters: turn whatever your agent already emits into an `ek` [`Episode`](_autosummary/ek.agents.html.md#ek.agents.Episode). | | [`cost`](_autosummary/ek.agents.cost.html.md#module-ek.agents.cost) | The cost model and the price SSOT: dollars per successfully completed task. | | [`harness`](_autosummary/ek.agents.harness.html.md#module-ek.agents.harness) | The agent task-suite harness: k trials, per-slice cuts, and a gate that survives noise. | | [`judge`](_autosummary/ek.agents.judge.html.md#module-ek.agents.judge) | LLM-as-judge: a reference-free [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal), a reference-based [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric), and the validation that must precede either. | | [`metrics`](_autosummary/ek.agents.metrics.html.md#module-ek.agents.metrics) | Task-success and cost-per-successful-task: the two metrics that drive the whole thing. | | [`toolcalls`](_autosummary/ek.agents.toolcalls.html.md#module-ek.agents.toolcalls) | Tool-call correctness: BFCL-style AST matching over a *multiset* of calls. | | [`trajectory`](_autosummary/ek.agents.trajectory.html.md#module-ek.agents.trajectory) | Trajectory (process) evaluation: a cost-weighted **sequence** edit distance over steps. | # _autosummary/ek.agents.judge.html.md # ek.agents.judge LLM-as-judge: a reference-free [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal), a reference-based [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric), and the validation that must precede either. When a task has no checkable outcome, an LLM judge is the fallback. It is also a \*\*liability unless validated and calibrated\*\*, because its known failure modes have double-digit effect sizes: *position* bias (which answer came first), *verbosity* bias (longer looks better), *self-preference* (a model prefers its own text). `misc/docs/ek_09` has the numbers. Three seams, deliberately kept apart (they are not interchangeable): - [`JudgeSignal`](_autosummary/ek.agents.judge.html.md#ek.agents.judge.JudgeSignal) – **reference-free, criteria-only**. This is the only judge that is a `Signal`: it scores an output against *criteria*, with no gold, so it plugs into [`ek.estimate_quality()`](_autosummary/ek.html.md#ek.estimate_quality)’s `signal -> calibrate -> validate -> decide` cascade. Its > `cost_tier` is the highest in the framework, so an escalation policy runs the cheap signals > first and pays for the judge only on residual uncertainty. - [`JudgeMetric`](_autosummary/ek.agents.judge.html.md#ek.agents.judge.JudgeMetric) – **reference-based**. A judge that needs the gold answer is not a Signal at all; it is a Metric, and it belongs on the [`ek.score()`](_autosummary/ek.html.md#ek.score) side. - [`pairwise_judge()`](_autosummary/ek.agents.judge.html.md#ek.agents.judge.pairwise_judge) – pairwise comparison needs *two* outputs and cannot fit `Signal.__call__(one_output)`. It is a helper, and it **swaps the order and averages** – the standard, necessary correction for position bias. **Hard Rule 1 applies to judges.** A raw judge score is not a probability. Calibrate it before any [`DecisionPolicy`](_autosummary/ek.base.html.md#ek.base.DecisionPolicy) reads it – [`ek.estimate_quality()`](_autosummary/ek.html.md#ek.estimate_quality) already refuses to gate an uncalibrated signal, and a judge is no exception (-> `misc/docs/ek_03`). **Calibrate a judge per signal family.** With a single calibrator, [`ek.estimate_quality()`](_autosummary/ek.html.md#ek.estimate_quality) averages *all* raw signals into one confidence and calibrates that mean. Mixing a judge score with, say, a logprob – different scales, different reliabilities – that way is not statistically sound. Pass `calibrator={"judge": judge_cal, "logprob": logprob_cal}` (or a [`SignalFamilyCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.SignalFamilyCalibrator), which can `fit` one calibrator per signal from labelled rows) so each family is calibrated on its own before the calibrated probabilities are fused. `ek` never imports an LLM SDK: `judge` is an **injected callable**. ### Example ```pycon >>> stub = lambda output, *, criteria="", reference=None: 0.9 if "meow" in output else 0.1 >>> sig = JudgeSignal(stub, criteria="Is the answer helpful?") >>> sig("the cat says meow") 0.9 >>> sig.cost_tier # the most expensive signal family 5 ``` ### Module Attributes | [`JUDGE_COST_TIER`](_autosummary/ek.agents.judge.html.md#ek.agents.judge.JUDGE_COST_TIER) | Judges are the most expensive signal family -- run cheap signals first and escalate. | |--------------------------------------------------------------------|----------------------------------------------------------------------------------------| | [`ALPHA_GOOD`](_autosummary/ek.agents.judge.html.md#ek.agents.judge.ALPHA_GOOD) | >= 0.8 is good agreement, 0.67-0.8 only tentative. | ### Functions | [`judge_validation`](_autosummary/ek.agents.judge.html.md#ek.agents.judge.judge_validation)(judge_labels, human_labels, \*) | Certify a judge against human labels **before** trusting it. | |-----------------------------------------------------------------------------------------------------|------------------------------------------------------------------------| | [`pairwise_judge`](_autosummary/ek.agents.judge.html.md#ek.agents.judge.pairwise_judge)(judge, a, b, \*[, criteria, swap]) | Pairwise preference for `a` over `b`, **corrected for position bias**. | ### Classes | [`JudgeMetric`](_autosummary/ek.agents.judge.html.md#ek.agents.judge.JudgeMetric)(judge[, criteria]) | A **reference-based** LLM judge as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric) (the `score()` side). | |--------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------| | [`JudgeSignal`](_autosummary/ek.agents.judge.html.md#ek.agents.judge.JudgeSignal)(judge[, criteria, cost_tier, name]) | A reference-free, criteria-only LLM judge as a [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal). | ### ek.agents.judge.ALPHA_GOOD *= 0.8* >= 0.8 is good agreement, 0.67-0.8 only tentative. * **Type:** Krippendorff conventions ### ek.agents.judge.JUDGE_COST_TIER *= 5* Judges are the most expensive signal family – run cheap signals first and escalate. ### *class* ek.agents.judge.JudgeMetric(judge, criteria='') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A **reference-based** LLM judge as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric) (the `score()` side). A judge that reads the gold answer is not a reference-free signal, so it does not belong in [`ek.estimate_quality()`](_autosummary/ek.html.md#ek.estimate_quality). It lives here. * **Parameters:** * **judge** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)) – Injected `(output, *, criteria, reference) -> float` in `[0, 1]`. * **criteria** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The rubric. #### aggregate(scores) Mean judge score (a judge score *is* per-item; there is no global accumulation). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.agents.judge.JudgeSignal(judge, criteria='', cost_tier=5, name='judge') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A reference-free, criteria-only LLM judge as a [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal). * **Parameters:** * **judge** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)) – An **injected** callable `(output, *, criteria) -> float`. `ek` core never imports an LLM client; you pass your own (or an `ek[judge]` adapter). * **criteria** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The rubric the judge scores against, bound at construction (exactly as [`LogprobSignal`](_autosummary/ek.qe.signals.html.md#ek.qe.signals.LogprobSignal) binds its aggregator), so the object still satisfies `Signal.__call__(output) -> float`. * **cost_tier** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – [`JUDGE_COST_TIER`](_autosummary/ek.agents.judge.html.md#ek.agents.judge.JUDGE_COST_TIER) – the most expensive tier. #### finding(target, rationale) Surface a judge’s *rationale* as a [`Finding`](_autosummary/ek.base.html.md#ek.base.Finding) (not just a scalar). A judge’s reasoning is auditable evidence; a bare number throws it away. * **Return type:** [`Finding`](_autosummary/ek.base.html.md#ek.base.Finding) ### ek.agents.judge.judge_validation(judge_labels, human_labels, , level='nominal') Certify a judge against human labels **before** trusting it. No new code – reuse the IAA. A judge is just another annotator, so `ek`’s existing inter-annotator-agreement machinery (which already tells you the *ceiling* of a gold standard) is exactly the right instrument: Krippendorff’s alpha is the default (it alone handles any number of raters, any measurement level, and missing data); Cohen’s kappa applies for exactly two raters on complete nominal data. This reproduces the MT-Bench judge-vs-human protocol. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) * **Returns:** `alpha`, `kappa`, `percent_agreement`, `n`, and a `verdict` against the conventional thresholds (alpha >= 0.8 good, 0.67-0.8 tentative, below that unreliable). ### Example ```pycon >>> r = judge_validation([1, 1, 0, 0], [1, 1, 0, 0]) >>> r["alpha"], r["verdict"] (1.0, 'good') ``` ### ek.agents.judge.pairwise_judge(judge, a, b, , criteria='', swap=True) Pairwise preference for `a` over `b`, **corrected for position bias**. Position bias is not a rounding error – judges materially prefer whichever answer they see first. The standard correction is to ask twice with the order swapped and average, which is why this cannot be a plain `Signal` (it needs two outputs). * **Parameters:** * **judge** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)) – Injected `(a, b, *, criteria) -> float` in `[0, 1]`, the preference for `a`. * **swap** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Ask again with the arguments swapped and average (default; disable only if your judge is known to be order-invariant). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) * **Returns:** Preference for `a` in `[0, 1]` (`0.5` = a tie). ### Example ```pycon >>> first_wins = lambda x, y, *, criteria="": 1.0 # a maximally position-biased judge >>> pairwise_judge(first_wins, "A", "B") # swapping exposes it as a tie 0.5 ``` # _autosummary/ek.agents.metrics.html.md # ek.agents.metrics Task-success and cost-per-successful-task: the two metrics that drive the whole thing. Two reference-based metrics over an [`Episode`](_autosummary/ek.agents.base.html.md#ek.agents.base.Episode), both of which fit [`ek.evaluate()`](_autosummary/ek.html.md#ek.evaluate) exactly because their corpus aggregate is a **ratio of two sums**, never a mean of per-item scores – the same discipline that makes `ek`’s WER accumulate globally and its F1 micro-average (`misc/docs/ek_08`, `misc/docs/ek_11`): - [`TaskSuccessMetric`](_autosummary/ek.agents.metrics.html.md#ek.agents.metrics.TaskSuccessMetric) – grade the **final state**, not the surface text. The oracle is *injected*: `ek` never owns an LLM or a sandbox, so a checker is a plugged-in strategy (a database-state comparison, a hidden-test run, a programmatic predicate, a judge). Aggregate = the success **rate**. - [`CostPerSuccessMetric`](_autosummary/ek.agents.metrics.html.md#ek.agents.metrics.CostPerSuccessMetric) – **Cost-of-Pass**: `E[cost] / P(success)`. Aggregate = `sum(dollars) / sum(successes)`, diverging to `inf` when nothing succeeded. A benchmark that scores accuracy *without* cost lets an agent chase tiny gains with unbounded API calls, so this metric is the point, not a nicety. **A safety property, not a nicety:** the checker must run somewhere the agent’s trajectory cannot reach. An audited agent-benchmark red-team produced a ~10-line `conftest.py` that pytest auto-loads and that rewrote *every* test outcome to “passed” – scoring near-perfect without solving a single task. Verifier isolation is a security boundary. ### Example ```pycon >>> from ek.agents.base import Cost, Episode >>> from ek.agents.cost import per_million >>> ep = Episode(task_id="t1", output="42", cost=Cost(input_tokens=1_000_000)) >>> TaskSuccessMetric()(ep, "42").value 1.0 >>> m = CostPerSuccessMetric(price=per_million(2.0, 6.0)) >>> round(m(ep, "42").value, 4) # this episode's dollars 2.0 ``` ### Functions | [`final_state_match`](_autosummary/ek.agents.metrics.html.md#ek.agents.metrics.final_state_match)(episode, gold) | Compare the episode's **final state** with the goal state (the tau-bench oracle). | |-------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------| | [`output_match`](_autosummary/ek.agents.metrics.html.md#ek.agents.metrics.output_match)(episode, gold) | Canonicalized exact match on the final answer (the GAIA-style oracle). | | [`recorded_success`](_autosummary/ek.agents.metrics.html.md#ek.agents.metrics.recorded_success)(episode, gold) | Trust a `success` already recorded on the episode (e.g. by an external harness). | ### Classes | [`Checker`](_autosummary/ek.agents.metrics.html.md#ek.agents.metrics.Checker)(\*args, \*\*kwargs) | The success oracle: did this episode leave the world in the required state? | |-----------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------| | [`CostPerSuccessMetric`](_autosummary/ek.agents.metrics.html.md#ek.agents.metrics.CostPerSuccessMetric)([check, price, prices]) | Cost-of-Pass: dollars per **successfully completed** task (**lower is better**). | | [`TaskSuccessMetric`](_autosummary/ek.agents.metrics.html.md#ek.agents.metrics.TaskSuccessMetric)([check]) | Did the episode complete the task? (`1.0`/`0.0`; aggregate = success rate). | ### *class* ek.agents.metrics.Checker(\*args, \*\*kwargs) Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol) The success oracle: did this episode leave the world in the required state? Injected, never owned by `ek` – a real suite plugs in a database-state comparison, a hidden-test runner, or a programmatic predicate. It must run **isolated** from the agent. ### *class* ek.agents.metrics.CostPerSuccessMetric(check=None, , price=None, prices=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Cost-of-Pass: dollars per **successfully completed** task (**lower is better**). The per-episode `Score.value` is that episode’s dollar cost – the real number is the corpus `aggregate`, `sum(dollars) / sum(successes)`, which is `inf` when nothing succeeded (infeasibility is the honest answer; a per-token metric reporting “$0.003/call” for an agent that never succeeds is actively misleading). * **Parameters:** * **check** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The success oracle (as in [`TaskSuccessMetric`](_autosummary/ek.agents.metrics.html.md#ek.agents.metrics.TaskSuccessMetric)). * **price** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`ModelPrice`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.ModelPrice)]) – A [`ModelPrice`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.ModelPrice) applied to every episode, or `None` to resolve each episode’s model against `prices`. * **prices** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`ModelPrice`](_autosummary/ek.agents.cost.html.md#ek.agents.cost.ModelPrice)]]) – A `{model: ModelPrice}` catalog (see [`ek.agents.load_prices()`](_autosummary/ek.agents.html.md#ek.agents.load_prices)). #### aggregate(scores) `sum(dollars) / sum(successes)` – a ratio of two sums, never a mean of ratios. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.agents.metrics.TaskSuccessMetric(check=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Did the episode complete the task? (`1.0`/`0.0`; aggregate = success rate). * **Parameters:** **check** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – A [`Checker`](_autosummary/ek.agents.metrics.html.md#ek.agents.metrics.Checker) – a registered name (`"output"`, `"final_state"`, `"recorded"`) or any `(episode, gold) -> bool` callable. Defaults to `"output"`; **inject a state-based or executable oracle for a real suite.** #### aggregate(scores) The success **rate** over the corpus. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.agents.metrics.final_state_match(episode, gold) Compare the episode’s **final state** with the goal state (the tau-bench oracle). * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.agents.metrics.output_match(episode, gold) Canonicalized exact match on the final answer (the GAIA-style oracle). The weakest useful oracle, and the default only because it needs nothing. Prefer a state-based or executable checker whenever the task has one. * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.agents.metrics.recorded_success(episode, gold) Trust a `success` already recorded on the episode (e.g. by an external harness). * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) # _autosummary/ek.agents.toolcalls.html.md # ek.agents.toolcalls Tool-call correctness: BFCL-style AST matching over a *multiset* of calls. The Berkeley Function Calling Leaderboard defines the reference algorithm: parse the emitted call and – **without executing it** – check that the function name matches, that every expected argument is present with the right value (case-insensitive, whitespace-folded string comparison), and that no argument outside the schema was hallucinated. Arguments are matched **by name**, so argument *order* is irrelevant; element order *within a list value* is not (`misc/docs/ek_08`, `misc/docs/ek_12`). **Why this cannot just reuse** [`FieldMetric`](_autosummary/ek.metrics.fields.html.md#ek.metrics.fields.FieldMetric). That metric compares two flat records *keyed by field name* – the key **is** the alignment. A trajectory’s tool calls are a **keyless multiset**: two `search(q=...)` calls collide onto one key, and reordering or repeating a call silently miscounts. So this module adds the missing layer – an **assignment** between predicted and gold calls – and then reuses `FieldMetric`’s exact TP/FP/FN accounting and micro-averaged aggregation on top of it. Cost-sensitivity is the reason to build this rather than borrow one: the per-argument weights come from the Layer-A grammar (`FieldSpec.importance`), so a wrong argument to a *destructive* tool is not one unit of error. No off-the-shelf tool-call metric models that. ### Example ```pycon >>> from ek.agents.base import Step >>> pred = [Step("search", {"q": "cats"}), Step("answer", {"a": "meow"})] >>> gold = [("search", {"q": "cats"}), ("answer", {"a": "meow"})] >>> ToolCallMetric()(pred, gold).f1 1.0 >>> hallucinated = [Step("search", {"q": "cats"}), Step("delete_all", {})] >>> round(ToolCallMetric()(hallucinated, gold).f1, 3) # 1 TP, 1 FP, 1 FN 0.5 ``` ### Functions | [`match_calls`](_autosummary/ek.agents.toolcalls.html.md#ek.agents.toolcalls.match_calls)(pred, gold, \*[, canon]) | Assign predicted calls to gold calls -- the layer `FieldMetric` does not provide. | |-----------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------| ### Classes | [`ToolCallMetric`](_autosummary/ek.agents.toolcalls.html.md#ek.agents.toolcalls.ToolCallMetric)([level, canonicalizer, grammar]) | Tool-call correctness (BFCL AST match) as a cost-weighted [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric). | |----------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------| ### *class* ek.agents.toolcalls.ToolCallMetric(level='call', , canonicalizer=None, grammar=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Tool-call correctness (BFCL AST match) as a cost-weighted [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric). * **Parameters:** * **level** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – `"call"` (default) scores whole calls – a call is a TP only on an exact AST match; `"arg"` gives partial credit at argument granularity. * **canonicalizer** – `str -> str` applied to string arguments before comparison (BFCL’s case/whitespace folding). Defaults to `ek`’s default canonicalizer. * **grammar** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar)]) – Layer-A grammar supplying per-tool and per-argument cost weights (may also be passed per call). The `Score` carries `tp`/`fp`/`fn` in `detail` so [`ek.evaluate()`](_autosummary/ek.html.md#ek.evaluate) aggregates a correct **micro**-averaged F1 over the corpus rather than averaging per-episode F1s. #### aggregate(scores) Micro-averaged F1 over the corpus (sum TP/FP/FN, then divide) – never a mean. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.agents.toolcalls.match_calls(pred, gold, , canon=None) Assign predicted calls to gold calls – the layer `FieldMetric` does not provide. Calls are grouped by tool name (a call to the *wrong tool* is never a match). Within a tool name, candidate pairs are scored and taken **globally best-first** (exact-argument matches before partial ones, more agreeing arguments before fewer), so identical repeated calls are interchangeable and a reordered pair is still matched – while a strong pairing can never be stolen by a weaker one that merely happened to come first in the prediction. This is a deterministic greedy approximation to the optimal assignment. It is exact for the common cases (distinct calls, repeats, reorderings); a full min-cost bipartite matching (Hungarian) could differ only on pathological many-to-many partial-overlap sets, and is not worth a dependency here. Returns `(pairs, unmatched_pred, unmatched_gold)`, `pairs` being `(pred_call, gold_call)`. * **Return type:** [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple) ### Example ```pycon >>> pairs, up, ug = match_calls([("s", {"q": "b"}), ("s", {"q": "a"})], ... [("s", {"q": "a"}), ("s", {"q": "b"})]) >>> len(pairs), up, ug # order-insensitive within a tool name (2, [], []) ``` # _autosummary/ek.agents.trajectory.html.md # ek.agents.trajectory Trajectory (process) evaluation: a cost-weighted **sequence** edit distance over steps. Outcome grading tells you *whether* the agent arrived; trajectory evaluation tells you *how*. This scores the ordered sequence of tool calls against a reference trajectory, with per-step and per-argument costs read from the Layer-A grammar – so taking a needless detour through a *destructive* tool costs more than a harmless one. **Why not reuse the flagship** [`TypedGraphMetric`](_autosummary/ek.metrics.graphs.html.md#ek.metrics.graphs.TypedGraphMetric)? It is tempting (a trajectory *is* a typed graph), but the graph-edit-distance engine is the wrong tool for a **linear** object, on four counts: 1. **It ignores order.** `networkx.graph_edit_distance` is an isomorphism search; node substitution matches on type + fields and ignores node identity. Chain edges add substitutable mass; they do not impose an ordering constraint. Two identical calls at different positions become interchangeable – which is exactly the error a trajectory metric exists to catch. 2. **It is capped.** `DEFAULT_MAX_NODES = 60` raises on a 100-step episode. 3. **It is nondeterministic.** The NP-hard search runs under a wall-clock `timeout` and returns a best-so-far bound – a wall-clock-dependent score for what is a *linear* alignment. 4. **Its denominator is polluted** by the synthetic ordering edges. So we reuse the **cost model** (the grammar’s importance weights) and not the **engine**: an O(n\*m) Needleman-Wunsch alignment, which is order-honoring, exact, deterministic and unbounded in length. (A genuine DAG mode – for trajectories carrying explicit step *dependencies* – would justify the graph engine; `Step` does not model dependency edges yet, so it is deliberately not offered rather than shipped as a mode that quietly ignores order.) The four schemes **disagree with each other** – pick one deliberately, exactly as with the partial-match schemes on the IE side (`misc/docs/ek_02`, `misc/docs/ek_10`). ### Example ```pycon >>> from ek.agents.base import Step >>> gold = [Step("search", {"q": "cat"}), Step("answer", {"a": "meow"})] >>> TrajectoryMetric()(gold, gold).value # identical -> distance 0 0.0 >>> detour = [Step("search", {"q": "cat"}), Step("search", {"q": "cat"}), ... Step("answer", {"a": "meow"})] >>> round(TrajectoryMetric()(detour, gold).value, 3) > 0 # a needless extra step costs True >>> TrajectoryMetric(scheme="superset")(detour, gold).value # but it IS a superset of gold 0.0 ``` ### Module Attributes | [`SCHEMES`](_autosummary/ek.agents.trajectory.html.md#ek.agents.trajectory.SCHEMES) | The trajectory-match schemes. | |------------------------------------------------------------|---------------------------------| ### Classes | [`TrajectoryMetric`](_autosummary/ek.agents.trajectory.html.md#ek.agents.trajectory.TrajectoryMetric)([scheme, grammar, ...]) | Cost-weighted trajectory distance as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric) (**lower is better**). | |---------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------| ### ek.agents.trajectory.SCHEMES *= ('in_order', 'exact', 'any_order', 'superset')* The trajectory-match schemes. They disagree; choose deliberately. ### *class* ek.agents.trajectory.TrajectoryMetric(scheme='in_order', , grammar=None, canonicalizer=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Cost-weighted trajectory distance as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric) (**lower is better**). * **Parameters:** * **scheme** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – One of [`SCHEMES`](_autosummary/ek.agents.trajectory.html.md#ek.agents.trajectory.SCHEMES). `in_order` (default) – Needleman-Wunsch edit distance; order matters, gaps allowed. `exact` – 0 iff the call sequences are identical, else the full mass. `any_order` – multiset distance; order ignored (the calls, not the path). `superset` – 0 iff every gold call occurs in the prediction; extra steps are free (use when the reference lists *required* calls rather than the whole path). * **grammar** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar)]) – Layer-A grammar supplying step/argument cost weights (may be passed per call). * **canonicalizer** – `str -> str` applied to string arguments before comparison. #### aggregate(scores) Corpus distance = total raw distance / total maximum distance (never a mean). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) # _autosummary/ek.base.html.md # ek.base Foundational data structures and strategy protocols for `ek`. This module is the single source of truth (SSOT) for `ek`’s *two-layer* data model and for the small set of `typing.Protocol` strategy interfaces that every swappable behaviour in the framework implements. It contains \*\*data structures and contracts only\*\* – no business logic, no heavy imports. Importing it is cheap and pulls in nothing beyond the standard library. ## The two layers Evaluating an information-extraction (IE) output splits into two halves – scoring against a gold reference (*offline*) and estimating quality with no reference (*online*) – and both operate on the **same** object: - **Layer A – the schema** ([`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar) and friends): a *frozen* description of the typed graph you extract, carrying both the **types** and the **importance/cost weights** per field/node/edge. It is the SSOT consulted by cost-sensitive metrics, by validators, and (eventually) by constrained decoders. - **Layer B – the extraction metadata** ([`AnnotatedExtraction`](_autosummary/ek.base.html.md#ek.base.AnnotatedExtraction) and friends): per-extraction runtime metadata – provenance, raw signals, (calibrated) confidence, validator findings, and the accept/flag/block decision – keyed by a [`NodePath`](_autosummary/ek.base.html.md#ek.base.NodePath) into the Layer-A schema, riding *alongside* the grammar without ever mutating it. ## The strategy protocols [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric), [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator), [`Calibrator`](_autosummary/ek.base.html.md#ek.base.Calibrator), [`DecisionPolicy`](_autosummary/ek.base.html.md#ek.base.DecisionPolicy) and [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal) are structural (duck-typed) contracts. Concrete implementations are resolved from [`ek.registry`](_autosummary/ek.registry.html.md#module-ek.registry) by name and injected as keyword-only arguments with smart defaults, so the simple path Just Works while every layer stays replaceable (open-closed). ### Example ```pycon >>> g = GraphGrammar( ... node_types={"donation": NodeType( ... "donation", ... fields={ ... "amount": FieldSpec("amount", "number", importance=10.0), ... "city": FieldSpec("city", "string", importance=1.0), ... }, ... )}, ... ) >>> g.field_cost("donation", "amount") 10.0 >>> g.field_cost("donation", "city") 1.0 ``` ### Module Attributes | [`Normalizer`](_autosummary/ek.base.html.md#ek.base.Normalizer) | Canonicalize a string before comparison (lowercase, fold, normalize, ...). | |-------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------| | [`ConfidenceSource`](_autosummary/ek.base.html.md#ek.base.ConfidenceSource) | Read named raw signals off a [`FieldEstimate`](_autosummary/ek.base.html.md#ek.base.FieldEstimate) (e.g. for calibration). | | [`CostWeight`](_autosummary/ek.base.html.md#ek.base.CostWeight) | Cost of an edit on a typed element; defaults to reading `*.importance`. | | [`OcrBackend`](_autosummary/ek.base.html.md#ek.base.OcrBackend) | Anything that turns an image (path/bytes/array) into an `OcrResult`-shaped object (with `.text` and `.blocks`). | ### Functions | [`default_cost_weight`](_autosummary/ek.base.html.md#ek.base.default_cost_weight)(grammar, ref) | The default [`CostWeight`](_autosummary/ek.base.html.md#ek.base.CostWeight): read the importance weight off the schema. | |--------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------| ### Classes | [`AnnotatedExtraction`](_autosummary/ek.base.html.md#ek.base.AnnotatedExtraction)(grammar[, estimates]) | Layer B: the verification metadata, referencing a frozen Layer-A grammar. | |---------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------| | [`Calibrator`](_autosummary/ek.base.html.md#ek.base.Calibrator)(\*args, \*\*kwargs) | Maps a raw scalar signal to a probability after being `fit` on labels. | | [`Decision`](_autosummary/ek.base.html.md#ek.base.Decision)(\*values) | Terminal action a selective-prediction gate emits for an extracted value. | | [`DecisionPolicy`](_autosummary/ek.base.html.md#ek.base.DecisionPolicy)(\*args, \*\*kwargs) | Turns a (calibrated) confidence into an accept/flag/block [`Decision`](_autosummary/ek.base.html.md#ek.base.Decision). | | [`EdgeType`](_autosummary/ek.base.html.md#ek.base.EdgeType)(name, src, dst[, importance]) | A directed relation kind between two [`NodeType`](_autosummary/ek.base.html.md#ek.base.NodeType) names. | | [`FieldEstimate`](_autosummary/ek.base.html.md#ek.base.FieldEstimate)(value[, raw_signals, ...]) | One extracted value plus all of its reference-free verification metadata. | | [`FieldSpec`](_autosummary/ek.base.html.md#ek.base.FieldSpec)(name[, type, importance, domain, ...]) | A leaf attribute of a node: its type, validation domain and cost weight. | | [`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)(field, layer[, severity, message, ...]) | One validator observation about a value: a flag, optionally a correction. | | [`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar)([node_types, edge_types]) | The frozen schema SSOT: typed nodes/edges plus their importance weights. | | [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric)(\*args, \*\*kwargs) | Compares one prediction to one gold reference, returning a [`Score`](_autosummary/ek.base.html.md#ek.base.Score). | | [`NodePath`](_autosummary/ek.base.html.md#ek.base.NodePath)(node_id, node_type[, field]) | Addresses a node (or one of its fields) in an extracted graph; keys Layer B. | | [`NodeType`](_autosummary/ek.base.html.md#ek.base.NodeType)(name[, fields, importance]) | A node kind in the typed graph (e.g. an "invoice" or a "line item"). | | [`Provenance`](_autosummary/ek.base.html.md#ek.base.Provenance)([engine, source_span, bbox, ...]) | Where an extracted value came from, for audit and click-to-source review. | | [`QualityReport`](_autosummary/ek.base.html.md#ek.base.QualityReport)([calibrated_confidence, ...]) | The result of a reference-free quality estimate for one extraction. | | [`Report`](_autosummary/ek.base.html.md#ek.base.Report)([metric, aggregate, n, scores, ...]) | Aggregate of many [`Score`](_autosummary/ek.base.html.md#ek.base.Score) s over a corpus, with optional per-slice cuts. | | [`Score`](_autosummary/ek.base.html.md#ek.base.Score)(value[, precision, recall, f1, ...]) | The result of comparing one prediction against one reference. | | [`Severity`](_autosummary/ek.base.html.md#ek.base.Severity)(\*values) | Whether a [`Finding`](_autosummary/ek.base.html.md#ek.base.Finding) merely flags or can also correct a value. | | [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal)(\*args, \*\*kwargs) | A reference-free quality signal: extractor output -> raw scalar score. | | [`TypeRef`](_autosummary/ek.base.html.md#ek.base.TypeRef)(kind, name[, field]) | A reference into the schema, so an injected cost function can read weights. | | [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator)(\*args, \*\*kwargs) | Reference-free check on a value, yielding zero or more [`Finding`](_autosummary/ek.base.html.md#ek.base.Finding) s. | ### *class* ek.base.AnnotatedExtraction(grammar, estimates=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Layer B: the verification metadata, referencing a frozen Layer-A grammar. The grammar is held *by reference* and never mutated; `estimates` maps a [`NodePath`](_autosummary/ek.base.html.md#ek.base.NodePath) to the [`FieldEstimate`](_autosummary/ek.base.html.md#ek.base.FieldEstimate) for that node/field. ### *class* ek.base.Calibrator(\*args, \*\*kwargs) Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol) Maps a raw scalar signal to a probability after being `fit` on labels. ### ek.base.ConfidenceSource Read named raw signals off a [`FieldEstimate`](_autosummary/ek.base.html.md#ek.base.FieldEstimate) (e.g. for calibration). alias of `Callable`[[[`FieldEstimate`](_autosummary/ek.base.html.md#ek.base.FieldEstimate)], [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`float`](https://docs.python.org/3/builtins/functions.html#float)]] ### ek.base.CostWeight Cost of an edit on a typed element; defaults to reading `*.importance`. alias of `Callable`[[[`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar), [`TypeRef`](_autosummary/ek.base.html.md#ek.base.TypeRef)], [`float`](https://docs.python.org/3/builtins/functions.html#float)] ### *class* ek.base.Decision(\*values) Bases: [`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Enum`](https://docs.python.org/3/library/enum.html#enum.Enum) Terminal action a selective-prediction gate emits for an extracted value. ### *class* ek.base.DecisionPolicy(\*args, \*\*kwargs) Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol) Turns a (calibrated) confidence into an accept/flag/block [`Decision`](_autosummary/ek.base.html.md#ek.base.Decision). ### *class* ek.base.EdgeType(name, src, dst, importance=1.0) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A directed relation kind between two [`NodeType`](_autosummary/ek.base.html.md#ek.base.NodeType) names. ### *class* ek.base.FieldEstimate(value, raw_signals=, confidence=None, findings=(), provenance=None, decision=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One extracted value plus all of its reference-free verification metadata. ### *class* ek.base.FieldSpec(name, type='string', importance=1.0, domain=(), normalizer=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A leaf attribute of a node: its type, validation domain and cost weight. * **Parameters:** * **name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Attribute name. * **type** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – A free-form type tag (e.g. `"string"`, `"number"`, `"date"`); `ek` does not impose a type system – the tag drives metric/validator dispatch and is yours to define. * **importance** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Per-field cost weight; the lever that makes “two extra digits on a donation amount” outweigh “a misspelled city”. Defaults to `1.0`. * **domain** ([`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)) – Optional allowed values: an enum tuple, a `(lo, hi)` range, or a regex string – consumed by validators, never by metrics. * **normalizer** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – Optional key naming a canonicalizer (resolved from the registry) to apply to this field before comparison. ### *class* ek.base.Finding(field, layer, severity=Severity.FLAG, message='', suggestion=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One validator observation about a value: a flag, optionally a correction. A `suggestion` is present iff the producing layer can *correct* (only the deterministic canonicalizer and the gated LLM corrector freely do so). The original value is always retained elsewhere; a Finding never mutates. ### *class* ek.base.GraphGrammar(node_types=, edge_types=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The frozen schema SSOT: typed nodes/edges plus their importance weights. Methods return the cost weight for a referenced type, defaulting to `1.0` for anything not declared, so a partial grammar is always usable. #### cost(ref) Importance weight for any [`TypeRef`](_autosummary/ek.base.html.md#ek.base.TypeRef) (the default `CostWeight`). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) #### edge_cost(name) Importance weight of an edge type (`1.0` if undeclared). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) #### field_cost(node, field_name) Importance weight of a field on a node type (`1.0` if undeclared). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) #### node_cost(name) Importance weight of a node type (`1.0` if undeclared). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.base.Metric(\*args, \*\*kwargs) Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol) Compares one prediction to one gold reference, returning a [`Score`](_autosummary/ek.base.html.md#ek.base.Score). ### *class* ek.base.NodePath(node_id, node_type, field=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Addresses a node (or one of its fields) in an extracted graph; keys Layer B. ### *class* ek.base.NodeType(name, fields=, importance=1.0) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A node kind in the typed graph (e.g. an “invoice” or a “line item”). ### ek.base.Normalizer Canonicalize a string before comparison (lowercase, fold, normalize, …). alias of `Callable`[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] ### ek.base.OcrBackend Anything that turns an image (path/bytes/array) into an `OcrResult`-shaped object (with `.text` and `.blocks`). Typed loosely so `ek` core never imports an OCR package. alias of `Callable`[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] ### *class* ek.base.Provenance(engine='', source_span=None, bbox=None, raw_transcripts=()) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Where an extracted value came from, for audit and click-to-source review. `bbox` is intentionally typed `Any` so `ek` core need not import any OCR package; the OCR instance stores normalized `[0, 1]` top-left-origin geometry. ### *class* ek.base.QualityReport(calibrated_confidence=None, decision=None, findings=(), raw_signals=, provenance=, per_field=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The result of a reference-free quality estimate for one extraction. For a single value the scalar fields carry the verdict. For a whole [`AnnotatedExtraction`](_autosummary/ek.base.html.md#ek.base.AnnotatedExtraction), `per_field` maps each [`NodePath`](_autosummary/ek.base.html.md#ek.base.NodePath) to its updated [`FieldEstimate`](_autosummary/ek.base.html.md#ek.base.FieldEstimate) (confidence/findings/decision filled in), and the scalar fields summarize: `calibrated_confidence` is the weakest field, and `decision` the most severe (`block` > `flag` > `accept`). ### *class* ek.base.Report(metric='', aggregate=None, n=0, scores=, per_slice=, detail=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Aggregate of many [`Score`](_autosummary/ek.base.html.md#ek.base.Score) s over a corpus, with optional per-slice cuts. `aggregate` is the corpus-level headline (computed by the metric’s own aggregator – e.g. globally accumulated WER – not a naive mean of per-item values). `per_slice` maps a slice label to its own aggregate. ### *class* ek.base.Score(value, precision=None, recall=None, f1=None, metric='', detail=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The result of comparing one prediction against one reference. `value` is the headline number (higher is better unless a metric documents otherwise). The optional decomposition fields let structured metrics carry precision/recall/F1, an alignment, and raw counts (e.g. edit operations) so that corpus aggregation can be done *correctly* (e.g. global WER accumulation) rather than by averaging per-item scores. `float(score)` returns `value`. ### *class* ek.base.Severity(\*values) Bases: [`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Enum`](https://docs.python.org/3/library/enum.html#enum.Enum) Whether a [`Finding`](_autosummary/ek.base.html.md#ek.base.Finding) merely flags or can also correct a value. ### *class* ek.base.Signal(\*args, \*\*kwargs) Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol) A reference-free quality signal: extractor output -> raw scalar score. The `cost_tier` attribute (lower is cheaper) lets an escalation policy run cheap signals first and pay for expensive ones only on residual uncertainty. ### *class* ek.base.TypeRef(kind, name, field=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A reference into the schema, so an injected cost function can read weights. `kind` is one of `"node"`, `"edge"` or `"field"`; for a field, `name` is the node-type name and `field` the field name. ### *class* ek.base.Validator(\*args, \*\*kwargs) Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol) Reference-free check on a value, yielding zero or more [`Finding`](_autosummary/ek.base.html.md#ek.base.Finding) s. ### ek.base.default_cost_weight(grammar, ref) The default [`CostWeight`](_autosummary/ek.base.html.md#ek.base.CostWeight): read the importance weight off the schema. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) # _autosummary/ek.canonicalize.html.md # ek.canonicalize Versioned, composable canonicalization applied *before* scoring. Under-specified normalization is the single biggest source of bogus evaluation scores – it can swing results by tens of points, and a reused English normalizer catastrophically corrupts Indic/abugida scripts by stripping Unicode Mark-class characters. So canonicalization is a first-class, *versioned* component here, not a metric detail: a [`Canonicalizer`](_autosummary/ek.canonicalize.html.md#ek.canonicalize.Canonicalizer) is an ordered pipeline of small named steps with a stable `version` you can persist alongside a gold set. Steps are plain `str -> str` functions registered under the `"normalizers"` namespace, so a schema can name a per-field normalizer (via `normalizer`) and third parties can add their own. ### Example ```pycon >>> canon = Canonicalizer(["nfc", "lower", "collapse_whitespace"]) >>> canon(" Héllo WORLD ") 'héllo world' >>> canon.version # stable hash of the step names 'nfc+lower+collapse_whitespace' ``` Always audit before folding a non-Latin script – [`mark_class_count()`](_autosummary/ek.canonicalize.html.md#ek.canonicalize.mark_class_count) reports how many combining marks a string carries, the signal that aggressive folding would destroy meaning: ```pycon >>> mark_class_count("e\u0301") # 'e' + combining acute 1 ``` ### Functions | [`collapse_whitespace`](_autosummary/ek.canonicalize.html.md#ek.canonicalize.collapse_whitespace)(s) | Collapse runs of whitespace to a single space and strip the ends. | |-----------------------------------------------------------------------------------|-----------------------------------------------------------------------------| | [`default_canonicalizer`](_autosummary/ek.canonicalize.html.md#ek.canonicalize.default_canonicalizer)() | A conservative default: NFC -> casefold -> collapse whitespace. | | [`lower`](_autosummary/ek.canonicalize.html.md#ek.canonicalize.lower)(s) | Casefold to lowercase (full Unicode casefold). | | [`mark_class_count`](_autosummary/ek.canonicalize.html.md#ek.canonicalize.mark_class_count)(s) | Number of Unicode Mark-class (combining) characters in `s`. | | [`nfc`](_autosummary/ek.canonicalize.html.md#ek.canonicalize.nfc)(s) | Unicode NFC normalization (compose); the safe default. | | [`nfkc`](_autosummary/ek.canonicalize.html.md#ek.canonicalize.nfkc)(s) | Unicode NFKC normalization (compatibility compose); lossy -- use with care. | | [`resolve_canonicalizer`](_autosummary/ek.canonicalize.html.md#ek.canonicalize.resolve_canonicalizer)(normalize) | Coerce a `normalize` argument to a callable canonicalizer (or `None`). | | [`strip`](_autosummary/ek.canonicalize.html.md#ek.canonicalize.strip)(s) | Strip leading/trailing whitespace. | | [`strip_punctuation`](_autosummary/ek.canonicalize.html.md#ek.canonicalize.strip_punctuation)(s) | Remove Unicode punctuation characters (category starting with `P`). | ### Classes | [`Canonicalizer`](_autosummary/ek.canonicalize.html.md#ek.canonicalize.Canonicalizer)([steps, version, audit_marks]) | An ordered, named, versioned pipeline of `str -> str` steps. | |-------------------------------------------------------------------------------------------------|----------------------------------------------------------------| ### *class* ek.canonicalize.Canonicalizer(steps=(), , version='', audit_marks=False) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) An ordered, named, versioned pipeline of `str -> str` steps. * **Parameters:** * **steps** ([`Sequence`](https://docs.python.org/3/library/typing.html#typing.Sequence)[`Union`[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]]]) – Step references – names registered under `"normalizers"` or plain callables. Applied left to right. * **version** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Override the auto-derived version string (the `+`-joined names). * **audit_marks** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – If true, warn when a step would run on text carrying combining marks (the cross-script damage guard). #### *property* steps *: [list](https://docs.python.org/3/builtins/stdtypes.html#list)* The resolved step names, in order. #### *property* version *: [str](https://docs.python.org/3/builtins/stdtypes.html#str)* Stable identifier of this pipeline (persist it next to a gold set). ### ek.canonicalize.collapse_whitespace(s) Collapse runs of whitespace to a single space and strip the ends. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### ek.canonicalize.default_canonicalizer() A conservative default: NFC -> casefold -> collapse whitespace. * **Return type:** [`Canonicalizer`](_autosummary/ek.canonicalize.html.md#ek.canonicalize.Canonicalizer) ### ek.canonicalize.lower(s) Casefold to lowercase (full Unicode casefold). * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### ek.canonicalize.mark_class_count(s) Number of Unicode Mark-class (combining) characters in `s`. A non-zero count on a non-Latin script is a warning sign that case/diacritic folding will corrupt meaning (the Indic-script catastrophe). Use it to gate or audit a canonicalizer before trusting cross-script scores. * **Return type:** [`int`](https://docs.python.org/3/builtins/functions.html#int) ### ek.canonicalize.nfc(s) Unicode NFC normalization (compose); the safe default. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### ek.canonicalize.nfkc(s) Unicode NFKC normalization (compatibility compose); lossy – use with care. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### ek.canonicalize.resolve_canonicalizer(normalize) Coerce a `normalize` argument to a callable canonicalizer (or `None`). Accepts `None`, a registered normalizer name, a callable, a sequence of steps, or an existing [`Canonicalizer`](_autosummary/ek.canonicalize.html.md#ek.canonicalize.Canonicalizer). * **Return type:** `Union`[[`Canonicalizer`](_autosummary/ek.canonicalize.html.md#ek.canonicalize.Canonicalizer), [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`None`](https://docs.python.org/3/builtins/constants.html#None)] ### ek.canonicalize.strip(s) Strip leading/trailing whitespace. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### ek.canonicalize.strip_punctuation(s) Remove Unicode punctuation characters (category starting with `P`). * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) # _autosummary/ek.facade.html.md # ek.facade The two top-level facades: [`score()`](_autosummary/ek.facade.html.md#ek.facade.score) (offline) and [`estimate_quality()`](_autosummary/ek.facade.html.md#ek.facade.estimate_quality) (online). Both operate on the same Layer-A object and follow progressive disclosure – the simple call Just Works, every strategy is replaceable by keyword: - [`score()`](_autosummary/ek.facade.html.md#ek.facade.score) – reference-based: compare one prediction to one gold reference, the metric chosen by output type (string -> CER, record -> field-F1) unless you name or pass one. - [`evaluate()`](_autosummary/ek.facade.html.md#ek.facade.evaluate) – reference-based at corpus scale: aggregate many comparisons *correctly* (global error-rate accumulation, micro-F1), with optional per-slice cuts. This is what the OCR benchmark and the regression harness build on. - [`estimate_quality()`](_autosummary/ek.facade.html.md#ek.facade.estimate_quality) – reference-free: gather signals -> calibrate -> validate -> decide, returning a [`QualityReport`](_autosummary/ek.base.html.md#ek.base.QualityReport). The heavy signal families (ROVER, conformal gates) are injected; this facade composes whatever you give it and ships sensible no-op defaults. ### Example ```pycon >>> score("hello wrld", "hello world").metric 'cer' >>> round(score("hello wrld", "hello world", metric="wer").value, 3) 0.5 >>> r = evaluate([("ct", "cat"), ("dg", "dog")], metric="cer") >>> r.n, round(r.aggregate, 3) (2, 0.333) ``` ### Functions | [`estimate_quality`](_autosummary/ek.facade.html.md#ek.facade.estimate_quality)(extraction, \*[, sources, ...]) | Estimate the quality of an extraction with no gold reference. | |-----------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------| | [`evaluate`](_autosummary/ek.facade.html.md#ek.facade.evaluate)(cases, \*[, metric, grammar, ...]) | Aggregate many comparisons into a [`Report`](_autosummary/ek.base.html.md#ek.base.Report) (corpus level). | | [`score`](_autosummary/ek.facade.html.md#ek.facade.score)(pred, gold, \*[, grammar, metric, ...]) | Score one prediction against one gold reference (reference-based). | ### ek.facade.estimate_quality(extraction, , sources=(), signals=(), calibrator=None, validators=(), policy=None, agreement=True, assume_calibrated=False) Estimate the quality of an extraction with no gold reference. Composes the strict `signal -> calibrate -> validate -> decide` pipeline (`misc/docs/ek_03`) over a single value, a [`FieldEstimate`](_autosummary/ek.base.html.md#ek.base.FieldEstimate), or a whole [`AnnotatedExtraction`](_autosummary/ek.base.html.md#ek.base.AnnotatedExtraction) (scored per field, with per-field specs feeding the validators and the node type as the Mondrian group key). The simple call `estimate_quality(value)` Just Works; every stage is injectable. * **Parameters:** * **extraction** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – A raw value, a [`FieldEstimate`](_autosummary/ek.base.html.md#ek.base.FieldEstimate), or an [`AnnotatedExtraction`](_autosummary/ek.base.html.md#ek.base.AnnotatedExtraction). * **sources** ([`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)) – Additional hypotheses of the *same* content (strings or `OcrResult`-shaped objects) to fuse with ROVER – their mean agreement becomes a `raw_signals["agreement"]` entry (the flagship online signal). * **signals** ([`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)) – Explicit [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal) callables `target -> float | Mapping` producing further raw signals. * **calibrator** – A [`Calibrator`](_autosummary/ek.base.html.md#ek.base.Calibrator) mapping raw score -> probability, applied to the field’s confidence (else the mean of its raw signals). When the signals are *heterogeneous* (e.g. an LLM-judge score next to a logprob), pass a `Mapping[str, Calibrator]` keyed by signal name (a `"*"` entry is the fallback) or a [`SignalFamilyCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.SignalFamilyCalibrator): each signal family is then calibrated on its own and the calibrated probabilities are fused by its `combine` rule (mean by default) followed by its learned `fusion` calibrator. A field’s own `confidence` is the `"confidence"` family. The per-family values are reported in `provenance["calibrated_signals"]`. **Calibration is non-optional before gating** (Hard Rule 1): passing a `policy` with no calibrator and `assume_calibrated` false **raises** – a raw, uncalibrated score must never reach a DecisionPolicy. * **validators** ([`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)) – [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) callables yielding findings. A flat iterable runs on every field (use spec-driven validators like [`schema_validator()`](_autosummary/ek.qe.verifiers.html.md#ek.qe.verifiers.schema_validator)). For per-field scoping pass a `Mapping` keyed by field name or node type (`"*"` runs on all). * **policy** – A [`DecisionPolicy`](_autosummary/ek.base.html.md#ek.base.DecisionPolicy) producing accept/flag/block from the *calibrated* confidence. * **agreement** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Auto-run ROVER over `sources` when any are given (default true). * **assume_calibrated** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Treat the incoming confidence as already calibrated (silences the uncalibrated-gating warning). With a per-family calibrator it also waives the check that a value fused from two or more families went through a learned `fusion` calibrator. * **Return type:** [`QualityReport`](_autosummary/ek.base.html.md#ek.base.QualityReport) ### ek.facade.evaluate(cases, , metric=None, grammar=None, normalize=None, weights=None) Aggregate many comparisons into a [`Report`](_autosummary/ek.base.html.md#ek.base.Report) (corpus level). * **Parameters:** * **cases** ([`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)) – Iterable of `(pred, gold)` or `(pred, gold, slice_label)` tuples. * **metric** (`Union`[[`None`](https://docs.python.org/3/builtins/constants.html#None), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric)]) – As in [`score()`](_autosummary/ek.facade.html.md#ek.facade.score) (resolved once from the first case’s types). * **grammar** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar)]) – Optional Layer-A grammar passed to every comparison. * **normalize** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Optional canonicalizer applied before every comparison. * **Return type:** [`Report`](_autosummary/ek.base.html.md#ek.base.Report) * **Returns:** A Report whose `aggregate` is computed by the metric’s own aggregator (e.g. globally accumulated CER/WER, micro-F1) – never a naive mean. ### ek.facade.score(pred, gold, , grammar=None, metric=None, normalize=None, weights=None) Score one prediction against one gold reference (reference-based). * **Parameters:** * **pred** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The predicted output (string, record dict, or anything with `.text`). * **gold** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The gold reference, same shape as `pred`. * **grammar** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar)]) – Optional Layer-A [`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar) (carries cost weights). * **metric** (`Union`[[`None`](https://docs.python.org/3/builtins/constants.html#None), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric)]) – A registered metric name (`"cer"`, `"wer"`, `"fields"`, …), a callable [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric), or `None` to dispatch by type. * **normalize** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Optional canonicalizer (name, callable, step list, or [`Canonicalizer`](_autosummary/ek.canonicalize.html.md#ek.canonicalize.Canonicalizer)) applied before comparison. * **weights** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – A [`CostWeight`](_autosummary/ek.base.html.md#ek.base.CostWeight) for cost-weighted metrics (the typed-graph distance); overrides the schema’s importance weights. * **Return type:** [`Score`](_autosummary/ek.base.html.md#ek.base.Score) * **Returns:** A [`Score`](_autosummary/ek.base.html.md#ek.base.Score). # _autosummary/ek.harness.html.md # ek.harness The offline evaluation harness: run a predictor over gold, gate regressions. Reference-based benchmarking (Need #1) is only useful if it is *repeatable* and *regression-safe*: a “better” system must not silently get worse on some slice. The harness adds that discipline on top of [`ek.score()`](_autosummary/ek.html.md#ek.score)/[`ek.evaluate()`](_autosummary/ek.html.md#ek.evaluate): - [`evaluate_store()`](_autosummary/ek.harness.html.md#ek.harness.evaluate_store) – run any `input -> prediction` predictor over a gold store (a dict or a `ek` gold store), scored per slice (grouped by each record’s slice label), with results persisted. (The OCR benchmark in [`ek.ocr`](_autosummary/ek.ocr.html.md#module-ek.ocr) is the OCR-specific specialization of this.) - [`save_baseline()`](_autosummary/ek.harness.html.md#ek.harness.save_baseline) / [`regression_gate()`](_autosummary/ek.harness.html.md#ek.harness.regression_gate) – freeze a baseline and fail when a later run regresses beyond a tolerance, **per slice**, not just on the aggregate. This is the golden-set CI gate. - [`cohen_kappa()`](_autosummary/ek.harness.html.md#ek.harness.cohen_kappa) / [`percent_agreement()`](_autosummary/ek.harness.html.md#ek.harness.percent_agreement) / [`krippendorff_alpha()`](_autosummary/ek.harness.html.md#ek.harness.krippendorff_alpha) – inter-annotator agreement (IAA), to know the ceiling of your gold standard before trusting any model score (all pure-Python, no optional dependency). See `misc/docs/ek_02` for the harness design (canonicalizer versioning, slicing, IAA conventions, golden-set CI). ### Functions | [`cohen_kappa`](_autosummary/ek.harness.html.md#ek.harness.cohen_kappa)(rater_a, rater_b) | Cohen's kappa for two raters on nominal labels (chance-corrected agreement). | |----------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------| | [`evaluate_store`](_autosummary/ek.harness.html.md#ek.harness.evaluate_store)(predict, gold, \*[, metric, ...]) | Run `predict` over a gold store and score it per slice. | | [`krippendorff_alpha`](_autosummary/ek.harness.html.md#ek.harness.krippendorff_alpha)(reliability_data, \*[, level]) | Krippendorff's alpha -- the general inter-annotator agreement coefficient. | | [`load_baseline`](_autosummary/ek.harness.html.md#ek.harness.load_baseline)(name, \*[, rootdir]) | Load a named baseline (or `None` if it does not exist). | | [`percent_agreement`](_autosummary/ek.harness.html.md#ek.harness.percent_agreement)(rater_a, rater_b) | Raw fraction of items two raters labelled identically. | | [`regression_gate`](_autosummary/ek.harness.html.md#ek.harness.regression_gate)(report, baseline, \*[, ...]) | Fail if `report` regresses beyond `tolerance` vs a baseline, per slice. | | [`save_baseline`](_autosummary/ek.harness.html.md#ek.harness.save_baseline)(report, name, \*[, rootdir]) | Freeze a report's aggregate + per-slice scores as a named baseline. | ### Classes | [`GateResult`](_autosummary/ek.harness.html.md#ek.harness.GateResult)(passed, metric, higher_is_better, ...) | Outcome of a [`regression_gate()`](_autosummary/ek.harness.html.md#ek.harness.regression_gate) check. | |------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------| ### *class* ek.harness.GateResult(passed, metric, higher_is_better, tolerance, aggregate_current=None, aggregate_baseline=None, regressions=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Outcome of a [`regression_gate()`](_autosummary/ek.harness.html.md#ek.harness.regression_gate) check. ### ek.harness.cohen_kappa(rater_a, rater_b) Cohen’s kappa for two raters on nominal labels (chance-corrected agreement). Use only for exactly two raters, nominal labels, complete data; prefer [`krippendorff_alpha()`](_autosummary/ek.harness.html.md#ek.harness.krippendorff_alpha) for the general case (missing data, any measurement level). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.harness.evaluate_store(predict, gold, , metric=None, grammar=None, normalize=None, weights=None, input_key='input', reference_key='reference', slice_key='slice', persist=False, run_id=None, rootdir=None) Run `predict` over a gold store and score it per slice. * **Parameters:** * **predict** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – `input -> prediction` callable (the system under test). * **gold** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)) – Mapping `key -> {input, reference, [slice], ...}` (a dict or a `ek` gold store). * **metric** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – Metric name/callable (defaults to type-dispatch on the first case). * **grammar/normalize/weights** – forwarded to scoring. * **input_key/reference_key/slice_key** – field names within each gold record. * **persist/run_id/rootdir** – persist the run to the `results`/`runs` stores. * **Returns:** A [`Report`](_autosummary/ek.base.html.md#ek.base.Report); `detail['per_item']` maps each key to its prediction, reference, slice, and score. ### ek.harness.krippendorff_alpha(reliability_data, , level='nominal') Krippendorff’s alpha – the general inter-annotator agreement coefficient. Pure-Python (no dependency, permissive core): any number of raters, any measurement `level` (`"nominal"`, `"ordinal"`, `"interval"`, `"ratio"`), and missing data (use `None` or `float('nan')` for an unlabelled cell). `reliability_data` is one row per rater, one column per item. Alpha is `1.0` for perfect agreement, `0.0` at chance, and goes negative for systematic disagreement. Computed via the coincidence-matrix method (Krippendorff 2011): observed vs expected disagreement under the level-specific difference metric. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### Example ```pycon >>> data = [[1, 1, 1, 1], [1, 1, 1, 1]] # two raters, perfect agreement >>> krippendorff_alpha(data) 1.0 ``` ### ek.harness.load_baseline(name, , rootdir=None) Load a named baseline (or `None` if it does not exist). * **Return type:** [`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)] ### ek.harness.percent_agreement(rater_a, rater_b) Raw fraction of items two raters labelled identically. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.harness.regression_gate(report, baseline, , tolerance=0.0, higher_is_better=None, rootdir=None) Fail if `report` regresses beyond `tolerance` vs a baseline, per slice. * **Parameters:** * **report** – The current [`Report`](_autosummary/ek.base.html.md#ek.base.Report). * **baseline** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – A baseline name (loaded from the `baselines` store) or a baseline dict from [`save_baseline()`](_autosummary/ek.harness.html.md#ek.harness.save_baseline). * **tolerance** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Allowed drift before a change counts as a regression. * **higher_is_better** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`bool`](https://docs.python.org/3/builtins/functions.html#bool)]) – Override metric-direction inference (CER/WER/graph are lower-is-better; F1/similarity are higher-is-better). * **rootdir** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – data root (when `baseline` is a name). * **Return type:** [`GateResult`](_autosummary/ek.harness.html.md#ek.harness.GateResult) * **Returns:** A [`GateResult`](_autosummary/ek.harness.html.md#ek.harness.GateResult) (falsy if any regression was found). ### ek.harness.save_baseline(report, name, , rootdir=None) Freeze a report’s aggregate + per-slice scores as a named baseline. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) # _autosummary/ek.html.md # ek `ek` – a framework for building Knowledge Evaluation systems. `ek` evaluates the outputs of information-extraction systems. OCR is treated as the noisiest *special case* of a general problem, so the core is source-agnostic (PDF, DOCX, tables, DB responses, LLM extractors) and the OCR specifics live in [`ek.ocr`](_autosummary/ek.ocr.html.md#module-ek.ocr). Two facades cover the two halves of evaluation, both operating on the same typed schema: - [`score()`](_autosummary/ek.html.md#ek.score) / [`evaluate()`](_autosummary/ek.html.md#ek.evaluate) – *reference-based* (offline): compare against a gold answer, one item or a whole corpus, the metric chosen by output type. - [`estimate_quality()`](_autosummary/ek.html.md#ek.estimate_quality) – *reference-free* (online): gather signals, calibrate, validate, and decide accept/flag/block with no gold answer. Everything swappable is a `typing.Protocol` strategy resolved from [`ek.registry`](_autosummary/ek.registry.html.md#module-ek.registry) and injected as a keyword-only argument with a smart default – so the simple path Just Works and every layer stays replaceable (open-closed). Quickstart: ```pycon >>> import ek >>> round(ek.score("hello wrld", "hello world").value, 3) # CER by default 0.091 >>> ek.evaluate([("ct", "cat"), ("dg", "dog")], metric="cer").n 2 ``` The two-layer data model ([`GraphGrammar`](_autosummary/ek.html.md#ek.GraphGrammar) for the schema + cost weights, [`AnnotatedExtraction`](_autosummary/ek.html.md#ek.AnnotatedExtraction) for per-extraction verification metadata) is the SSOT every component plugs into – see [`ek.base`](_autosummary/ek.base.html.md#module-ek.base). Persistence is local-file `dol` stores under `~/.local/share/ek/` – see [`ek.stores`](_autosummary/ek.stores.html.md#module-ek.stores). ### Functions | [`score`](_autosummary/ek.html.md#ek.score)(pred, gold, \*[, grammar, metric, ...]) | Score one prediction against one gold reference (reference-based). | |-----------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------| | [`evaluate`](_autosummary/ek.html.md#ek.evaluate)(cases, \*[, metric, grammar, ...]) | Aggregate many comparisons into a [`Report`](_autosummary/ek.base.html.md#ek.base.Report) (corpus level). | | [`estimate_quality`](_autosummary/ek.html.md#ek.estimate_quality)(extraction, \*[, sources, ...]) | Estimate the quality of an extraction with no gold reference. | | [`evaluate_store`](_autosummary/ek.html.md#ek.evaluate_store)(predict, gold, \*[, metric, ...]) | Run `predict` over a gold store and score it per slice. | | [`save_baseline`](_autosummary/ek.html.md#ek.save_baseline)(report, name, \*[, rootdir]) | Freeze a report's aggregate + per-slice scores as a named baseline. | | [`load_baseline`](_autosummary/ek.html.md#ek.load_baseline)(name, \*[, rootdir]) | Load a named baseline (or `None` if it does not exist). | | [`regression_gate`](_autosummary/ek.html.md#ek.regression_gate)(report, baseline, \*[, ...]) | Fail if `report` regresses beyond `tolerance` vs a baseline, per slice. | | [`cohen_kappa`](_autosummary/ek.html.md#ek.cohen_kappa)(rater_a, rater_b) | Cohen's kappa for two raters on nominal labels (chance-corrected agreement). | | [`percent_agreement`](_autosummary/ek.html.md#ek.percent_agreement)(rater_a, rater_b) | Raw fraction of items two raters labelled identically. | | [`krippendorff_alpha`](_autosummary/ek.html.md#ek.krippendorff_alpha)(reliability_data, \*[, level]) | Krippendorff's alpha -- the general inter-annotator agreement coefficient. | | [`default_cost_weight`](_autosummary/ek.html.md#ek.default_cost_weight)(grammar, ref) | The default `CostWeight`: read the importance weight off the schema. | | [`default_canonicalizer`](_autosummary/ek.html.md#ek.default_canonicalizer)() | A conservative default: NFC -> casefold -> collapse whitespace. | | [`table_from_ocr`](_autosummary/ek.html.md#ek.table_from_ocr)(ocr_result, \*[, parser]) | Extract a normalized [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table) from an `OcrResult`. | | [`has_table_structure`](_autosummary/ek.html.md#ek.has_table_structure)(ocr_result, \*[, parser]) | Whether [`table_from_ocr()`](_autosummary/ek.html.md#ek.table_from_ocr) recovers a non-empty table from `ocr_result`. | | [`resolve_table_parser`](_autosummary/ek.html.md#ek.resolve_table_parser)(ref) | Resolve a table parser reference (a registered name or a callable) to a callable. | | [`engine_yields_tables`](_autosummary/ek.html.md#ek.engine_yields_tables)(engine) | Whether `engine`'s capability profile claims scoreable table structure. | | [`rover`](_autosummary/ek.html.md#ek.rover)(hypotheses, \*[, use_confidence, ...]) | Align N hypotheses, vote per slot, and emit consensus + per-position agreement. | | [`expected_calibration_error`](_autosummary/ek.html.md#ek.expected_calibration_error)(probs, correct, \*) | Expected Calibration Error: weighted mean gap between confidence and accuracy. | | [`reliability_curve`](_autosummary/ek.html.md#ek.reliability_curve)(probs, correct, \*[, n_bins]) | Per-bin `{confidence, accuracy, count}` for a reliability diagram. | | [`save_calibrator`](_autosummary/ek.html.md#ek.save_calibrator)(calibrator, name, \*[, rootdir]) | Persist a fitted calibrator's parameters to the `calibrators` store. | | [`load_calibrator`](_autosummary/ek.html.md#ek.load_calibrator)(name, \*[, rootdir]) | Reconstruct a persisted calibrator by name (dispatched on its `kind`). | | [`mean_probability`](_autosummary/ek.html.md#ek.mean_probability)(probs) | Fuse calibrated per-family probabilities by their mean (the default combiner). | | [`min_probability`](_autosummary/ek.html.md#ek.min_probability)(probs) | Fuse calibrated per-family probabilities by their minimum (conservative). | | [`risk_coverage_curve`](_autosummary/ek.html.md#ek.risk_coverage_curve)(probs, correct, \*[, ...]) | The risk-coverage trade-off: `{threshold, coverage, selective_risk}` per point. | | [`split_conformal_quantile`](_autosummary/ek.html.md#ek.split_conformal_quantile)(scores, alpha) | The split-conformal threshold: the `ceil((n+1)(1-alpha)) / n` empirical quantile. | | [`checksum_validator`](_autosummary/ek.html.md#ek.checksum_validator)([kind, field_name, layer]) | A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a value failing a named checksum. | | [`regex_validator`](_autosummary/ek.html.md#ek.regex_validator)(pattern, \*[, field_name, layer]) | A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a value not fully matching `pattern`. | | [`range_validator`](_autosummary/ek.html.md#ek.range_validator)(lo, hi, \*[, field_name, layer]) | A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a number outside `[lo, hi]`. | | [`enum_validator`](_autosummary/ek.html.md#ek.enum_validator)(members, \*[, field_name, layer]) | A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a value not in an allowed set. | | [`schema_validator`](_autosummary/ek.html.md#ek.schema_validator)(value, \*[, spec]) | A spec-driven [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator): type and `domain` checks from a FieldSpec. | | [`totals_consistent`](_autosummary/ek.html.md#ek.totals_consistent)(record, \*, total_key, ...) | Flag when `record[total_key]` does not equal the sum of `item_keys` (±tol). | | [`luhn_check`](_autosummary/ek.html.md#ek.luhn_check)(number) | Luhn (mod-10) checksum: credit cards, IMEIs, some national IDs. | | [`iban_check`](_autosummary/ek.html.md#ek.iban_check)(iban) | IBAN validity via the ISO 13616 / ISO 7064 mod-97 rule (must equal 1). | | [`isbn_check`](_autosummary/ek.html.md#ek.isbn_check)(value) | Validate an ISBN-10 or ISBN-13 by length (digits/hyphens/spaces ignored). | | [`validation_pipeline`](_autosummary/ek.html.md#ek.validation_pipeline)(\*layers[, ...]) | Compose validators/correctors into a cheapest -> most-expensive pipeline. | | [`canonicalize_corrector`](_autosummary/ek.html.md#ek.canonicalize_corrector)(normalize, \*[, ...]) | L0: fold a value to canonical form, emitting a CORRECT finding when it changes. | | [`lexicon_corrector`](_autosummary/ek.html.md#ek.lexicon_corrector)(vocabulary, \*[, ...]) | L2: resolve a value against a **closed** vocabulary by fuzzy match (`rapidfuzz`). | | [`cross_field_validator`](_autosummary/ek.html.md#ek.cross_field_validator)(predicate, \*, message) | FLAG a record when `predicate(record)` is False (a general cross-field check). | | [`ordering_validator`](_autosummary/ek.html.md#ek.ordering_validator)(keys, \*[, strict, layer]) | FLAG a record whose comparable values at `keys` are not ascending. | | [`benford_findings`](_autosummary/ek.html.md#ek.benford_findings)(numbers, \*[, field, tol, ...]) | FLAG a numeric field whose first-digit distribution deviates from Benford's law. | | [`zscore_anomaly_findings`](_autosummary/ek.html.md#ek.zscore_anomaly_findings)(numbers, \*[, field, ...]) | FLAG numeric values that are robust-z-score outliers (median + MAD based). | | [`lm_surprisal_validator`](_autosummary/ek.html.md#ek.lm_surprisal_validator)(scorer, \*, threshold) | L3: FLAG a value whose language-model surprisal crosses `threshold`. | | [`llm_corrector`](_autosummary/ek.html.md#ek.llm_corrector)(correct_fn, \*[, only_flagged, ...]) | L5: a **gated** neural/LLM corrector -- the only layer that invents content. | | [`stop_on_correction`](_autosummary/ek.html.md#ek.stop_on_correction)(findings) | Stop policy: end the pass as soon as any layer proposes a correction. | | [`stop_on_flag`](_autosummary/ek.html.md#ek.stop_on_flag)(findings) | Stop policy: end the pass at the first FLAG (fail fast). | | [`register`](_autosummary/ek.html.md#ek.register)(namespace, name[, obj]) | Register `obj` under `namespace`/`name` (usable as a decorator). | | [`get`](_autosummary/ek.html.md#ek.get)(namespace, name) | Resolve a registered strategy by name (discovering entry points first). | | [`resolve`](_autosummary/ek.html.md#ek.resolve)(namespace, ref, \*[, default]) | Coerce a strategy reference to a callable. | | [`names`](_autosummary/ek.html.md#ek.names)(namespace) | All registered names in a namespace (after entry-point discovery). | | [`requires_extra`](_autosummary/ek.html.md#ek.requires_extra)(extra, \*[, packages]) | Decorator: fail with an actionable install hint if an extra is missing. | | [`check_requirements`](_autosummary/ek.html.md#ek.check_requirements)(\*[, engine, extra]) | Report what an OCR engine or feature extra needs, without raising. | | [`json_store`](_autosummary/ek.html.md#ek.json_store)(kind, \*[, rootdir]) | A JSON `MutableMapping` for one `kind` of artifact. | | [`mall`](_autosummary/ek.html.md#ek.mall)(\*[, rootdir]) | The full stores-of-stores mall (one JSON store per `KINDS`). | | [`app_folder`](_autosummary/ek.html.md#ek.app_folder)() | The root `ek` data folder (e.g. `~/.local/share/ek`). | | [`persistent_cache`](_autosummary/ek.html.md#ek.persistent_cache)([func, kind, key, rootdir]) | Memoize a plain function's result in a `kind` JSON store, across sessions. | | [`cache_this`](_autosummary/ek.html.md#ek.cache_this)([func, cache, key, pre_cache, ...]) | Unified caching decorator for properties and methods with persistent storage support. | | [`run_suite`](_autosummary/ek.html.md#ek.run_suite)(agent, tasks, \*[, k, check, ...]) | Run `agent` over a task suite, `k` trials per task, and report reliability + cost. | | [`pass_at_k`](_autosummary/ek.html.md#ek.pass_at_k)(\*, n, c, k) | Unbiased `pass@k`: probability that **at least one** of k samples succeeds. | | [`pass_hat_k`](_autosummary/ek.html.md#ek.pass_hat_k)(\*, n, c, k) | Unbiased `pass^k`: probability that **all k** independent trials succeed. | ### Classes | [`GateResult`](_autosummary/ek.html.md#ek.GateResult)(passed, metric, higher_is_better, ...) | Outcome of a [`regression_gate()`](_autosummary/ek.html.md#ek.regression_gate) check. | |------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------| | [`GraphGrammar`](_autosummary/ek.html.md#ek.GraphGrammar)([node_types, edge_types]) | The frozen schema SSOT: typed nodes/edges plus their importance weights. | | [`FieldSpec`](_autosummary/ek.html.md#ek.FieldSpec)(name[, type, importance, domain, ...]) | A leaf attribute of a node: its type, validation domain and cost weight. | | [`NodeType`](_autosummary/ek.html.md#ek.NodeType)(name[, fields, importance]) | A node kind in the typed graph (e.g. an "invoice" or a "line item"). | | [`EdgeType`](_autosummary/ek.html.md#ek.EdgeType)(name, src, dst[, importance]) | A directed relation kind between two [`NodeType`](_autosummary/ek.html.md#ek.NodeType) names. | | [`TypeRef`](_autosummary/ek.html.md#ek.TypeRef)(kind, name[, field]) | A reference into the schema, so an injected cost function can read weights. | | [`AnnotatedExtraction`](_autosummary/ek.html.md#ek.AnnotatedExtraction)(grammar[, estimates]) | Layer B: the verification metadata, referencing a frozen Layer-A grammar. | | [`FieldEstimate`](_autosummary/ek.html.md#ek.FieldEstimate)(value[, raw_signals, ...]) | One extracted value plus all of its reference-free verification metadata. | | [`NodePath`](_autosummary/ek.html.md#ek.NodePath)(node_id, node_type[, field]) | Addresses a node (or one of its fields) in an extracted graph; keys Layer B. | | [`Provenance`](_autosummary/ek.html.md#ek.Provenance)([engine, source_span, bbox, ...]) | Where an extracted value came from, for audit and click-to-source review. | | [`Finding`](_autosummary/ek.html.md#ek.Finding)(field, layer[, severity, message, ...]) | One validator observation about a value: a flag, optionally a correction. | | [`Decision`](_autosummary/ek.html.md#ek.Decision)(\*values) | Terminal action a selective-prediction gate emits for an extracted value. | | [`Severity`](_autosummary/ek.html.md#ek.Severity)(\*values) | Whether a [`Finding`](_autosummary/ek.html.md#ek.Finding) merely flags or can also correct a value. | | [`Score`](_autosummary/ek.html.md#ek.Score)(value[, precision, recall, f1, ...]) | The result of comparing one prediction against one reference. | | [`Report`](_autosummary/ek.html.md#ek.Report)([metric, aggregate, n, scores, ...]) | Aggregate of many [`Score`](_autosummary/ek.html.md#ek.Score) s over a corpus, with optional per-slice cuts. | | [`QualityReport`](_autosummary/ek.html.md#ek.QualityReport)([calibrated_confidence, ...]) | The result of a reference-free quality estimate for one extraction. | | [`Metric`](_autosummary/ek.html.md#ek.Metric)(\*args, \*\*kwargs) | Compares one prediction to one gold reference, returning a [`Score`](_autosummary/ek.html.md#ek.Score). | | [`Validator`](_autosummary/ek.html.md#ek.Validator)(\*args, \*\*kwargs) | Reference-free check on a value, yielding zero or more [`Finding`](_autosummary/ek.html.md#ek.Finding) s. | | [`Calibrator`](_autosummary/ek.html.md#ek.Calibrator)(\*args, \*\*kwargs) | Maps a raw scalar signal to a probability after being `fit` on labels. | | [`DecisionPolicy`](_autosummary/ek.html.md#ek.DecisionPolicy)(\*args, \*\*kwargs) | Turns a (calibrated) confidence into an accept/flag/block [`Decision`](_autosummary/ek.html.md#ek.Decision). | | [`Signal`](_autosummary/ek.html.md#ek.Signal)(\*args, \*\*kwargs) | A reference-free quality signal: extractor output -> raw scalar score. | | [`Canonicalizer`](_autosummary/ek.html.md#ek.Canonicalizer)([steps, version, audit_marks]) | An ordered, named, versioned pipeline of `str -> str` steps. | | [`StringMetric`](_autosummary/ek.html.md#ek.StringMetric)([mode, canonicalizer]) | CER (`mode='cer'`) or WER (`mode='wer'`) as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric). | | [`FieldMetric`](_autosummary/ek.html.md#ek.FieldMetric)(\*[, canonicalizer]) | Per-field precision/recall/F1 for two dict records. | | [`TypedGraphMetric`](_autosummary/ek.html.md#ek.TypedGraphMetric)([grammar, weights, ...]) | Cost-weighted, type-aware typed-graph edit distance as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric). | | [`TypedGraph`](_autosummary/ek.html.md#ek.TypedGraph)([nodes, edges]) | A typed graph: typed nodes plus directed typed edges between them. | | [`TypedNode`](_autosummary/ek.html.md#ek.TypedNode)(node_id, node_type[, fields]) | A node in a typed graph: an id, a node-type tag, and its field values. | | [`TypedEdge`](_autosummary/ek.html.md#ek.TypedEdge)(src, dst[, edge_type]) | A directed, typed relation between two node ids. | | [`AnlsMetric`](_autosummary/ek.html.md#ek.AnlsMetric)(\*[, threshold]) | ANLS / ANLS\* (nested-JSON) similarity as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric). | | [`SpanF1Metric`](_autosummary/ek.html.md#ek.SpanF1Metric)([scheme, tagging_scheme]) | Entity/slot precision/recall/F1 under an **explicit** [`MatchScheme`](_autosummary/ek.html.md#ek.MatchScheme). | | [`MatchScheme`](_autosummary/ek.html.md#ek.MatchScheme)(\*values) | The (backend, scheme) contract a span/slot F1 is computed under. | | [`TedsMetric`](_autosummary/ek.html.md#ek.TedsMetric)(\*[, structure_only]) | TEDS / TEDS-Struct as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric) (clean-room on `apted`). | | [`GritsMetric`](_autosummary/ek.html.md#ek.GritsMetric)(\*[, structure_only]) | GriTS-Top / GriTS-Con as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric) (pure-Python, P/R aware). | | [`Table`](_autosummary/ek.html.md#ek.Table)([rows]) | A table as an ordered list of rows of [`Cell`](_autosummary/ek.html.md#ek.Cell) s (the logical cells). | | [`Cell`](_autosummary/ek.html.md#ek.Cell)([text, rowspan, colspan]) | One logical table cell: its text plus its row/column span (>=1 each). | | [`RoverConsensus`](_autosummary/ek.html.md#ek.RoverConsensus)([tokens, slots, agreement, ...]) | The result of a ROVER pass over N hypotheses. | | [`AgreementSignal`](_autosummary/ek.html.md#ek.AgreementSignal)([cost_tier, use_confidence, ...]) | ROVER multi-engine agreement as a reference-free [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal). | | [`VerifierSignal`](_autosummary/ek.html.md#ek.VerifierSignal)([validators, cost_tier]) | Deterministic verifier evidence as a reference-free [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal). | | [`LogprobSignal`](_autosummary/ek.html.md#ek.LogprobSignal)([aggregator, alpha, cost_tier]) | Aggregate token log-probabilities into one field score (a [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal)). | | [`IntrinsicConfidenceSignal`](_autosummary/ek.html.md#ek.IntrinsicConfidenceSignal)([pool, cost_tier]) | Aggregate an extractor's per-unit confidences into a field score. | | [`PlattCalibrator`](_autosummary/ek.html.md#ek.PlattCalibrator)([a, b, max_iter, kind]) | Platt scaling: `sigmoid(a * score + b)`, fit by Newton/IRLS on labels. | | [`IsotonicCalibrator`](_autosummary/ek.html.md#ek.IsotonicCalibrator)([x, y, kind]) | Isotonic (monotonic non-decreasing) calibration via pool-adjacent-violators. | | [`TemperatureCalibrator`](_autosummary/ek.html.md#ek.TemperatureCalibrator)([T, kind, t_min, ...]) | Temperature scaling: `sigmoid(logit / T)` with one `T` fit on a holdout. | | [`GroupCalibrator`](_autosummary/ek.html.md#ek.GroupCalibrator)([factory, by_group, pooled, ...]) | Per-group (Mondrian) calibration: one calibrator per `NodeType`/`FieldSpec`. | | [`SignalFamilyCalibrator`](_autosummary/ek.html.md#ek.SignalFamilyCalibrator)([calibrators, ...]) | One calibrator per signal family, fused by a documented combination rule. | | [`CostSensitiveGate`](_autosummary/ek.html.md#ek.CostSensitiveGate)([rho, block_threshold, ...]) | Accept/flag/block from the cost ratio `rho = c_FN / c_FP` on calibrated `p`. | | [`ConformalGate`](_autosummary/ek.html.md#ek.ConformalGate)([alpha, nonconformity, q, \_cal]) | Split-conformal accept/flag with a finite-sample **marginal** guarantee. | | [`GroupConformalGate`](_autosummary/ek.html.md#ek.GroupConformalGate)([alpha, nonconformity, ...]) | Mondrian (class-conditional) conformal: one [`ConformalGate`](_autosummary/ek.html.md#ek.ConformalGate) per group. | | [`RiskControlGate`](_autosummary/ek.html.md#ek.RiskControlGate)([target, loss, loss_bound, lam]) | Conformal Risk Control: accept `p >= lambda` bounding the accepted-error rate. | | [`ValidationResult`](_autosummary/ek.html.md#ek.ValidationResult)(original, value[, findings]) | The outcome of running a value through a [`validation_pipeline()`](_autosummary/ek.html.md#ek.validation_pipeline). | | [`Corrector`](_autosummary/ek.html.md#ek.Corrector)(\*args, \*\*kwargs) | A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that may also CORRECT. | | [`Episode`](_autosummary/ek.html.md#ek.Episode)([task_id, trajectory, output, ...]) | Layer B: one agent run over one task -- the object agent evaluation scores. | | [`TaskSpec`](_autosummary/ek.html.md#ek.TaskSpec)(task_id[, input, gold, value, ...]) | One task in a suite: its input, its oracle, its value, and its allowed tools. | | [`ToolSpec`](_autosummary/ek.html.md#ek.ToolSpec)(name[, params, importance, destructive]) | One tool the agent may call: its argument schema and its error costs. | ### *class* ek.AgreementSignal(cost_tier=3, use_confidence=True, conf_weight=0.5, null_conf=0.7, tokenize=None, max_tokens=5000) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) ROVER multi-engine agreement as a reference-free [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal). Cost tier 3 (`N` engine runs): try the deterministic verifier layer and any free intrinsic confidence first. Called on a collection of hypotheses, it runs [`rover()`](_autosummary/ek.html.md#ek.rover) and returns the mean per-position agreement as the raw signal – *uncalibrated*, like every signal, so it must pass through a [`Calibrator`](_autosummary/ek.base.html.md#ek.base.Calibrator) before any gate reads it. ### Example ```pycon >>> sig = AgreementSignal() >>> round(sig(["the cat sat", "the cat sit", "the bat sat"]), 3) 0.778 ``` ### *class* ek.AnlsMetric(, threshold=0.5) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) ANLS / ANLS\* (nested-JSON) similarity as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric). Handles a bare string (classic ANLS) or an arbitrarily nested dict/list/tuple/`None` structure (ANLS\*); the backend dispatches on type. * **Parameters:** **threshold** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – ANLS zeroes a per-leaf similarity below this value before averaging (the classic ANLS 0.5 cut tolerates minor OCR/spelling noise). `anls_star` exposes no parameter for this – it reads a class constant `ANLSTree.THRESHOLD` (default `0.5`) – so the metric applies the threshold by setting that constant around the call and restoring it. **Caveat:** this mutates a process-global, so it is not thread-safe; concurrent `AnlsMetric` calls with different thresholds in the same process can interfere. Defaults to `0.5`. #### aggregate(scores) Corpus ANLS = mean of per-item ANLS (each is already length-normalized). Unlike CER/WER (which accumulate global edit counts) or field-F1 (micro-TP/ FP/FN), ANLS is defined as the *average* normalized similarity over the evaluation set – the “A” in ANLS – so the corpus statistic is the plain mean of the per-item scores. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.AnnotatedExtraction(grammar, estimates=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Layer B: the verification metadata, referencing a frozen Layer-A grammar. The grammar is held *by reference* and never mutated; `estimates` maps a [`NodePath`](_autosummary/ek.html.md#ek.NodePath) to the [`FieldEstimate`](_autosummary/ek.html.md#ek.FieldEstimate) for that node/field. ### *class* ek.Calibrator(\*args, \*\*kwargs) Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol) Maps a raw scalar signal to a probability after being `fit` on labels. ### *class* ek.Canonicalizer(steps=(), , version='', audit_marks=False) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) An ordered, named, versioned pipeline of `str -> str` steps. * **Parameters:** * **steps** ([`Sequence`](https://docs.python.org/3/library/typing.html#typing.Sequence)[`Union`[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]]]) – Step references – names registered under `"normalizers"` or plain callables. Applied left to right. * **version** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Override the auto-derived version string (the `+`-joined names). * **audit_marks** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – If true, warn when a step would run on text carrying combining marks (the cross-script damage guard). #### *property* steps *: [list](https://docs.python.org/3/builtins/stdtypes.html#list)* The resolved step names, in order. #### *property* version *: [str](https://docs.python.org/3/builtins/stdtypes.html#str)* Stable identifier of this pipeline (persist it next to a gold set). ### *class* ek.Cell(text='', rowspan=1, colspan=1) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One logical table cell: its text plus its row/column span (>=1 each). ### *class* ek.ConformalGate(alpha=0.1, nonconformity=>, q=inf, \_cal=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Split-conformal accept/flag with a finite-sample **marginal** guarantee. Fit on `(calibrated_prob, correct)`: the nonconformity scores of the *correct* calibration items set a threshold `q` such that, among truly-correct test items exchangeable with them, at most `alpha` are flagged. The guarantee is *marginal*, not per-field-type – use [`GroupConformalGate`](_autosummary/ek.html.md#ek.GroupConformalGate) for that. * **Parameters:** * **alpha** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Target false-flag rate on correct items (e.g. `0.1`). * **nonconformity** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`float`](https://docs.python.org/3/builtins/functions.html#float)], [`float`](https://docs.python.org/3/builtins/functions.html#float)]) – Maps calibrated confidence -> nonconformity (default `1 - confidence`: low confidence is nonconforming). #### fit(probs, correct) Fit `q` from the nonconformity scores of the *correct* calibration items. Non-finite confidences are dropped (a single `NaN` would otherwise sort to the end and corrupt the quantile, silently breaking the coverage guarantee). * **Return type:** [`ConformalGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.ConformalGate) #### p_value(confidence) Per-instance conformal p-value (small = atypical of correct items). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.Corrector(\*args, \*\*kwargs) Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol) A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that may also CORRECT. Same callable shape as a Validator, but it can emit a `severity=CORRECT` [`Finding`](_autosummary/ek.base.html.md#ek.base.Finding) whose `suggestion` is the proposed replacement value. The `layer` attribute names its place on the cheap->expensive spine. ### *class* ek.CostSensitiveGate(rho=1.0, block_threshold=0.0, accept_threshold=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Accept/flag/block from the cost ratio `rho = c_FN / c_FP` on calibrated `p`. Accept when the expected cost of accepting is below that of routing to a human. Accepting risks an undetected error: cost `(1 - p) * c_FN`. Flagging a value that was actually *correct* wastes a review: cost `p * c_FP` (the false-positive cost is incurred only with probability `p`, not unconditionally). So accept iff `(1 - p) * c_FN <= p * c_FP`, i.e. the Bayes-optimal threshold `p >= c_FN / (c_FN + c_FP) = rho / (1 + rho)`. A larger `rho` (misses much costlier than reviews) raises the bar to auto-accept (`tau -> 1` as `rho -> inf`); `rho = 1` gives the symmetric `tau = 0.5`. `block_threshold` adds an optional hard-fail band at the bottom. * **Parameters:** * **rho** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Cost ratio `c_FN / c_FP` (>= 0). The single lever; no magic numbers. * **block_threshold** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Calibrated probability at or below which to `block` rather than `flag` (default `0` -> never block on probability alone; a failed verifier is what forces a block, upstream). * **accept_threshold** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`float`](https://docs.python.org/3/builtins/functions.html#float)]) – Override the derived accept threshold explicitly. #### *property* tau *: [float](https://docs.python.org/3/builtins/functions.html#float)* The accept threshold on calibrated probability (derived from `rho`). ### *class* ek.Decision(\*values) Bases: [`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Enum`](https://docs.python.org/3/library/enum.html#enum.Enum) Terminal action a selective-prediction gate emits for an extracted value. ### *class* ek.DecisionPolicy(\*args, \*\*kwargs) Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol) Turns a (calibrated) confidence into an accept/flag/block [`Decision`](_autosummary/ek.html.md#ek.Decision). ### *class* ek.EdgeType(name, src, dst, importance=1.0) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A directed relation kind between two [`NodeType`](_autosummary/ek.html.md#ek.NodeType) names. ### *class* ek.Episode(task_id='', trajectory=, output=None, final_state=None, cost=None, success=None, run=None, provenance=None, meta=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Layer B: one agent run over one task – the object agent evaluation scores. The direct analog of `AnnotatedExtraction`: it rides alongside the frozen Layer-A grammar and carries the run’s metadata. It is scored *offline* against gold ([`ek.score()`](_autosummary/ek.html.md#ek.score) / [`ek.evaluate()`](_autosummary/ek.html.md#ek.evaluate)) and estimated *online* against a judge or a consensus ([`ek.estimate_quality()`](_autosummary/ek.html.md#ek.estimate_quality)) – the same object, both halves. * **Parameters:** * **task_id** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Which task this episode answers (the k-trial grouping key). * **trajectory** ([`Trajectory`](_autosummary/ek.agents.base.html.md#ek.agents.base.Trajectory)) – The tool calls taken. * **output** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The agent’s final user-facing answer. * **final_state** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The end state of the world (the DB, the filesystem) – what a state-based oracle actually grades. Success is a *final-state* check, not a surface-text check. * **cost** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Cost`](_autosummary/ek.agents.base.html.md#ek.agents.base.Cost)]) – The `Cost` consumed. * **success** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`bool`](https://docs.python.org/3/builtins/functions.html#bool)]) – Filled in by a checker; `None` until graded. * **run** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`RunProvenance`](_autosummary/ek.agents.base.html.md#ek.agents.base.RunProvenance)]) – The `RunProvenance` (seed, model, simulator, suite version). * **provenance** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Provenance`](_autosummary/ek.base.html.md#ek.base.Provenance)]) – Optional core [`Provenance`](_autosummary/ek.base.html.md#ek.base.Provenance) for click-to-source audit. * **meta** ([`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)) – Anything else the bridge captured. #### *property* calls *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)* The `(tool, args)` pairs taken, in order. #### graded(success) A copy with `success` set (never mutates – grading stays idempotent). `meta` is copied too: `dataclasses.replace` is a *shallow* copy, so without this the “copy” would share the original’s mutable `meta` dict. * **Return type:** [`Episode`](_autosummary/ek.agents.base.html.md#ek.agents.base.Episode) ### *class* ek.FieldEstimate(value, raw_signals=, confidence=None, findings=(), provenance=None, decision=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One extracted value plus all of its reference-free verification metadata. ### *class* ek.FieldMetric(, canonicalizer=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Per-field precision/recall/F1 for two dict records. * **Parameters:** **canonicalizer** – Optional `str -> str` applied to string values before comparison (e.g. casefold/whitespace folding). #### aggregate(scores) Micro-averaged F1 over the corpus (sum TP/FP/FN, then divide). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.FieldSpec(name, type='string', importance=1.0, domain=(), normalizer=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A leaf attribute of a node: its type, validation domain and cost weight. * **Parameters:** * **name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Attribute name. * **type** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – A free-form type tag (e.g. `"string"`, `"number"`, `"date"`); `ek` does not impose a type system – the tag drives metric/validator dispatch and is yours to define. * **importance** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Per-field cost weight; the lever that makes “two extra digits on a donation amount” outweigh “a misspelled city”. Defaults to `1.0`. * **domain** ([`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)) – Optional allowed values: an enum tuple, a `(lo, hi)` range, or a regex string – consumed by validators, never by metrics. * **normalizer** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – Optional key naming a canonicalizer (resolved from the registry) to apply to this field before comparison. ### *class* ek.Finding(field, layer, severity=Severity.FLAG, message='', suggestion=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One validator observation about a value: a flag, optionally a correction. A `suggestion` is present iff the producing layer can *correct* (only the deterministic canonicalizer and the gated LLM corrector freely do so). The original value is always retained elsewhere; a Finding never mutates. ### *class* ek.GateResult(passed, metric, higher_is_better, tolerance, aggregate_current=None, aggregate_baseline=None, regressions=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Outcome of a [`regression_gate()`](_autosummary/ek.html.md#ek.regression_gate) check. ### *class* ek.GraphGrammar(node_types=, edge_types=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The frozen schema SSOT: typed nodes/edges plus their importance weights. Methods return the cost weight for a referenced type, defaulting to `1.0` for anything not declared, so a partial grammar is always usable. #### cost(ref) Importance weight for any [`TypeRef`](_autosummary/ek.html.md#ek.TypeRef) (the default `CostWeight`). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) #### edge_cost(name) Importance weight of an edge type (`1.0` if undeclared). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) #### field_cost(node, field_name) Importance weight of a field on a node type (`1.0` if undeclared). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) #### node_cost(name) Importance weight of a node type (`1.0` if undeclared). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.GritsMetric(, structure_only=False) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) GriTS-Top / GriTS-Con as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric) (pure-Python, P/R aware). * **Parameters:** **structure_only** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – When `True`, score cell topology only (GriTS-Top); when `False` (default), average topology with cell-content similarity (GriTS-Con). #### aggregate(scores) Corpus GriTS F1: pool overlap and per-table cell counts, then divide once. GriTS precision = overlap / ``` |pred cells| ``` and recall = overlap / ``` |gold cells| ``` ; the corpus statistic micro-averages by summing the numerators and denominators globally – never a mean of per-table F1s. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.GroupCalibrator(factory=, by_group=, pooled=None, kind='group') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Per-group (Mondrian) calibration: one calibrator per `NodeType`/`FieldSpec`. Distribution-free *conditional* (per-field-type) coverage is impossible in general (Barber et al. 2019); calibrating separately per group restores it approximately. `fit` takes a parallel `groups` sequence; `__call__` routes by group key, falling back to a pooled calibrator for unseen groups. * **Parameters:** **factory** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – Zero-arg callable producing a fresh per-group calibrator (default [`PlattCalibrator`](_autosummary/ek.html.md#ek.PlattCalibrator)). #### factory alias of [`PlattCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.PlattCalibrator) #### fit(scores, correct, , groups) Fit one calibrator per distinct group key, plus a pooled fallback. * **Return type:** [`GroupCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.GroupCalibrator) #### to_dict() Serialize each per-group calibrator (keyed by stringified group) plus the pooled fallback, so a fitted Mondrian calibrator round-trips like the others. Group keys are stringified (they are `NodeType`/`FieldSpec` names). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### *class* ek.GroupConformalGate(alpha=0.1, nonconformity=>, by_group=, pooled=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Mondrian (class-conditional) conformal: one [`ConformalGate`](_autosummary/ek.html.md#ek.ConformalGate) per group. Restores approximate per-field-type validity by calibrating separately per group key (e.g. `NodeType`/`FieldSpec`), with a pooled fallback for unseen groups. ### *class* ek.IntrinsicConfidenceSignal(pool='min', cost_tier=2) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Aggregate an extractor’s per-unit confidences into a field score. * **Parameters:** * **pool** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – `"min"` (weakest unit), `"mean"`, or `"geo_mean"` over the per-unit confidences (already probabilities in `[0, 1]`). * **cost_tier** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – `2` – free (the posteriors are emitted at inference). ### Example ```pycon >>> IntrinsicConfidenceSignal(pool="min")([0.99, 0.4, 0.95]) 0.4 ``` ### *class* ek.IsotonicCalibrator(x=, y=, kind='isotonic') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Isotonic (monotonic non-decreasing) calibration via pool-adjacent-violators. More flexible than Platt; needs more calibration data and can overfit small sets. Predicts by linear interpolation between fitted points, clipped at the ends. #### fit(scores, correct) Fit the monotonic step function to `(score, correct)` pairs. * **Return type:** [`IsotonicCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.IsotonicCalibrator) ### *class* ek.LogprobSignal(aggregator=, alpha=0.6, cost_tier=2) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Aggregate token log-probabilities into one field score (a [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal)). * **Parameters:** * **aggregator** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – A logprob aggregator (name registered under `aggregators` or a callable). Defaults to `geo_mean()`. * **alpha** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Length penalty passed to `length_normalized()` only. * **cost_tier** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – `2` – free intrinsic signal (already computed at inference). #### aggregator() Geometric mean of token probabilities, `exp(mean log p)` (perplexity-inverse). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.MatchScheme(\*values) Bases: [`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Enum`](https://docs.python.org/3/library/enum.html#enum.Enum) The (backend, scheme) contract a span/slot F1 is computed under. The two backends legitimately disagree (seqeval ignores other-type tags at the tag level; nervaluate includes them), so the scheme is always explicit. The `SEQEVAL_*` members consume BIO/IOBES *tag sequences*; the others consume SemEval-2013 *span lists* via nervaluate. #### EXACT *= 'exact'* exact span boundaries, type ignored. * **Type:** nervaluate exact #### PARTIAL *= 'partial'* boundary *overlap* (half credit), type ignored. * **Type:** nervaluate partial #### SEQEVAL_CONLL *= 'seqeval_conll'* seqeval, CoNLL `conlleval`-compatible entity F1 (the default seqeval mode). #### SEQEVAL_STRICT *= 'seqeval_strict'* span boundaries AND type must match, keyed to a scheme. * **Type:** seqeval strict mode #### STRICT *= 'strict'* exact span boundaries AND correct type. * **Type:** nervaluate SemEval-2013 strict #### TYPE *= 'type'* correct type with at least some span overlap. * **Type:** nervaluate type ### *class* ek.Metric(\*args, \*\*kwargs) Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol) Compares one prediction to one gold reference, returning a [`Score`](_autosummary/ek.html.md#ek.Score). ### *class* ek.NodePath(node_id, node_type, field=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Addresses a node (or one of its fields) in an extracted graph; keys Layer B. ### *class* ek.NodeType(name, fields=, importance=1.0) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A node kind in the typed graph (e.g. an “invoice” or a “line item”). ### *class* ek.PlattCalibrator(a=1.0, b=0.0, max_iter=100, kind='platt') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Platt scaling: `sigmoid(a * score + b)`, fit by Newton/IRLS on labels. The default calibrator: works on any scalar (aggregated confidence/logprob), no logits required. Uses Platt’s target smoothing so it does not overfit small calibration sets. * **Parameters:** **max_iter** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Newton iterations (converges in a handful for a 2-parameter model). #### fit(scores, correct) Fit `a, b` to maximise the likelihood of `correct` given `scores`. * **Return type:** [`PlattCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.PlattCalibrator) ### *class* ek.Provenance(engine='', source_span=None, bbox=None, raw_transcripts=()) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Where an extracted value came from, for audit and click-to-source review. `bbox` is intentionally typed `Any` so `ek` core need not import any OCR package; the OCR instance stores normalized `[0, 1]` top-left-origin geometry. ### *class* ek.QualityReport(calibrated_confidence=None, decision=None, findings=(), raw_signals=, provenance=, per_field=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The result of a reference-free quality estimate for one extraction. For a single value the scalar fields carry the verdict. For a whole [`AnnotatedExtraction`](_autosummary/ek.html.md#ek.AnnotatedExtraction), `per_field` maps each [`NodePath`](_autosummary/ek.html.md#ek.NodePath) to its updated [`FieldEstimate`](_autosummary/ek.html.md#ek.FieldEstimate) (confidence/findings/decision filled in), and the scalar fields summarize: `calibrated_confidence` is the weakest field, and `decision` the most severe (`block` > `flag` > `accept`). ### *class* ek.Report(metric='', aggregate=None, n=0, scores=, per_slice=, detail=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Aggregate of many [`Score`](_autosummary/ek.html.md#ek.Score) s over a corpus, with optional per-slice cuts. `aggregate` is the corpus-level headline (computed by the metric’s own aggregator – e.g. globally accumulated WER – not a naive mean of per-item values). `per_slice` maps a slice label to its own aggregate. ### *class* ek.RiskControlGate(target=0.05, loss=>, loss_bound=1.0, lam=1.0) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Conformal Risk Control: accept `p >= lambda` bounding the accepted-error rate. Picks the *smallest* threshold `lambda` (the most coverage) whose finite-sample risk bound is at or below `target`. The controlled quantity is `E[ loss(item) * 1(accepted) ]` – with the default 0/1 loss, the population rate of accepted-yet-wrong items. * **Parameters:** * **target** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Upper bound on the accepted-error rate (e.g. `0.05`). * **loss** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`bool`](https://docs.python.org/3/builtins/functions.html#bool)], [`float`](https://docs.python.org/3/builtins/functions.html#float)]) – `correct -> loss` in `[0, loss_bound]`; default `0` if correct else `1`. Any monotone loss bounded by `loss_bound` works. * **loss_bound** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – The loss’s upper bound `B` – the CRC finite-sample slack is `+B/(n+1)`. Must match `loss`’s range or the guarantee is invalid; keep the default `1.0` for the 0/1 loss. #### fit(probs, correct) Find the smallest accept threshold whose risk bound is <= `target`. * **Return type:** [`RiskControlGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.RiskControlGate) ### *class* ek.RoverConsensus(tokens=, slots=, agreement=, n_engines=0) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The result of a ROVER pass over N hypotheses. #### *property* mean_agreement *: [float](https://docs.python.org/3/builtins/functions.html#float)* Mean per-position agreement over consensus tokens, a reference-free confidence in `[0, 1]`: `1.0` means every engine agreed at every emitted position; lower means at least one engine dissented. An **empty** consensus (no token survived voting – NULL won every slot) means the engines agreed on *nothing*, so this returns `0.0` when 2+ engines were fused, NOT a misleading `1.0` (which would feed a maximal raw confidence to the calibrator/gate). A single engine – or none – is vacuously `1.0`. #### *property* text *: [str](https://docs.python.org/3/builtins/stdtypes.html#str)* The consensus transcription (space-joined winning tokens). ### *class* ek.Score(value, precision=None, recall=None, f1=None, metric='', detail=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The result of comparing one prediction against one reference. `value` is the headline number (higher is better unless a metric documents otherwise). The optional decomposition fields let structured metrics carry precision/recall/F1, an alignment, and raw counts (e.g. edit operations) so that corpus aggregation can be done *correctly* (e.g. global WER accumulation) rather than by averaging per-item scores. `float(score)` returns `value`. ### *class* ek.Severity(\*values) Bases: [`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Enum`](https://docs.python.org/3/library/enum.html#enum.Enum) Whether a [`Finding`](_autosummary/ek.html.md#ek.Finding) merely flags or can also correct a value. ### *class* ek.Signal(\*args, \*\*kwargs) Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol) A reference-free quality signal: extractor output -> raw scalar score. The `cost_tier` attribute (lower is cheaper) lets an escalation policy run cheap signals first and pay for expensive ones only on residual uncertainty. ### *class* ek.SignalFamilyCalibrator(calibrators=, combine=, factory=, fusion=None, kind='signal_family') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One calibrator per signal family, fused by a documented combination rule. Averaging heterogeneous raw signals (an LLM-judge score, a token logprob, a ROVER agreement) and calibrating the mean discards exactly what a calibrator needs: each family has its own scale, reliability and direction of miscalibration. This calibrates every signal *by name* with its own calibrator, then fuses the calibrated probabilities with `combine`. Pass it (or simply a `Mapping[str, Calibrator]`) as `estimate_quality(..., calibrator=...)`. **The fused value must itself be calibrated before it is gated.** Fusing calibrated probabilities (by mean, min, …) does not yield a calibrated probability: agreeing families make the mean under-confident, a weak family drags it toward the base rate. So [`fit()`](_autosummary/ek.html.md#ek.SignalFamilyCalibrator.fit) also learns a second-stage `fusion` calibrator on `combine`’s output, and `__call__` applies it. Without a `fusion` calibrator (e.g. a hand-built mapping), [`ek.estimate_quality()`](_autosummary/ek.html.md#ek.estimate_quality) refuses to gate a value fused from two or more families unless told `assume_calibrated=True` (Hard Rule 1). A value from a single family is that family’s calibrated probability and gates as is. A field’s own upstream `confidence` is just one more family here, named `"confidence"`: include it in the training rows under that name. * **Parameters:** * **calibrators** ([`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)) – `signal name -> Calibrator`. A `"*"` entry is the fallback for any signal without its own entry. A signal with neither is refused (`ValueError`) rather than silently dropped or passed through raw (Hard Rule 1). * **combine** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`float`](https://docs.python.org/3/builtins/functions.html#float)]], [`float`](https://docs.python.org/3/builtins/functions.html#float)]) – `Mapping[str, float] -> float` fusing the calibrated values; [`mean_probability()`](_autosummary/ek.html.md#ek.mean_probability) (default) or [`min_probability()`](_autosummary/ek.html.md#ek.min_probability), or any callable of that shape (e.g. a learned stacker). * **factory** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – Zero-arg callable producing a fresh calibrator, used by [`fit()`](_autosummary/ek.html.md#ek.SignalFamilyCalibrator.fit). * **fusion** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Second-stage calibrator applied to `combine`’s output (learned by [`fit()`](_autosummary/ek.html.md#ek.SignalFamilyCalibrator.fit)); `None` leaves the fused value uncalibrated. ### Example ```pycon >>> rows = [{"judge": j, "logprob": l} for j, l in ... [(0.9, -0.1), (0.8, -0.2), (0.85, -2.0), (0.3, -0.1), (0.2, -2.5), (0.7, -1.8)]] >>> correct = [True, True, False, True, False, False] >>> cal = SignalFamilyCalibrator().fit(rows, correct) >>> sorted(cal.calibrators) ['judge', 'logprob'] >>> 0.0 <= cal({"judge": 0.9, "logprob": -0.1}) <= 1.0 True ``` #### calibrate_signals(raw_signals, , group=None) `{name: calibrated probability}` for each raw signal (before fusion). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) #### combine() Fuse calibrated per-family probabilities by their mean (the default combiner). Each family’s calibrated value estimates the *same* event (“this field is correct”), so the mean is a defensible, assumption-light pooling rule. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ```pycon >>> mean_probability({"judge": 0.9, "logprob": 0.7}) 0.8 ``` #### factory alias of [`PlattCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.PlattCalibrator) #### fit(signals, correct) Fit one calibrator per signal name, then the `fusion` calibrator. Each family is fit over the rows where it appears with a finite value. A family with fewer than two such rows, or only one label class, cannot be calibrated and is refused (`ValueError`) rather than turned into a constant that would then dilute the fusion. * **Return type:** [`SignalFamilyCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.SignalFamilyCalibrator) #### fuse(calibrated) Fuse per-family calibrated probabilities: `combine`, then `fusion`. A single family’s value is returned as is: it is already calibrated, and `fusion` was learned on fused values. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) #### to_dict() Serialize every family’s calibrator and the (registered) combiner’s name. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### *class* ek.SpanF1Metric(scheme=None, , tagging_scheme='IOB2') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Entity/slot precision/recall/F1 under an **explicit** [`MatchScheme`](_autosummary/ek.html.md#ek.MatchScheme). * **Parameters:** * **scheme** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`MatchScheme`](_autosummary/ek.metrics.spans.html.md#ek.metrics.spans.MatchScheme)]) – REQUIRED. The (backend, scheme) contract – there is deliberately no default, because seqeval and nervaluate disagree and a bare F1 is ambiguous (`misc/docs/ek_02` §2.2). Pass a [`MatchScheme`](_autosummary/ek.html.md#ek.MatchScheme). * **tagging_scheme** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – For [`MatchScheme.SEQEVAL_STRICT`](_autosummary/ek.html.md#ek.MatchScheme.SEQEVAL_STRICT) only: the BIO/IOBES tagging scheme name (e.g. `"IOB2"`, `"IOBES"`, `"BILOU"`) that strict-mode entity decoding is keyed to. Defaults to `"IOB2"`. * **Raises:** [**TypeError**](https://docs.python.org/3/builtins/exceptions.html#TypeError) – if `scheme` is omitted (the whole point of this metric). #### aggregate(scores) Micro-averaged F1 over the corpus (sum the raw counts, then divide once). seqeval schemes carry TP/FP/FN; nervaluate schemes carry COR/INC/PAR/MIS/SPU (PAR worth half). Both micro-average by summing counts globally – never a mean of per-document F1s. The two backends’ counts are not commensurable, so a corpus mixing them is rejected rather than silently dropping one. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.StringMetric(mode='cer', , canonicalizer=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) CER (`mode='cer'`) or WER (`mode='wer'`) as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric). * **Parameters:** * **mode** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – `"cer"` (character) or `"wer"` (word) error rate. * **canonicalizer** – Optional `str -> str` applied to both sides before scoring (see [`ek.canonicalize`](_autosummary/ek.canonicalize.html.md#module-ek.canonicalize)). Usually supplied by the facade instead. #### aggregate(scores) Corpus error rate = total edits / total reference length (the correct way). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.Table(rows=()) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A table as an ordered list of rows of [`Cell`](_autosummary/ek.html.md#ek.Cell) s (the logical cells). `rows[i]` is the i-th row’s logical cells in reading order; spans are carried on each [`Cell`](_autosummary/ek.html.md#ek.Cell). Use [`from_html()`](_autosummary/ek.html.md#ek.Table.from_html) / [`from_grid()`](_autosummary/ek.html.md#ek.Table.from_grid) to build one, and [`as_grid()`](_autosummary/ek.html.md#ek.Table.as_grid) for the dense 2-D occupancy view GriTS scores. #### as_grid() Expand spans into a dense 2-D grid; a spanned position repeats its cell. Each physical grid position holds the (shared) [`Cell`](_autosummary/ek.html.md#ek.Cell) covering it, so two tables with the same topology have grids of the same shape – the view GriTS aligns position-by-position. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Cell`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Cell)]]] #### *classmethod* coerce(obj) Coerce a [`Table`](_autosummary/ek.html.md#ek.Table), an HTML string, or a 2-D grid into a Table. * **Return type:** [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table) #### *classmethod* from_grid(grid) Build a Table from a 2-D grid of cell texts (`None` = spanned-over). * **Return type:** [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table) #### *classmethod* from_html(html) Parse an HTML `` (with `colspan`/`rowspan`) into a Table. * **Return type:** [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table) ### *class* ek.TaskSpec(task_id, input=None, gold=None, value=1.0, tools=(), slice=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One task in a suite: its input, its oracle, its value, and its allowed tools. * **Parameters:** * **task_id** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Stable identifier (the grouping key for k-trial reliability). * **input** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Whatever the agent under test is called with. * **gold** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The reference the checker grades against (a goal state, an expected answer, a gold trajectory – the checker decides how to read it). * **value** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Task-level cost weight: how much a completed task is worth. The task-level analog of `FieldSpec.importance`; feeds value-weighted cost accounting. * **tools** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`ToolSpec`](_autosummary/ek.agents.base.html.md#ek.agents.base.ToolSpec)]) – The tools this task permits (the Layer-A grammar for the task). * **slice** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – Optional slice label (domain, difficulty, language) – per-slice cuts are mandatory, not optional (a low pass^k concentrated in one hard slice is a different problem from uniform flakiness). #### grammar() The Layer-A grammar for this task’s tool set. * **Return type:** [`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar) ### *class* ek.TedsMetric(, structure_only=False) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) TEDS / TEDS-Struct as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric) (clean-room on `apted`). * **Parameters:** **structure_only** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – When `True`, ignore cell text and score structure alone (TEDS-Struct), isolating table-structure accuracy from OCR noise. Defaults to `False` (full TEDS: structure + content). #### aggregate(scores) Corpus TEDS = **mean** of per-table TEDS (the PubTabNet/OmniDocBench convention). Each per-table TEDS is already node-count-normalized, so the reported corpus statistic is the average, not a pooled edit-rate. The pooled `1 - sum(dist)/sum(nodes)` is available from the per-item `detail` if a size-weighted variant is wanted instead. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.TemperatureCalibrator(T=1.0, kind='temperature', t_min=0.05, t_max=10.0, max_iter=60) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Temperature scaling: `sigmoid(logit / T)` with one `T` fit on a holdout. Use **only when you have logits**: `__call__` expects a *logit*, not a probability. `T > 1` softens overconfidence; the argmax is unchanged. #### fit(logits, correct) Fit `T` by minimising NLL with a bounded 1-D search. * **Return type:** [`TemperatureCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.TemperatureCalibrator) #### t_min *: [float](https://docs.python.org/3/builtins/functions.html#float)* *= 0.05* Golden-section search bounds and iteration count (config, not magic numbers). ### *class* ek.ToolSpec(name, params=, importance=None, destructive=False) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One tool the agent may call: its argument schema and its error costs. * **Parameters:** * **name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The tool/function name (the AST-match ground truth). * **params** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`FieldSpec`](_autosummary/ek.base.html.md#ek.base.FieldSpec)]) – Argument name -> [`FieldSpec`](_autosummary/ek.base.html.md#ek.base.FieldSpec). Each spec’s `importance` is the cost of getting *that argument* wrong, and its `domain` feeds validators – the same contract as an extracted field. * **importance** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`float`](https://docs.python.org/3/builtins/functions.html#float)]) – Cost weight of the *call itself* (a missing or spurious call). Defaults to `DESTRUCTIVE_WEIGHT` when `destructive` is set, else 1.0. * **destructive** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether calling this tool mutates the world irreversibly (a refund, a delete, a send). Drives the default weight and the safety validators. #### to_node_type() Render this tool as a Layer-A [`NodeType`](_autosummary/ek.base.html.md#ek.base.NodeType) (args as fields). * **Return type:** [`NodeType`](_autosummary/ek.base.html.md#ek.base.NodeType) #### *property* weight *: [float](https://docs.python.org/3/builtins/functions.html#float)* The call-level cost weight (explicit, else destructive-aware default). ### *class* ek.TypeRef(kind, name, field=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A reference into the schema, so an injected cost function can read weights. `kind` is one of `"node"`, `"edge"` or `"field"`; for a field, `name` is the node-type name and `field` the field name. ### *class* ek.TypedEdge(src, dst, edge_type='') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A directed, typed relation between two node ids. ### *class* ek.TypedGraph(nodes=(), edges=()) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A typed graph: typed nodes plus directed typed edges between them. #### to_networkx() Build a `networkx.DiGraph` carrying type/fields as node/edge attrs. ### *class* ek.TypedGraphMetric(grammar=None, weights=None, , canonicalizer=None, timeout=10.0, max_nodes=60) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Cost-weighted, type-aware typed-graph edit distance as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric). * **Parameters:** * **grammar** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar)]) – Layer-A [`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar) supplying cost weights (may also be passed per-call to `__call__`). * **weights** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar), [`TypeRef`](_autosummary/ek.base.html.md#ek.base.TypeRef)], [`float`](https://docs.python.org/3/builtins/functions.html#float)]]) – Optional [`CostWeight`](_autosummary/ek.base.html.md#ek.base.CostWeight) overriding the schema weights. * **canonicalizer** – Optional `str -> str` applied to field values before the equality check (the facade passes `normalize=` through here); a schema’s per-field `FieldSpec.normalizer` takes precedence for the fields it names. * **timeout** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Seconds budget for the (NP-hard) GED search; on timeout the best cost found so far is used (an approximation). Default 10.0. #### aggregate(scores) Corpus normalized distance = total raw distance / total max-distance. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.TypedNode(node_id, node_type, fields=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A node in a typed graph: an id, a node-type tag, and its field values. ### *class* ek.ValidationResult(original, value, findings=()) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The outcome of running a value through a [`validation_pipeline()`](_autosummary/ek.html.md#ek.validation_pipeline). `value` is the final value after any applied corrections; `original` is the input; `findings` is the full audit trail (FLAG and CORRECT), in layer order. #### *property* clean *: [bool](https://docs.python.org/3/builtins/functions.html#bool)* No findings at all – the value passed every layer untouched. #### *property* corrected *: [bool](https://docs.python.org/3/builtins/functions.html#bool)* Whether any correction changed the value. #### *property* flagged *: [bool](https://docs.python.org/3/builtins/functions.html#bool)* Whether any layer raised a FLAG (something a human should look at). ### *class* ek.Validator(\*args, \*\*kwargs) Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol) Reference-free check on a value, yielding zero or more [`Finding`](_autosummary/ek.html.md#ek.Finding) s. ### *class* ek.VerifierSignal(validators=, cost_tier=1) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Deterministic verifier evidence as a reference-free [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal). Cost tier 1 – the free, first-to-run layer. Runs a list of [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) s on a value and returns the **fraction that passed** in `[0, 1]` as the raw signal (`1.0` = every check passed). It also exposes the [`Finding`](_autosummary/ek.base.html.md#ek.base.Finding) s it produced via [`findings()`](_autosummary/ek.html.md#ek.VerifierSignal.findings), so the same object feeds both the score path (-> calibrate) and the audit path (-> review). Like every signal it is uncalibrated: a verifier failing is strong evidence, not a probability, so a [`Calibrator`](_autosummary/ek.base.html.md#ek.base.Calibrator) still runs before any gate. ### Example ```pycon >>> v = VerifierSignal([checksum_validator("luhn")]) >>> v("79927398713") # valid -> all checks pass 1.0 >>> v("79927398710") # invalid -> the one check fails 0.0 ``` #### findings(value, , spec=None) All findings the validators produce for `value` (empty == all clear). * **Return type:** [`List`](https://docs.python.org/3/library/typing.html#typing.List)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)] ### ek.app_folder() The root `ek` data folder (e.g. `~/.local/share/ek`). * **Return type:** [`Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path) ### ek.benford_findings(numbers, , field='', tol=0.15, min_n=30, layer='anomaly') FLAG a numeric field whose first-digit distribution deviates from Benford’s law. A reference-free anomaly check for naturally-occurring magnitudes (amounts, populations, counts): the first significant digit should follow `P(d)=log10(1+1/d)`. A large deviation flags fabricated or systematically-wrong data. **Corpus-level** (takes the whole column of values) and **FLAG-only** – it routes a field to review, never auto-edits, and can false-positive on legitimately bounded/skewed fields. Skipped (returns `[]`) below `min_n` usable values, since the law is asymptotic. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list) ### ek.cache_this(func=None, , cache=None, key=None, pre_cache=False, as_property=None, ignore=None, serialize=None, deserialize=None) Unified caching decorator for properties and methods with persistent storage support. `cache_this` extends the capabilities of Python’s built-in `functools.cached_property` and `functools.lru_cache` by providing: - **Persistent caching**: Store cached values in files, databases, or any MutableMapping - **Flexible cache backends**: Use instance attributes, external stores, or cache factories - **Smart key generation**: Automatic argument-based keys for methods with parameter filtering - **Serialization support**: Custom serialize/deserialize functions for complex data - **Auto-detection**: Automatically chooses property vs method caching based on signature - **No LRU eviction**: Unlike lru_cache, values persist until explicitly removed Unlike functools.cached_property (properties only) and lru_cache (memory-only with eviction), cache_this provides a unified interface for both use cases with persistent storage options. * **Parameters:** * **func** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)]) – The function to be decorated (usually left empty). * **cache** (`Union`[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`VT`)], [`None`](https://docs.python.org/3/builtins/constants.html#None)]) – The cache storage. Can be: - A MutableMapping instance (shared across instances) - A string naming an instance attribute containing a MutableMapping - A callable taking (instance) and returning a MutableMapping This enables instance-specific caching, e.g.: cache=lambda self: Files(f’/cache/{self.user_id}/’) * **key** (`Union`[[`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`), [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`TypeVar`](https://docs.python.org/3/library/typing.html#typing.TypeVar)(`KT`)], [`None`](https://docs.python.org/3/builtins/constants.html#None)]) – For properties: the key to store the cache value, can be a callable that will be applied to the method name to make a key, or an explicit string. For methods: a callable that takes `(self, *args, **kwargs)` and returns a cache key. * **pre_cache** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool) | [`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)) – Default is False. If True, adds an in-memory cache to the method to (also) cache the results in memory. If a MutableMapping is given, it will be used as the pre-cache. This is useful when you want a persistent cache but also want to speed up access to the method in the same session. * **as_property** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – If True, force use of CachedProperty. If False, force use of CachedMethod. If None (default), auto-detect based on function signature. * **ignore** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Parameter name(s) to exclude from cache key computation. Can be a string (single parameter) or list of strings (multiple parameters). Commonly used to ignore ‘self’ or parameters like ‘verbose’ that don’t affect the result. * **serialize** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Optional function to serialize values before caching (e.g. `pickle.dumps` for binary file storage). * **deserialize** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Optional function to deserialize cached values (e.g. `pickle.loads`). * **Returns:** The decorated function. ### Comprehensive Example Here’s a complete example showcasing all major features of cache_this: ```pycon >>> import tempfile >>> import os >>> from pathlib import Path >>> >>> class DataProcessor: ... def __init__(self, user_id="user123"): ... self.user_id = user_id ... self.memory_cache = {} # In-memory cache ... self.call_counts = {} # Track function calls for demo ... ... # 1. Basic property caching (like functools.cached_property) ... @cache_this ... def basic_property(self): ... '''Cached in instance.__dict__ by default''' ... self.call_counts['basic_property'] = self.call_counts.get('basic_property', 0) + 1 ... return f"computed_value_{self.call_counts['basic_property']}" ... ... # 2. Property with custom cache and key ... @cache_this(cache='memory_cache', key='custom_prop_key') ... def custom_cached_property(self): ... '''Cached in instance.memory_cache with custom key''' ... self.call_counts['custom_cached_property'] = self.call_counts.get('custom_cached_property', 0) + 1 ... return f"custom_value_{self.call_counts['custom_cached_property']}" ... ... # 3. Method caching with argument-based keys ... @cache_this(cache='memory_cache') ... def compute_result(self, x, y, mode='fast'): ... '''Cached based on arguments (x, y, mode)''' ... key = ('compute_result', x, y, mode) ... self.call_counts[key] = self.call_counts.get(key, 0) + 1 ... return x * y * (2 if mode == 'fast' else 3) ... ... # 4. Method caching with ignored parameters ... @cache_this(cache='memory_cache', ignore={'verbose', 'debug'}) ... def process_data(self, data, algorithm='default', verbose=False, debug=False): ... '''Cache ignores verbose and debug parameters''' ... key = ('process_data', tuple(data), algorithm) ... self.call_counts[key] = self.call_counts.get(key, 0) + 1 ... if verbose: print(f"Processing {data} with {algorithm}") ... return sum(data) * (2 if algorithm == 'default' else 3) ... ... # 5. Instance-specific cache factory ... @cache_this(cache=lambda self: {f'{self.user_id}_cache': {}}.get(f'{self.user_id}_cache')) ... def user_specific_computation(self, value): ... '''Each instance gets its own cache based on user_id''' ... key = ('user_specific_computation', value) ... self.call_counts[key] = self.call_counts.get(key, 0) + 1 ... return value ** 2 ``` Now let’s test all the features: ```pycon >>> processor = DataProcessor("alice") >>> >>> # Test basic property caching >>> result1 = processor.basic_property >>> result2 = processor.basic_property # Should use cache >>> assert result1 == result2 == "computed_value_1" >>> assert 'basic_property' in processor.__dict__ # Cached in instance dict >>> >>> # Test custom cache and key >>> result1 = processor.custom_cached_property >>> result2 = processor.custom_cached_property # Should use cache >>> assert result1 == result2 == "custom_value_1" >>> assert 'custom_prop_key' in processor.memory_cache >>> >>> # Test method caching with arguments >>> result1 = processor.compute_result(3, 4, 'fast') >>> result2 = processor.compute_result(3, 4, 'fast') # Should use cache >>> result3 = processor.compute_result(3, 4, 'slow') # Different args, new computation >>> assert result1 == result2 == 24 # 3 * 4 * 2 >>> assert result3 == 36 # 3 * 4 * 3 >>> >>> # Test parameter ignoring >>> result1 = processor.process_data([1, 2, 3], verbose=True) Processing [1, 2, 3] with default >>> result2 = processor.process_data([1, 2, 3], verbose=False) # Should use same cache >>> result3 = processor.process_data([1, 2, 3], debug=True) # Should use same cache >>> assert result1 == result2 == result3 == 12 # sum([1,2,3]) * 2 >>> >>> # Test instance-specific caching >>> result1 = processor.user_specific_computation(5) >>> result2 = processor.user_specific_computation(5) # Should use cache >>> assert result1 == result2 == 25 # 5 ** 2 >>> >>> # Different instance should have separate cache >>> processor2 = DataProcessor("bob") >>> result3 = processor2.user_specific_computation(5) # Fresh computation >>> assert result3 == 25 ``` Used with no arguments, `cache_this` will cache just as the builtin `cached_property` does – in the instance’s `__dict__` attribute. ```pycon >>> class SameAsCachedProperty: ... @cache_this ... def foo(self): ... print("In SameAsCachedProperty.foo...") ... return 42 ... >>> obj = SameAsCachedProperty() >>> obj.__dict__ # the cache is empty {} >>> obj.foo # when we access foo, it's computed and returned... In SameAsCachedProperty.foo... 42 >>> obj.__dict__ # ... but also cached {'foo': 42} >>> obj.foo # so that the next time we access foo, it's returned from the cache. 42 ``` Not that if you specify `cache=False`, you get a property that is computed every time it’s accessed: ```pycon >>> class NoCache: ... @cache_this(cache=False) ... def foo(self): ... print("In NoCache.foo...") ... return 42 ... >>> obj = NoCache() >>> obj.foo In NoCache.foo... 42 >>> obj.foo In NoCache.foo... 42 ``` Specify the cache as a dictionary that lives outside the instance: ```pycon >>> external_cache = {} >>> >>> class CacheWithExternalMapping: ... @cache_this(cache=external_cache) ... def foo(self): ... print("In CacheWithExternalMapping.foo...") ... return 42 ... >>> obj = CacheWithExternalMapping() >>> external_cache {} >>> obj.foo In CacheWithExternalMapping.foo... 42 >>> external_cache {'foo': 42} >>> obj.foo 42 ``` Specify the cache as an attribute of the instance, and an explicit key: ```pycon >>> class WithCacheInInstanceAttribute: ... ... def __init__(self): ... self.my_cache = {} ... ... @cache_this(cache='my_cache', key='key_for_foo') ... def foo(self): ... print("In WithCacheInInstanceAttribute.foo...") ... return 42 ... >>> obj = WithCacheInInstanceAttribute() >>> obj.my_cache {} >>> obj.foo In WithCacheInInstanceAttribute.foo... 42 >>> obj.my_cache {'key_for_foo': 42} >>> obj.foo 42 ``` Now let’s see a more involved example that exhibits how `cache_this` would be used in real life. Note two things in the example below. First, that we use `functools.partial` to fix the parameters of our `cache_this`. This enables us to reuse the same `cache_this` in multiple places without all the verbosity. We fix that the cache is the attribute `cache` of the instance, and that the key is a function that will be computed from the name of the method adding a `'.pkl'` extension to it. Secondly, we use the `ValueCodecs` from `dol` to provide a pickle codec for storying values. The backend store used here is a dictionary, so we don’t really need a codec to store values, but in real life you would use a persistent storage that would require a codec, such as files or a database. Thirdly, we’ll use a `pre_cache` to store the values in a different cache “before” (setting and getting) them in the main cache. This is useful, for instance, when you want to persist the values (in the main cache), but keep them in memory for faster access in the same session (the pre-cache, a dict() instance usually). It can also be used to store and use things locally (pre-cache) while sharing them with others by storing them in a remote store (main cache). Finally, we’ll use a dict that logs any setting and getting of values to show how the caches are being used. ```pycon >>> from dol import cache_this >>> >>> from functools import partial >>> from dol import ValueCodecs >>> from collections import UserDict >>> >>> >>> class LoggedCache(UserDict): ... name = 'cache' ... ... def __setitem__(self, key, value): ... print(f"In {self.name}: setting {key} to {value}") ... return super().__setitem__(key, value) ... ... def __getitem__(self, key): ... print(f"In {self.name}: getting value of {key}") ... return super().__getitem__(key) ... >>> >>> class CacheA(LoggedCache): ... name = 'CacheA' ... >>> >>> class CacheB(LoggedCache): ... name = 'CacheB' ... >>> >>> cache_with_pickle = partial( ... cache_this, ... cache='cache', # the cache can be found on the instance attribute `cache` ... key=lambda x: f"{x}.pkl", # the key is the method name with a '.pkl' extension ... pre_cache=CacheB(), ... ) >>> >>> >>> class PickleCached: ... def __init__(self, backend_store_factory=CacheA): ... # usually this would be a mapping interface to persistent storage: ... self._backend_store = backend_store_factory() ... self.cache = ValueCodecs.default.pickle(self._backend_store) ... ... @cache_with_pickle ... def foo(self): ... print("In PickleCached.foo...") ... return 42 ... ``` ```pycon >>> obj = PickleCached() >>> list(obj.cache) [] ``` ```pycon >>> obj.foo In CacheA: getting value of foo.pkl In CacheA: getting value of foo.pkl In PickleCached.foo... In CacheA: setting foo.pkl to b'\x80\x04K*.' 42 >>> obj.foo In CacheA: getting value of foo.pkl In CacheB: setting foo.pkl to 42 42 ``` As usual, it’s because the cache now holds something that has to do with `foo`: ```pycon >>> list(obj.cache) ['foo.pkl'] >>> # == ['foo.pkl'] ``` The value of `'foo.pkl'` is indeed `42`: ```pycon >>> obj.cache['foo.pkl'] In CacheA: getting value of foo.pkl 42 ``` But note that the actual way it’s stored in the `_backend_store` is as pickle bytes: ```pycon >>> obj._backend_store['foo.pkl'] In CacheA: getting value of foo.pkl b'\x80\x04K*.' >>> # == b'\x80\x04K*.' ``` ### ek.canonicalize_corrector(normalize, , field_name='', layer='canonicalize') L0: fold a value to canonical form, emitting a CORRECT finding when it changes. `normalize` is anything [`ek.canonicalize.resolve_canonicalizer()`](_autosummary/ek.canonicalize.html.md#ek.canonicalize.resolve_canonicalizer) accepts (a registered name, a callable, a step list). The narrowest, safest corrector – run it first so later layers compare canonical forms. * **Return type:** [`Corrector`](_autosummary/ek.validate.html.md#ek.validate.Corrector) ### ek.check_requirements(, engine=None, extra=None) Report what an OCR engine or feature extra needs, without raising. For an OCR `engine` and an installed `ocracy`, defers to `ocracy`’s own doctor (which knows each backend’s pip package, system binaries and credentials). Otherwise returns a generic hint. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) * **Returns:** A dict with at least `{"ok": bool, "hint": str}`. ### ek.checksum_validator(kind='luhn', , field_name='', layer='checksum') A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a value failing a named checksum. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)]] ### ek.cohen_kappa(rater_a, rater_b) Cohen’s kappa for two raters on nominal labels (chance-corrected agreement). Use only for exactly two raters, nominal labels, complete data; prefer [`krippendorff_alpha()`](_autosummary/ek.html.md#ek.krippendorff_alpha) for the general case (missing data, any measurement level). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.cross_field_validator(predicate, , message, fields=(), layer='cross_field') FLAG a record when `predicate(record)` is False (a general cross-field check). Skips silently when `predicate` raises on a partial record (a missing field is not itself a cross-field violation). * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)]] ### ek.default_canonicalizer() A conservative default: NFC -> casefold -> collapse whitespace. * **Return type:** [`Canonicalizer`](_autosummary/ek.canonicalize.html.md#ek.canonicalize.Canonicalizer) ### ek.default_cost_weight(grammar, ref) The default `CostWeight`: read the importance weight off the schema. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.engine_yields_tables(engine) Whether `engine`’s capability profile claims scoreable table structure. A convenience over [`ek.ocr.profile()`](_autosummary/ek.ocr.html.md#ek.ocr.profile): returns the engine profile’s `tables` flag. ek treats the profile as a *prior* (Azure DI, AWS Textract, PaddleOCR and the markdown engines are expected to yield scoreable tables; Google Vision is not), to be verified empirically by actually running [`table_from_ocr()`](_autosummary/ek.html.md#ek.table_from_ocr). Use it to skip table scoring for engines that cannot emit cells in the first place. * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.enum_validator(members, , field_name='', layer='enum') A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a value not in an allowed set. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)]] ### ek.estimate_quality(extraction, , sources=(), signals=(), calibrator=None, validators=(), policy=None, agreement=True, assume_calibrated=False) Estimate the quality of an extraction with no gold reference. Composes the strict `signal -> calibrate -> validate -> decide` pipeline (`misc/docs/ek_03`) over a single value, a [`FieldEstimate`](_autosummary/ek.base.html.md#ek.base.FieldEstimate), or a whole [`AnnotatedExtraction`](_autosummary/ek.base.html.md#ek.base.AnnotatedExtraction) (scored per field, with per-field specs feeding the validators and the node type as the Mondrian group key). The simple call `estimate_quality(value)` Just Works; every stage is injectable. * **Parameters:** * **extraction** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – A raw value, a [`FieldEstimate`](_autosummary/ek.base.html.md#ek.base.FieldEstimate), or an [`AnnotatedExtraction`](_autosummary/ek.base.html.md#ek.base.AnnotatedExtraction). * **sources** ([`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)) – Additional hypotheses of the *same* content (strings or `OcrResult`-shaped objects) to fuse with ROVER – their mean agreement becomes a `raw_signals["agreement"]` entry (the flagship online signal). * **signals** ([`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)) – Explicit [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal) callables `target -> float | Mapping` producing further raw signals. * **calibrator** – A [`Calibrator`](_autosummary/ek.base.html.md#ek.base.Calibrator) mapping raw score -> probability, applied to the field’s confidence (else the mean of its raw signals). When the signals are *heterogeneous* (e.g. an LLM-judge score next to a logprob), pass a `Mapping[str, Calibrator]` keyed by signal name (a `"*"` entry is the fallback) or a [`SignalFamilyCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.SignalFamilyCalibrator): each signal family is then calibrated on its own and the calibrated probabilities are fused by its `combine` rule (mean by default) followed by its learned `fusion` calibrator. A field’s own `confidence` is the `"confidence"` family. The per-family values are reported in `provenance["calibrated_signals"]`. **Calibration is non-optional before gating** (Hard Rule 1): passing a `policy` with no calibrator and `assume_calibrated` false **raises** – a raw, uncalibrated score must never reach a DecisionPolicy. * **validators** ([`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)) – [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) callables yielding findings. A flat iterable runs on every field (use spec-driven validators like [`schema_validator()`](_autosummary/ek.qe.verifiers.html.md#ek.qe.verifiers.schema_validator)). For per-field scoping pass a `Mapping` keyed by field name or node type (`"*"` runs on all). * **policy** – A [`DecisionPolicy`](_autosummary/ek.base.html.md#ek.base.DecisionPolicy) producing accept/flag/block from the *calibrated* confidence. * **agreement** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Auto-run ROVER over `sources` when any are given (default true). * **assume_calibrated** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Treat the incoming confidence as already calibrated (silences the uncalibrated-gating warning). With a per-family calibrator it also waives the check that a value fused from two or more families went through a learned `fusion` calibrator. * **Return type:** [`QualityReport`](_autosummary/ek.base.html.md#ek.base.QualityReport) ### ek.evaluate(cases, , metric=None, grammar=None, normalize=None, weights=None) Aggregate many comparisons into a [`Report`](_autosummary/ek.base.html.md#ek.base.Report) (corpus level). * **Parameters:** * **cases** ([`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)) – Iterable of `(pred, gold)` or `(pred, gold, slice_label)` tuples. * **metric** (`Union`[[`None`](https://docs.python.org/3/builtins/constants.html#None), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric)]) – As in [`score()`](_autosummary/ek.html.md#ek.score) (resolved once from the first case’s types). * **grammar** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar)]) – Optional Layer-A grammar passed to every comparison. * **normalize** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Optional canonicalizer applied before every comparison. * **Return type:** [`Report`](_autosummary/ek.base.html.md#ek.base.Report) * **Returns:** A Report whose `aggregate` is computed by the metric’s own aggregator (e.g. globally accumulated CER/WER, micro-F1) – never a naive mean. ### ek.evaluate_store(predict, gold, , metric=None, grammar=None, normalize=None, weights=None, input_key='input', reference_key='reference', slice_key='slice', persist=False, run_id=None, rootdir=None) Run `predict` over a gold store and score it per slice. * **Parameters:** * **predict** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – `input -> prediction` callable (the system under test). * **gold** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)) – Mapping `key -> {input, reference, [slice], ...}` (a dict or a `ek` gold store). * **metric** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – Metric name/callable (defaults to type-dispatch on the first case). * **grammar/normalize/weights** – forwarded to scoring. * **input_key/reference_key/slice_key** – field names within each gold record. * **persist/run_id/rootdir** – persist the run to the `results`/`runs` stores. * **Returns:** A [`Report`](_autosummary/ek.base.html.md#ek.base.Report); `detail['per_item']` maps each key to its prediction, reference, slice, and score. ### ek.expected_calibration_error(probs, correct, , n_bins=10) Expected Calibration Error: weighted mean gap between confidence and accuracy. Bins predictions by confidence into `n_bins` equal-width bins and averages `|mean_confidence - accuracy|` weighted by bin population. `0` is perfect. Non-finite probs are skipped; out-of-range probs are clamped into `[0, 1]`. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.get(namespace, name) Resolve a registered strategy by name (discovering entry points first). * **Raises:** [**KeyError**](https://docs.python.org/3/builtins/exceptions.html#KeyError) – with the list of available names, if `name` is unknown. * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) ### ek.has_table_structure(ocr_result, , parser=None) Whether [`table_from_ocr()`](_autosummary/ek.html.md#ek.table_from_ocr) recovers a non-empty table from `ocr_result`. A thin predicate over [`table_from_ocr()`](_autosummary/ek.html.md#ek.table_from_ocr) (same `parser` semantics): true iff a table with at least one cell can be recovered. Use it to route an OCR result to the table metrics vs the text-only CER/WER fallback. * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.iban_check(iban) IBAN validity via the ISO 13616 / ISO 7064 mod-97 rule (must equal 1). * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.isbn_check(value) Validate an ISBN-10 or ISBN-13 by length (digits/hyphens/spaces ignored). * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.json_store(kind, , rootdir=None) A JSON `MutableMapping` for one `kind` of artifact. Keys are plain names (no extension); values are JSON-serializable objects. * **Return type:** [`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping) ### ek.krippendorff_alpha(reliability_data, , level='nominal') Krippendorff’s alpha – the general inter-annotator agreement coefficient. Pure-Python (no dependency, permissive core): any number of raters, any measurement `level` (`"nominal"`, `"ordinal"`, `"interval"`, `"ratio"`), and missing data (use `None` or `float('nan')` for an unlabelled cell). `reliability_data` is one row per rater, one column per item. Alpha is `1.0` for perfect agreement, `0.0` at chance, and goes negative for systematic disagreement. Computed via the coincidence-matrix method (Krippendorff 2011): observed vs expected disagreement under the level-specific difference metric. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### Example ```pycon >>> data = [[1, 1, 1, 1], [1, 1, 1, 1]] # two raters, perfect agreement >>> krippendorff_alpha(data) 1.0 ``` ### ek.lexicon_corrector(vocabulary, , threshold=0.8, scorer=None, field_name='', layer='lexicon', flag_unmatched=True) L2: resolve a value against a **closed** vocabulary by fuzzy match (`rapidfuzz`). The safest corrector in the stack when the candidate set is closed (country codes, SKUs, enums): an exact member passes untouched; a single close match (similarity `>= threshold`) is applied as a CORRECT finding; otherwise the value is FLAGged as out-of-vocabulary (when `flag_unmatched`). Against an *open* vocabulary, set `flag_unmatched=False` and treat matches as suggestions only. * **Parameters:** * **vocabulary** ([`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – The closed set of valid values. * **threshold** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Minimum `rapidfuzz` similarity in `[0, 1]` to auto-correct. * **scorer** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – A `rapidfuzz` scorer (default `fuzz.ratio`). * **flag_unmatched** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – FLAG a value with no close match (default `True`). * **Return type:** [`Corrector`](_autosummary/ek.validate.html.md#ek.validate.Corrector) ### ek.llm_corrector(correct_fn, , only_flagged=True, field_name='', layer='llm_correct') L5: a **gated** neural/LLM corrector – the only layer that invents content. `correct_fn` is an **injected** `(value) -> Optional[str]` (bring your own LLM/seq2seq call; return a corrected string, or `None` to leave the value unchanged). It is the most expensive and stochastic layer, so by default (`only_flagged`) it fires \*\*only on a value that a cheaper layer already FLAGged\*\* – gate it to the residual, never let it silently rewrite clean fields (this layer reads the pipeline’s accumulated findings via `wants_findings`). Emits a CORRECT finding; the pipeline applies the rewrite and keeps the original for audit. Always verify its output downstream – it can make text worse. Recipe (Anthropic): `correct_fn=lambda v: client.messages.create(model=..., messages=[{"role": "user", "content": prompt(v)}]).content[0].text.strip()` with `client = anthropic.Anthropic()` (MIT). Constrain/verify the result and keep an audit trail. * **Return type:** [`Corrector`](_autosummary/ek.validate.html.md#ek.validate.Corrector) ### ek.lm_surprisal_validator(scorer, , threshold, higher_is_worse=True, field_name='', layer='lm_prior') L3: FLAG a value whose language-model surprisal crosses `threshold`. `scorer` is an **injected** `(str) -> float` returning a surprisal / perplexity / negative-log-likelihood (dependency injection: bring your own in-domain n-gram or masked-LM scorer – an *in-domain* prior is what makes this useful). FLAG-only on its own; pair it with a candidate generator (`lexicon_corrector` / `llm_corrector`) to actually correct. Set `higher_is_worse=False` if your scorer returns a probability (higher = better). Recipe (masked-LM pseudo-log-likelihood): wrap `minicons` – `from minicons import scorer; m = scorer.MaskedLMScorer("bert-base-uncased", "cpu")` – and pass `scorer=lambda s: -m.sequence_score([s])[0]` (surprisal = negative PLL). `minicons` is MIT; install your own LM backend (it pulls torch). * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)]] ### ek.load_baseline(name, , rootdir=None) Load a named baseline (or `None` if it does not exist). * **Return type:** [`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)] ### ek.load_calibrator(name, , rootdir=None) Reconstruct a persisted calibrator by name (dispatched on its `kind`). Validates the stored record so a malformed or unknown `kind` fails with an actionable error rather than a raw `KeyError` or a load-then-crash-later. * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) ### ek.luhn_check(number) Luhn (mod-10) checksum: credit cards, IMEIs, some national IDs. * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.mall(, rootdir=None) The full stores-of-stores mall (one JSON store per `KINDS`). * **Return type:** [`Mall`](_autosummary/ek.stores.html.md#ek.stores.Mall) ### ek.mean_probability(probs) Fuse calibrated per-family probabilities by their mean (the default combiner). Each family’s calibrated value estimates the *same* event (“this field is correct”), so the mean is a defensible, assumption-light pooling rule. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ```pycon >>> mean_probability({"judge": 0.9, "logprob": 0.7}) 0.8 ``` ### ek.min_probability(probs) Fuse calibrated per-family probabilities by their minimum (conservative). Use when any one family flagging a problem should be enough to lower the gate. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ```pycon >>> min_probability({"judge": 0.9, "logprob": 0.7}) 0.7 ``` ### ek.names(namespace) All registered names in a namespace (after entry-point discovery). * **Return type:** [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] ### ek.ordering_validator(keys, , strict=True, layer='cross_field') FLAG a record whose comparable values at `keys` are not ascending. The canonical date/sequence check (`start <= end`, issue before due). Only the keys actually present and mutually comparable are checked, so it is safe on partial records. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)]] ### ek.pass_at_k(, n, c, k) Unbiased `pass@k`: probability that **at least one** of k samples succeeds. The HumanEval estimator `1 - C(n-c, k) / C(n, k)` over `n` trials with `c` successes. This is the **capability** metric – right when a single success is enough (offline candidate generation behind a verifier). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### Example ```pycon >>> round(pass_at_k(n=10, c=1, k=1), 3) 0.1 >>> pass_at_k(n=10, c=10, k=5) 1.0 ``` ### ek.pass_hat_k(, n, c, k) Unbiased `pass^k`: probability that **all k** independent trials succeed. The tau-bench estimator `C(c, k) / C(n, k)`; for a per-trial success probability `p` it decays to `p**k`. This is the **reliability** metric – the one that matters when consistency *is* the product. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### Example ```pycon >>> pass_hat_k(n=8, c=8, k=8) 1.0 >>> pass_hat_k(n=8, c=4, k=2) 0.21428571428571427 ``` ### ek.percent_agreement(rater_a, rater_b) Raw fraction of items two raters labelled identically. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.persistent_cache(func=None, , kind='runs', key=None, rootdir=None) Memoize a plain function’s result in a `kind` JSON store, across sessions. Use as `@persistent_cache(kind="runs", key=...)`. Returned values must be JSON-serializable. `key` may be a constant or a callable receiving the call arguments; it defaults to the function’s qualified name (so provide a `key` for functions that take arguments). For caching a *class property/method*, prefer [`cache_this`](_autosummary/ek.html.md#ek.cache_this) (dol’s descriptor) instead. ### Example ```pycon >>> import tempfile; root = tempfile.mkdtemp() >>> calls = [] >>> @persistent_cache(kind="runs", key="answer", rootdir=root) ... def compute(): ... calls.append(1) ... return {"answer": 42} >>> compute(), compute(), len(calls) ({'answer': 42}, {'answer': 42}, 1) ``` ### ek.range_validator(lo, hi, , field_name='', layer='range') A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a number outside `[lo, hi]`. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)]] ### ek.regex_validator(pattern, , field_name='', layer='format') A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a value not fully matching `pattern`. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)]] ### ek.register(namespace, name, obj=None) Register `obj` under `namespace`/`name` (usable as a decorator). * **Parameters:** * **namespace** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – A strategy family, e.g. `"metrics"`, `"signals"`, `"calibrators"`, `"policies"`, `"normalizers"`, `"ocr"`. * **name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The lookup key within the namespace. * **obj** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – The object to register; omit to use as a decorator. * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) * **Returns:** `obj` (so it can be used as a decorator). ### ek.regression_gate(report, baseline, , tolerance=0.0, higher_is_better=None, rootdir=None) Fail if `report` regresses beyond `tolerance` vs a baseline, per slice. * **Parameters:** * **report** – The current [`Report`](_autosummary/ek.base.html.md#ek.base.Report). * **baseline** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – A baseline name (loaded from the `baselines` store) or a baseline dict from [`save_baseline()`](_autosummary/ek.html.md#ek.save_baseline). * **tolerance** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Allowed drift before a change counts as a regression. * **higher_is_better** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`bool`](https://docs.python.org/3/builtins/functions.html#bool)]) – Override metric-direction inference (CER/WER/graph are lower-is-better; F1/similarity are higher-is-better). * **rootdir** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – data root (when `baseline` is a name). * **Return type:** [`GateResult`](_autosummary/ek.harness.html.md#ek.harness.GateResult) * **Returns:** A [`GateResult`](_autosummary/ek.html.md#ek.GateResult) (falsy if any regression was found). ### ek.reliability_curve(probs, correct, , n_bins=10) Per-bin `{confidence, accuracy, count}` for a reliability diagram. * **Return type:** [`List`](https://docs.python.org/3/library/typing.html#typing.List)[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)] ### ek.requires_extra(extra, , packages=None) Decorator: fail with an actionable install hint if an extra is missing. The wrapped callable runs only once every package in `packages` imports; the common case (everything installed) adds a single cheap import check. * **Parameters:** * **extra** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The extra name, used in the hint `pip install ek[]`. * **packages** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]]) – Import names to probe (defaults to `[extra]`). * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable) ### Example ```pycon >>> @requires_extra("ocr", packages=["definitely_not_installed_pkg"]) ... def run(): ... return "ran" >>> try: ... run() ... except MissingExtraError as e: ... print("ek[ocr]" in str(e)) True ``` ### ek.resolve(namespace, ref, , default=None) Coerce a strategy reference to a callable. `ref` may be a registered name (`str`), an already-resolved object, or `None` (in which case `default`, itself a name or object, is used). * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) ### ek.resolve_table_parser(ref) Resolve a table parser reference (a registered name or a callable) to a callable. Mirrors how other ek strategies resolve from the registry (see [`ek.registry.resolve()`](_autosummary/ek.registry.html.md#ek.registry.resolve)). A `str` is looked up under the `"table_parsers"` namespace; an already-callable parser is returned unchanged. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] ### ek.risk_coverage_curve(probs, correct, , thresholds=None) The risk-coverage trade-off: `{threshold, coverage, selective_risk}` per point. Coverage is the fraction auto-accepted at each threshold; selective risk is the error rate *among the accepted*. Lower-left is better; pick the operating point against a target risk or coverage. * **Return type:** [`List`](https://docs.python.org/3/library/typing.html#typing.List)[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)] ### ek.rover(hypotheses, , use_confidence=True, conf_weight=0.5, null_conf=0.7, tokenize=None, max_tokens=5000) Align N hypotheses, vote per slot, and emit consensus + per-position agreement. * **Parameters:** * **hypotheses** ([`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – The recognizer/extractor outputs to fuse. Each may be a string, an `OcrResult`-shaped object (`.text`/`.mean_confidence`), a list of token strings, or a list of `(token, confidence)` pairs. * **use_confidence** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Blend confidence into the vote (`True`) or vote purely by frequency (`False`, which forces `conf_weight` to `0`). * **conf_weight** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Weight on average confidence vs vote frequency in the slot score (Fiscus’s `1 - alpha`); `0` is frequency-only, `1` is confidence-only. Ignored when `use_confidence` is `False`. * **null_conf** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Confidence credited to a NULL vote (an engine that produced no token at a slot) – the lever for how readily a deletion wins. * **tokenize** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`List`](https://docs.python.org/3/library/typing.html#typing.List)[[`Tuple`](https://docs.python.org/3/library/typing.html#typing.Tuple)[[`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`float`](https://docs.python.org/3/builtins/functions.html#float)]]]]) – Optional `hypothesis -> [(token, confidence), ...]` override; by default `_as_units()` handles strings/OcrResults/token lists. * **max_tokens** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`int`](https://docs.python.org/3/builtins/functions.html#int)]) – Reject any hypothesis longer than this (the aligner is `O(N*l*L*L')`; an unbounded input is a quadratic time/memory DoS). `None` disables the guard. * **Return type:** [`RoverConsensus`](_autosummary/ek.qe.html.md#ek.qe.RoverConsensus) * **Returns:** A [`RoverConsensus`](_autosummary/ek.html.md#ek.RoverConsensus) with the consensus `tokens`/`text`, the per-slot breakdown, and the per-consensus-token `agreement` usable as a raw signal. ### ek.run_suite(agent, tasks, , k=1, check=None, metrics=None, price=None, prices=None, run=None, seed=None, persist=False, run_id=None, rootdir=None) Run `agent` over a task suite, `k` trials per task, and report reliability + cost. * **Parameters:** * **agent** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`TaskSpec`](_autosummary/ek.agents.base.html.md#ek.agents.base.TaskSpec)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – The system under test: `TaskSpec -> Episode` (a bare answer is wrapped). It receives the whole spec – `.input`, `.tools`, `.gold` – not just the input. If it accepts a keyword `seed`, the per-trial seed is passed to it. * **tasks** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – [`TaskSpec`](_autosummary/ek.agents.base.html.md#ek.agents.base.TaskSpec) s (or a `{task_id: spec}` mapping). * **k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Trials per task. `k > 1` is what makes `pass^k` meaningful – and it only means anything if the agent is genuinely stochastic (the report warns if it is not). * **check** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The success oracle – a registered checker name or an `(episode, gold) -> bool` callable. Defaults to `"output"`; **inject a state-based/executable oracle** for a real suite, and run it isolated from the agent. * **metrics** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)]) – Optional `{name: Metric}` scored on every episode against `task.gold`, with the **suite’s Layer-A grammar injected** – this is how the tool/argument cost weights reach a `tool_call`/`trajectory` metric. Results land in `report.detail["metrics"]`. * **price/prices** – Rates for costing the episodes (see [`ek.agents.cost`](_autosummary/ek.agents.cost.html.md#module-ek.agents.cost)). * **run** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`RunProvenance`](_autosummary/ek.agents.base.html.md#ek.agents.base.RunProvenance)]) – [`RunProvenance`](_autosummary/ek.agents.base.html.md#ek.agents.base.RunProvenance) for this run (model, simulator, suite version, scaffold). Recorded on every episode and on any baseline saved from it. * **seed** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`int`](https://docs.python.org/3/builtins/functions.html#int)]) – Base RNG seed. Trial *i* of each task uses `seed + i`: it is recorded on the episode’s provenance **and passed to the agent** when the agent accepts a `seed` keyword. * **persist/run_id/rootdir** – Persist the run to the `runs`/`results` stores. * **Return type:** [`ReliabilityReport`](_autosummary/ek.agents.html.md#ek.agents.ReliabilityReport) * **Returns:** A [`ReliabilityReport`](_autosummary/ek.agents.html.md#ek.agents.ReliabilityReport) carrying `pass@k`, `pass^k` (with a bootstrap CI), the success rate (with a Wilson CI), the per-slice cuts, and the cost report – because reliability without cost is only half the answer. ### ek.save_baseline(report, name, , rootdir=None) Freeze a report’s aggregate + per-slice scores as a named baseline. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### ek.save_calibrator(calibrator, name, , rootdir=None) Persist a fitted calibrator’s parameters to the `calibrators` store. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### ek.schema_validator(value, , spec=None) A spec-driven [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator): type and `domain` checks from a FieldSpec. Reads the [`FieldSpec`](_autosummary/ek.base.html.md#ek.base.FieldSpec) `type` and `domain` – a `(lo, hi)` numeric range, an enum tuple, or a regex string – and applies the matching check. With no spec it is a no-op, so it composes safely in any validator list. * **Return type:** [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)] ### ek.score(pred, gold, , grammar=None, metric=None, normalize=None, weights=None) Score one prediction against one gold reference (reference-based). * **Parameters:** * **pred** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The predicted output (string, record dict, or anything with `.text`). * **gold** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – The gold reference, same shape as `pred`. * **grammar** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar)]) – Optional Layer-A [`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar) (carries cost weights). * **metric** (`Union`[[`None`](https://docs.python.org/3/builtins/constants.html#None), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric)]) – A registered metric name (`"cer"`, `"wer"`, `"fields"`, …), a callable [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric), or `None` to dispatch by type. * **normalize** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Optional canonicalizer (name, callable, step list, or [`Canonicalizer`](_autosummary/ek.canonicalize.html.md#ek.canonicalize.Canonicalizer)) applied before comparison. * **weights** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – A [`CostWeight`](_autosummary/ek.base.html.md#ek.base.CostWeight) for cost-weighted metrics (the typed-graph distance); overrides the schema’s importance weights. * **Return type:** [`Score`](_autosummary/ek.base.html.md#ek.base.Score) * **Returns:** A [`Score`](_autosummary/ek.base.html.md#ek.base.Score). ### ek.split_conformal_quantile(scores, alpha) The split-conformal threshold: the `ceil((n+1)(1-alpha)) / n` empirical quantile. `scores` are nonconformity scores on an exchangeable calibration set. Returns `+inf` when `n` is too small to guarantee `1 - alpha` (so nothing is flagged – the honest behaviour at that sample size), and `-inf` at the `alpha -> 1` limit (flag everything). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.stop_on_correction(findings) Stop policy: end the pass as soon as any layer proposes a correction. * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.stop_on_flag(findings) Stop policy: end the pass at the first FLAG (fail fast). * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.table_from_ocr(ocr_result, , parser=None) Extract a normalized [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table) from an `OcrResult`. The single seam ek’s table metrics (TEDS / GriTS) use to score OCR table output uniformly. Returns the recovered Table, or `None` when the result carries no recoverable table structure. * **Parameters:** * **ocr_result** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Any `OcrResult`-shaped object. ek reads only its `.raw` (the engine-specific payload), `.markdown` (VLM/markdown engines), and `.meta` – never ocracy types, so any `image -> OcrResult` callable’s output works. * **parser** (`Union`[[`None`](https://docs.python.org/3/builtins/constants.html#None), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]]) – How to turn the raw payload into a table (dependency injection / open-closed). One of: - `None` (default): try the built-in **safe heuristics** – `.raw` is already a [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table), a 2-D grid, or a `
...
` HTML string; or `.raw`/`.meta` is a mapping carrying one of `{tables, table, cells, grid, html}`; or `.markdown` contains an HTML ``. Returns `None` if none match – it never guesses wildly. - a `str`: the name of a parser registered under `"table_parsers"` (per-engine extractor), resolved from [`ek.registry`](_autosummary/ek.registry.html.md#module-ek.registry). - a callable `(raw) -> Table | grid | html | None`: an inline per-engine extractor. It receives the result’s `.raw` (falling back to the result itself when there is no `.raw`). * **through** (*Any non-None value an injected parser returns is fed*) * **Return type:** [*Table*](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table) | *None* :param [`Table.coerce()`](_autosummary/ek.html.md#ek.Table.coerce): :param so a parser may yield a Table: :param a 2-D grid: :param or HTML.: * **Return type:** [`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table)] * **Returns:** A non-empty [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table), or `None` when no table structure is recoverable. An empty Table (no rows/cells) is normalized to `None` so the predicate [`has_table_structure()`](_autosummary/ek.html.md#ek.has_table_structure) and downstream metrics see “no table”, not a degenerate one. ### ek.totals_consistent(record, , total_key, item_keys, tol=0.01, layer='cross_field') Flag when `record[total_key]` does not equal the sum of `item_keys` (±tol). The canonical cross-field check (invoice total = Σ line items). Skipped (rather than raising or flagging) when the total or *all* item keys are absent or non-numeric, so it is safe on partial records. * **Return type:** [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)] ### ek.validation_pipeline(\*layers, apply_corrections=True, stop_when=) Compose validators/correctors into a cheapest -> most-expensive pipeline. `layers` run in the given order (the order encodes the cost spine). Each layer is a `(value, *, spec) -> Iterable[Finding]` callable. When `apply_corrections` (default), the first `CORRECT` finding a layer emits is applied to the value before the next layer runs – so a cheap deterministic fix is what the expensive layers then see (the noisy-channel chain). `stop_when` is an injectable early-exit policy over the findings accumulated so far (see [`stop_on_correction()`](_autosummary/ek.html.md#ek.stop_on_correction) / [`stop_on_flag()`](_autosummary/ek.html.md#ek.stop_on_flag)); the default runs every layer. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`ValidationResult`](_autosummary/ek.validate.html.md#ek.validate.ValidationResult)] ### ek.zscore_anomaly_findings(numbers, , field='', threshold=3.5, min_n=30, layer='anomaly') FLAG numeric values that are robust-z-score outliers (median + MAD based). A reference-free, **dependency-free** anomaly check that complements [`benford_findings()`](_autosummary/ek.html.md#ek.benford_findings) (which checks the leading-digit *distribution*; this checks individual *magnitudes*). For a numeric column it computes each value’s **modified z-score** – `0.6745 * (x - median) / MAD` – which is robust to the very outliers it is looking for, and FLAGs those whose absolute score exceeds `threshold`. One FLAG per outlying value (carrying its index); **FLAG-only** – it routes a value to review, never edits. Skipped below `min_n` (robust statistics on a handful of points are noise). When the MAD is 0 (a *near*-constant column, e.g. many identical values plus one outlier) it falls back to the mean-absolute-deviation scale (Iglewicz & Hoaglin), so a lone outlier is still caught; a *truly* constant column (no spread at all) yields no findings. For *multivariate* outliers, plug an isolation forest via `pyod` (already in `ek[validation]`) as a custom validator. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list) ### Modules | [`ocr`](_autosummary/ek.ocr.html.md#module-ek.ocr) | OCR evaluation -- `ek`'s first concrete instance, built on `ocracy`. | |--------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`agents`](_autosummary/ek.agents.html.md#module-ek.agents) | `ek.agents` -- evaluating AI agents and assistants, in cost per successful task. | | [`metrics`](_autosummary/ek.metrics.html.md#module-ek.metrics) | Reference-based (offline) metrics: the `score()` side of `ek`. | | [`canonicalize`](_autosummary/ek.canonicalize.html.md#module-ek.canonicalize) | Versioned, composable canonicalization applied *before* scoring. | | [`qe`](_autosummary/ek.qe.html.md#module-ek.qe) | Reference-free quality estimation: the `estimate_quality()` side of `ek`. | | [`base`](_autosummary/ek.base.html.md#module-ek.base) | Foundational data structures and strategy protocols for `ek`. | | [`facade`](_autosummary/ek.facade.html.md#module-ek.facade) | The two top-level facades: [`score()`](_autosummary/ek.html.md#ek.score) (offline) and [`estimate_quality()`](_autosummary/ek.html.md#ek.estimate_quality) (online). | | [`harness`](_autosummary/ek.harness.html.md#module-ek.harness) | The offline evaluation harness: run a predictor over gold, gate regressions. | | [`registry`](_autosummary/ek.registry.html.md#module-ek.registry) | Strategy registry, optional-dependency guarding, and requirement checks. | | [`stores`](_autosummary/ek.stores.html.md#module-ek.stores) | Local-file persistence for `ek`, as plain `MutableMapping` stores. | | [`tools`](_autosummary/ek.tools.html.md#module-ek.tools) | Small, CLI-friendly functions over the `ek` core (the dispatch surface). | | [`validate`](_autosummary/ek.validate.html.md#module-ek.validate) | Reference-free post-extraction validation & correction (issue #7). | # _autosummary/ek.metrics.anls.html.md # ek.metrics.anls ANLS / ANLS\* (nested-JSON) similarity, wrapping `anls_star`. ANLS (Average Normalized Levenshtein Similarity) is the DocVQA-family metric for a short answer against acceptable variants: a normalized edit similarity thresholded at ~0.5 so minor OCR/spelling differences do not zero a correct answer. \*\*ANLS\*\*\* (Peer et al., 2024, [arXiv 2402.03848](https://arxiv.org/abs/2402.03848)) generalizes it to *arbitrarily nested* JSON – strings, tuples (best match), lists (Hungarian matching, penalizing missing/hallucinated items), and dicts (key-value with penalties for missing or hallucinated keys) – by mapping both sides to a tree and comparing them. It is the closest thing to an off-the-shelf nested-JSON IE metric (`misc/docs/ek_02` §3.2). This wraps the `anls_star` package (Apache-2.0; its only dependency is `munkres` for the Hungarian matching, and it carries its own pure-Python Levenshtein – no GPL `Levenshtein` is pulled in). It ships behind the `[metrics]` extra and is imported lazily, so importing `ek` never requires it. Because `anls_star` already returns a `[0, 1]` higher-is-better similarity, the wrapper is thin: it flips `ek`’s `(pred, gold)` argument order to the library’s `(gt, pred)` order, records the raw similarity in `detail`, and – for a corpus – [`AnlsMetric.aggregate()`](_autosummary/ek.metrics.anls.html.md#ek.metrics.anls.AnlsMetric.aggregate) returns the *mean* ANLS (the “A” in ANLS: the score is already length-normalized per item, so the corpus statistic is the average of per-item similarities, unlike CER/WER which accumulate edit counts). ### Example ```pycon >>> from ek.metrics.anls import AnlsMetric >>> m = AnlsMetric() >>> round(m("Hello Wrld", "Hello World").value, 3) # one-char OCR slip 0.909 >>> m({"name": "Acme", "city": "Paris"}, ... {"name": "Acme", "city": "Paris"}).value # nested, exact 1.0 ``` ### Classes | [`AnlsMetric`](_autosummary/ek.metrics.anls.html.md#ek.metrics.anls.AnlsMetric)(\*[, threshold]) | ANLS / ANLS\* (nested-JSON) similarity as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric). | |--------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------| ### *class* ek.metrics.anls.AnlsMetric(, threshold=0.5) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) ANLS / ANLS\* (nested-JSON) similarity as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric). Handles a bare string (classic ANLS) or an arbitrarily nested dict/list/tuple/`None` structure (ANLS\*); the backend dispatches on type. * **Parameters:** **threshold** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – ANLS zeroes a per-leaf similarity below this value before averaging (the classic ANLS 0.5 cut tolerates minor OCR/spelling noise). `anls_star` exposes no parameter for this – it reads a class constant `ANLSTree.THRESHOLD` (default `0.5`) – so the metric applies the threshold by setting that constant around the call and restoring it. **Caveat:** this mutates a process-global, so it is not thread-safe; concurrent `AnlsMetric` calls with different thresholds in the same process can interfere. Defaults to `0.5`. #### aggregate(scores) Corpus ANLS = mean of per-item ANLS (each is already length-normalized). Unlike CER/WER (which accumulate global edit counts) or field-F1 (micro-TP/ FP/FN), ANLS is defined as the *average* normalized similarity over the evaluation set – the “A” in ANLS – so the corpus statistic is the plain mean of the per-item scores. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) # _autosummary/ek.metrics.fields.html.md # ek.metrics.fields Field-level metric for record / dict-shaped extractions. Compares two flat records key by key into precision / recall / F1, with a wrong value counting as both a spurious prediction and a missed gold (the conventional slot-error accounting). It is dependency-free, so it ships in core for the common “compare these two extracted records” case; the tag-sequence span metrics (`seqeval` / `nervaluate`, with their *explicit* match schemes) and nested metrics (ANLS\*) arrive behind the `[metrics]` extra – see `misc/docs/ek_02`. Aggregation is micro-averaged over the corpus (sum TP/FP/FN, then divide), carried in each [`Score`](_autosummary/ek.base.html.md#ek.base.Score)’s `detail` so [`evaluate()`](_autosummary/ek.facade.html.md#ek.facade.evaluate) does it correctly rather than averaging per-record F1s. ### Classes | [`FieldMetric`](_autosummary/ek.metrics.fields.html.md#ek.metrics.fields.FieldMetric)(\*[, canonicalizer]) | Per-field precision/recall/F1 for two dict records. | |-------------------------------------------------------------------------------------|-------------------------------------------------------| ### *class* ek.metrics.fields.FieldMetric(, canonicalizer=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Per-field precision/recall/F1 for two dict records. * **Parameters:** **canonicalizer** – Optional `str -> str` applied to string values before comparison (e.g. casefold/whitespace folding). #### aggregate(scores) Micro-averaged F1 over the corpus (sum TP/FP/FN, then divide). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) # _autosummary/ek.metrics.graphs.html.md # ek.metrics.graphs The flagship metric: cost-weighted, type-aware distance over a typed graph. No off-the-shelf library takes a typed schema + per-field/per-type cost weights and returns a weighted distance over the extracted graph – it is *the* must-build of this project (see `misc/docs/ek_02` and `misc/docs/ek_06`). The mechanism is a graph edit distance (GED) whose per-edit costs are supplied by `ek`: relabelling a node, deleting it, or changing a field each costs an amount read from the Layer-A [`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar) importance weights. That is how “two extra digits on a monetary amount” outweighs “a misspelled city” – no hardcoded costs, open-closed via the schema. GED is NP-hard; this uses `networkx.graph_edit_distance` with a configurable `timeout` (it returns the best edit cost found within the budget – an *approximation* on large graphs), normalizes the distance by the maximum possible edit cost (delete-all-gold + insert-all-pred), and reports both the normalized distance (`Score.value`; lower is better) and the derived similarity. `networkx` ships in the `[metrics]` extra and is imported lazily, so importing `ek` never pulls it in. ### Example ```pycon >>> from ek import GraphGrammar, NodeType, FieldSpec, TypedGraph, TypedNode >>> g = GraphGrammar(node_types={"donation": NodeType("donation", fields={ ... "amount": FieldSpec("amount", "number", importance=10.0), ... "city": FieldSpec("city", "string", importance=1.0)})}) >>> gold = TypedGraph([TypedNode("d1", "donation", {"amount": "100", "city": "Paris"})]) >>> # a wrong amount costs far more than a wrong city, per the schema weights: >>> wrong_amount = TypedGraph([TypedNode("d1", "donation", {"amount": "900", "city": "Paris"})]) >>> wrong_city = TypedGraph([TypedNode("d1", "donation", {"amount": "100", "city": "Lyon"})]) >>> m = TypedGraphMetric(grammar=g) >>> m(wrong_amount, gold).detail["raw_distance"] 10.0 >>> m(wrong_city, gold).detail["raw_distance"] 1.0 ``` ### Module Attributes | [`DEFAULT_MAX_NODES`](_autosummary/ek.metrics.graphs.html.md#ek.metrics.graphs.DEFAULT_MAX_NODES) | graph edit distance is NP-hard, so a larger pair is rejected (raise) rather than silently spinning. | |----------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------| ### Classes | [`TypedEdge`](_autosummary/ek.metrics.graphs.html.md#ek.metrics.graphs.TypedEdge)(src, dst[, edge_type]) | A directed, typed relation between two node ids. | |--------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------| | [`TypedGraph`](_autosummary/ek.metrics.graphs.html.md#ek.metrics.graphs.TypedGraph)([nodes, edges]) | A typed graph: typed nodes plus directed typed edges between them. | | [`TypedGraphMetric`](_autosummary/ek.metrics.graphs.html.md#ek.metrics.graphs.TypedGraphMetric)([grammar, weights, ...]) | Cost-weighted, type-aware typed-graph edit distance as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric). | | [`TypedNode`](_autosummary/ek.metrics.graphs.html.md#ek.metrics.graphs.TypedNode)(node_id, node_type[, fields]) | A node in a typed graph: an id, a node-type tag, and its field values. | ### ek.metrics.graphs.DEFAULT_MAX_NODES *= 60* graph edit distance is NP-hard, so a larger pair is rejected (raise) rather than silently spinning. A keyword override on [`TypedGraphMetric`](_autosummary/ek.metrics.graphs.html.md#ek.metrics.graphs.TypedGraphMetric) lifts it for callers who accept the cost. * **Type:** Default ceiling on per-graph node count ### *class* ek.metrics.graphs.TypedEdge(src, dst, edge_type='') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A directed, typed relation between two node ids. ### *class* ek.metrics.graphs.TypedGraph(nodes=(), edges=()) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A typed graph: typed nodes plus directed typed edges between them. #### to_networkx() Build a `networkx.DiGraph` carrying type/fields as node/edge attrs. ### *class* ek.metrics.graphs.TypedGraphMetric(grammar=None, weights=None, , canonicalizer=None, timeout=10.0, max_nodes=60) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Cost-weighted, type-aware typed-graph edit distance as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric). * **Parameters:** * **grammar** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar)]) – Layer-A [`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar) supplying cost weights (may also be passed per-call to `__call__`). * **weights** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar), [`TypeRef`](_autosummary/ek.base.html.md#ek.base.TypeRef)], [`float`](https://docs.python.org/3/builtins/functions.html#float)]]) – Optional [`CostWeight`](_autosummary/ek.base.html.md#ek.base.CostWeight) overriding the schema weights. * **canonicalizer** – Optional `str -> str` applied to field values before the equality check (the facade passes `normalize=` through here); a schema’s per-field `FieldSpec.normalizer` takes precedence for the fields it names. * **timeout** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Seconds budget for the (NP-hard) GED search; on timeout the best cost found so far is used (an approximation). Default 10.0. #### aggregate(scores) Corpus normalized distance = total raw distance / total max-distance. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.metrics.graphs.TypedNode(node_id, node_type, fields=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A node in a typed graph: an id, a node-type tag, and its field values. # _autosummary/ek.metrics.html.md # ek.metrics Reference-based (offline) metrics: the `score()` side of `ek`. A [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric) compares one prediction to one gold reference and returns a [`Score`](_autosummary/ek.base.html.md#ek.base.Score). The *right* metric is a function of the output **object type** – a string wants CER/WER, a record wants field-level F1, a table wants TEDS/GriTS, a typed graph wants a cost-weighted edit distance. This package holds the concrete metrics and registers each under the `"metrics"` namespace so the [`score()`](_autosummary/ek.facade.html.md#ek.facade.score) facade can dispatch by type or by name. Shipping in core (zero/low-dependency): [`StringMetric`](_autosummary/ek.metrics.html.md#ek.metrics.StringMetric) (CER/WER, via `jiwer` when present) and [`FieldMetric`](_autosummary/ek.metrics.html.md#ek.metrics.FieldMetric) (record/dict field F1). The richer metrics ship behind the `[metrics]` extra and import their backends lazily, so constructing or registering them never pulls a heavy dependency: - [`AnlsMetric`](_autosummary/ek.metrics.anls.html.md#ek.metrics.anls.AnlsMetric) – ANLS / ANLS\* (nested-JSON), via `anls_star`. - [`SpanF1Metric`](_autosummary/ek.metrics.spans.html.md#ek.metrics.spans.SpanF1Metric) – span/slot P/R/F1 via `seqeval` AND `nervaluate`, under an **explicit, required** [`MatchScheme`](_autosummary/ek.metrics.spans.html.md#ek.metrics.spans.MatchScheme). - [`TedsMetric`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.TedsMetric) / [`GritsMetric`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.GritsMetric) – table-structure metrics (TEDS/TEDS-Struct, GriTS-Top/GriTS-Con) with a `structure_only` toggle. - [`TypedGraphMetric`](_autosummary/ek.metrics.graphs.html.md#ek.metrics.graphs.TypedGraphMetric) – the flagship cost-weighted typed graph edit distance. See `misc/docs/ek_02` (the reference-based decision table and gotchas). ### Classes | [`StringMetric`](_autosummary/ek.metrics.html.md#ek.metrics.StringMetric)([mode, canonicalizer]) | CER (`mode='cer'`) or WER (`mode='wer'`) as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric). | |--------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------| | [`FieldMetric`](_autosummary/ek.metrics.html.md#ek.metrics.FieldMetric)(\*[, canonicalizer]) | Per-field precision/recall/F1 for two dict records. | | [`TypedGraphMetric`](_autosummary/ek.metrics.html.md#ek.metrics.TypedGraphMetric)([grammar, weights, ...]) | Cost-weighted, type-aware typed-graph edit distance as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric). | | [`TypedGraph`](_autosummary/ek.metrics.html.md#ek.metrics.TypedGraph)([nodes, edges]) | A typed graph: typed nodes plus directed typed edges between them. | | [`TypedNode`](_autosummary/ek.metrics.html.md#ek.metrics.TypedNode)(node_id, node_type[, fields]) | A node in a typed graph: an id, a node-type tag, and its field values. | | [`TypedEdge`](_autosummary/ek.metrics.html.md#ek.metrics.TypedEdge)(src, dst[, edge_type]) | A directed, typed relation between two node ids. | | [`AnlsMetric`](_autosummary/ek.metrics.html.md#ek.metrics.AnlsMetric)(\*[, threshold]) | ANLS / ANLS\* (nested-JSON) similarity as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric). | | [`SpanF1Metric`](_autosummary/ek.metrics.html.md#ek.metrics.SpanF1Metric)([scheme, tagging_scheme]) | Entity/slot precision/recall/F1 under an **explicit** [`MatchScheme`](_autosummary/ek.metrics.html.md#ek.metrics.MatchScheme). | | [`MatchScheme`](_autosummary/ek.metrics.html.md#ek.metrics.MatchScheme)(\*values) | The (backend, scheme) contract a span/slot F1 is computed under. | | [`TedsMetric`](_autosummary/ek.metrics.html.md#ek.metrics.TedsMetric)(\*[, structure_only]) | TEDS / TEDS-Struct as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric) (clean-room on `apted`). | | [`GritsMetric`](_autosummary/ek.metrics.html.md#ek.metrics.GritsMetric)(\*[, structure_only]) | GriTS-Top / GriTS-Con as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric) (pure-Python, P/R aware). | | [`Table`](_autosummary/ek.metrics.html.md#ek.metrics.Table)([rows]) | A table as an ordered list of rows of [`Cell`](_autosummary/ek.metrics.html.md#ek.metrics.Cell) s (the logical cells). | | [`Cell`](_autosummary/ek.metrics.html.md#ek.metrics.Cell)([text, rowspan, colspan]) | One logical table cell: its text plus its row/column span (>=1 each). | ### *class* ek.metrics.AnlsMetric(, threshold=0.5) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) ANLS / ANLS\* (nested-JSON) similarity as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric). Handles a bare string (classic ANLS) or an arbitrarily nested dict/list/tuple/`None` structure (ANLS\*); the backend dispatches on type. * **Parameters:** **threshold** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – ANLS zeroes a per-leaf similarity below this value before averaging (the classic ANLS 0.5 cut tolerates minor OCR/spelling noise). `anls_star` exposes no parameter for this – it reads a class constant `ANLSTree.THRESHOLD` (default `0.5`) – so the metric applies the threshold by setting that constant around the call and restoring it. **Caveat:** this mutates a process-global, so it is not thread-safe; concurrent `AnlsMetric` calls with different thresholds in the same process can interfere. Defaults to `0.5`. #### aggregate(scores) Corpus ANLS = mean of per-item ANLS (each is already length-normalized). Unlike CER/WER (which accumulate global edit counts) or field-F1 (micro-TP/ FP/FN), ANLS is defined as the *average* normalized similarity over the evaluation set – the “A” in ANLS – so the corpus statistic is the plain mean of the per-item scores. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.metrics.Cell(text='', rowspan=1, colspan=1) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One logical table cell: its text plus its row/column span (>=1 each). ### *class* ek.metrics.FieldMetric(, canonicalizer=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Per-field precision/recall/F1 for two dict records. * **Parameters:** **canonicalizer** – Optional `str -> str` applied to string values before comparison (e.g. casefold/whitespace folding). #### aggregate(scores) Micro-averaged F1 over the corpus (sum TP/FP/FN, then divide). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.metrics.GritsMetric(, structure_only=False) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) GriTS-Top / GriTS-Con as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric) (pure-Python, P/R aware). * **Parameters:** **structure_only** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – When `True`, score cell topology only (GriTS-Top); when `False` (default), average topology with cell-content similarity (GriTS-Con). #### aggregate(scores) Corpus GriTS F1: pool overlap and per-table cell counts, then divide once. GriTS precision = overlap / ``` |pred cells| ``` and recall = overlap / ``` |gold cells| ``` ; the corpus statistic micro-averages by summing the numerators and denominators globally – never a mean of per-table F1s. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.metrics.MatchScheme(\*values) Bases: [`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Enum`](https://docs.python.org/3/library/enum.html#enum.Enum) The (backend, scheme) contract a span/slot F1 is computed under. The two backends legitimately disagree (seqeval ignores other-type tags at the tag level; nervaluate includes them), so the scheme is always explicit. The `SEQEVAL_*` members consume BIO/IOBES *tag sequences*; the others consume SemEval-2013 *span lists* via nervaluate. #### EXACT *= 'exact'* exact span boundaries, type ignored. * **Type:** nervaluate exact #### PARTIAL *= 'partial'* boundary *overlap* (half credit), type ignored. * **Type:** nervaluate partial #### SEQEVAL_CONLL *= 'seqeval_conll'* seqeval, CoNLL `conlleval`-compatible entity F1 (the default seqeval mode). #### SEQEVAL_STRICT *= 'seqeval_strict'* span boundaries AND type must match, keyed to a scheme. * **Type:** seqeval strict mode #### STRICT *= 'strict'* exact span boundaries AND correct type. * **Type:** nervaluate SemEval-2013 strict #### TYPE *= 'type'* correct type with at least some span overlap. * **Type:** nervaluate type ### *class* ek.metrics.SpanF1Metric(scheme=None, , tagging_scheme='IOB2') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Entity/slot precision/recall/F1 under an **explicit** [`MatchScheme`](_autosummary/ek.metrics.html.md#ek.metrics.MatchScheme). * **Parameters:** * **scheme** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`MatchScheme`](_autosummary/ek.metrics.spans.html.md#ek.metrics.spans.MatchScheme)]) – REQUIRED. The (backend, scheme) contract – there is deliberately no default, because seqeval and nervaluate disagree and a bare F1 is ambiguous (`misc/docs/ek_02` §2.2). Pass a [`MatchScheme`](_autosummary/ek.metrics.html.md#ek.metrics.MatchScheme). * **tagging_scheme** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – For [`MatchScheme.SEQEVAL_STRICT`](_autosummary/ek.metrics.html.md#ek.metrics.MatchScheme.SEQEVAL_STRICT) only: the BIO/IOBES tagging scheme name (e.g. `"IOB2"`, `"IOBES"`, `"BILOU"`) that strict-mode entity decoding is keyed to. Defaults to `"IOB2"`. * **Raises:** [**TypeError**](https://docs.python.org/3/builtins/exceptions.html#TypeError) – if `scheme` is omitted (the whole point of this metric). #### aggregate(scores) Micro-averaged F1 over the corpus (sum the raw counts, then divide once). seqeval schemes carry TP/FP/FN; nervaluate schemes carry COR/INC/PAR/MIS/SPU (PAR worth half). Both micro-average by summing counts globally – never a mean of per-document F1s. The two backends’ counts are not commensurable, so a corpus mixing them is rejected rather than silently dropping one. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.metrics.StringMetric(mode='cer', , canonicalizer=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) CER (`mode='cer'`) or WER (`mode='wer'`) as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric). * **Parameters:** * **mode** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – `"cer"` (character) or `"wer"` (word) error rate. * **canonicalizer** – Optional `str -> str` applied to both sides before scoring (see [`ek.canonicalize`](_autosummary/ek.canonicalize.html.md#module-ek.canonicalize)). Usually supplied by the facade instead. #### aggregate(scores) Corpus error rate = total edits / total reference length (the correct way). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.metrics.Table(rows=()) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A table as an ordered list of rows of [`Cell`](_autosummary/ek.metrics.html.md#ek.metrics.Cell) s (the logical cells). `rows[i]` is the i-th row’s logical cells in reading order; spans are carried on each [`Cell`](_autosummary/ek.metrics.html.md#ek.metrics.Cell). Use [`from_html()`](_autosummary/ek.metrics.html.md#ek.metrics.Table.from_html) / [`from_grid()`](_autosummary/ek.metrics.html.md#ek.metrics.Table.from_grid) to build one, and [`as_grid()`](_autosummary/ek.metrics.html.md#ek.metrics.Table.as_grid) for the dense 2-D occupancy view GriTS scores. #### as_grid() Expand spans into a dense 2-D grid; a spanned position repeats its cell. Each physical grid position holds the (shared) [`Cell`](_autosummary/ek.metrics.html.md#ek.metrics.Cell) covering it, so two tables with the same topology have grids of the same shape – the view GriTS aligns position-by-position. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Cell`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Cell)]]] #### *classmethod* coerce(obj) Coerce a [`Table`](_autosummary/ek.metrics.html.md#ek.metrics.Table), an HTML string, or a 2-D grid into a Table. * **Return type:** [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table) #### *classmethod* from_grid(grid) Build a Table from a 2-D grid of cell texts (`None` = spanned-over). * **Return type:** [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table) #### *classmethod* from_html(html) Parse an HTML `
` (with `colspan`/`rowspan`) into a Table. * **Return type:** [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table) ### *class* ek.metrics.TedsMetric(, structure_only=False) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) TEDS / TEDS-Struct as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric) (clean-room on `apted`). * **Parameters:** **structure_only** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – When `True`, ignore cell text and score structure alone (TEDS-Struct), isolating table-structure accuracy from OCR noise. Defaults to `False` (full TEDS: structure + content). #### aggregate(scores) Corpus TEDS = **mean** of per-table TEDS (the PubTabNet/OmniDocBench convention). Each per-table TEDS is already node-count-normalized, so the reported corpus statistic is the average, not a pooled edit-rate. The pooled `1 - sum(dist)/sum(nodes)` is available from the per-item `detail` if a size-weighted variant is wanted instead. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.metrics.TypedEdge(src, dst, edge_type='') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A directed, typed relation between two node ids. ### *class* ek.metrics.TypedGraph(nodes=(), edges=()) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A typed graph: typed nodes plus directed typed edges between them. #### to_networkx() Build a `networkx.DiGraph` carrying type/fields as node/edge attrs. ### *class* ek.metrics.TypedGraphMetric(grammar=None, weights=None, , canonicalizer=None, timeout=10.0, max_nodes=60) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Cost-weighted, type-aware typed-graph edit distance as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric). * **Parameters:** * **grammar** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar)]) – Layer-A [`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar) supplying cost weights (may also be passed per-call to `__call__`). * **weights** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`GraphGrammar`](_autosummary/ek.base.html.md#ek.base.GraphGrammar), [`TypeRef`](_autosummary/ek.base.html.md#ek.base.TypeRef)], [`float`](https://docs.python.org/3/builtins/functions.html#float)]]) – Optional [`CostWeight`](_autosummary/ek.base.html.md#ek.base.CostWeight) overriding the schema weights. * **canonicalizer** – Optional `str -> str` applied to field values before the equality check (the facade passes `normalize=` through here); a schema’s per-field `FieldSpec.normalizer` takes precedence for the fields it names. * **timeout** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Seconds budget for the (NP-hard) GED search; on timeout the best cost found so far is used (an approximation). Default 10.0. #### aggregate(scores) Corpus normalized distance = total raw distance / total max-distance. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.metrics.TypedNode(node_id, node_type, fields=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A node in a typed graph: an id, a node-type tag, and its field values. ### Modules | [`anls`](_autosummary/ek.metrics.anls.html.md#module-ek.metrics.anls) | ANLS / ANLS\* (nested-JSON) similarity, wrapping `anls_star`. | |------------------------------------------------------------------------------------|--------------------------------------------------------------------------------| | [`fields`](_autosummary/ek.metrics.fields.html.md#module-ek.metrics.fields) | Field-level metric for record / dict-shaped extractions. | | [`graphs`](_autosummary/ek.metrics.graphs.html.md#module-ek.metrics.graphs) | The flagship metric: cost-weighted, type-aware distance over a typed graph. | | [`spans`](_autosummary/ek.metrics.spans.html.md#module-ek.metrics.spans) | Span / slot precision-recall-F1 over two backends that legitimately disagree. | | [`strings`](_autosummary/ek.metrics.strings.html.md#module-ek.metrics.strings) | String metrics: Character/Word Error Rate (CER/WER). | | [`tables`](_autosummary/ek.metrics.tables.html.md#module-ek.metrics.tables) | Table-structure metrics: TEDS / TEDS-Struct and GriTS (structure-only toggle). | # _autosummary/ek.metrics.spans.html.md # ek.metrics.spans Span / slot precision-recall-F1 over two backends that legitimately disagree. Entity/slot F1 is the workhorse for fields-and-spans extraction, but a *single* F1 is meaningless without naming its **match scheme** – and the two standard backends diverge widely *and disagree with each other* (`misc/docs/ek_02` §2.2): - **seqeval** (MIT, `chakki-works`) – CoNLL `conlleval`-compatible entity F1 over BIO/IOBES tag *sequences*. Its `conll` (default) and `strict` (span+type, keyed to a tagging scheme) modes are the reference for sequence-labelling NER. - **nervaluate** (MIT, `MantisAI`) – the SemEval-2013 Task 9.1 scheme over typed *spans*, with four schemas: **strict** (span + type), **exact** (span, any type), **partial** (boundary overlap, any type), **type** (type with span overlap). It counts COR/INC/PAR/MIS/SPU and gives partial spans half credit. A model that finds “Electric” instead of “General Electric” scores F1=0 under exact-span matching but ~0.667 under a partial/token scheme – so the scheme is *not* a default you can hide. This module makes [`MatchScheme`](_autosummary/ek.metrics.spans.html.md#ek.metrics.spans.MatchScheme) a **required keyword** with no default: you must say which contract you are scoring under. Both backends ship behind the `[metrics]` extra and are imported lazily, so importing `ek` never pulls them in. Each [`Score`](_autosummary/ek.base.html.md#ek.base.Score) carries the raw TP/FP/FN (or COR/INC/PAR/MIS/SPU) counts in its `detail` so [`SpanF1Metric.aggregate()`](_autosummary/ek.metrics.spans.html.md#ek.metrics.spans.SpanF1Metric.aggregate) micro-averages over a corpus – never a naive mean of per-document F1s. ## Input shapes - `seqeval` schemes ([`MatchScheme.SEQEVAL_CONLL`](_autosummary/ek.metrics.spans.html.md#ek.metrics.spans.MatchScheme.SEQEVAL_CONLL), [`MatchScheme.SEQEVAL_STRICT`](_autosummary/ek.metrics.spans.html.md#ek.metrics.spans.MatchScheme.SEQEVAL_STRICT)): `pred` and `gold` are **tag sequences** – > a list of BIO/IOBES tags (e.g. `["B-PER", "I-PER", "O"]`). - `nervaluate` schemes ([`MatchScheme.STRICT`](_autosummary/ek.metrics.spans.html.md#ek.metrics.spans.MatchScheme.STRICT), `EXACT`, `PARTIAL`, `TYPE`): `pred` and `gold` are **span lists** – a list of `{"label", "start", "end"}` dicts. ### Example ```pycon >>> from ek.metrics.spans import SpanF1Metric, MatchScheme >>> m = SpanF1Metric(scheme=MatchScheme.SEQEVAL_CONLL) >>> s = m(["B-PER", "I-PER", "O"], ["B-PER", "I-PER", "O"]) >>> s.f1 1.0 >>> SpanF1Metric() Traceback (most recent call last): TypeError: SpanF1Metric requires an explicit scheme=MatchScheme.<...> ``` ### Classes | [`MatchScheme`](_autosummary/ek.metrics.spans.html.md#ek.metrics.spans.MatchScheme)(\*values) | The (backend, scheme) contract a span/slot F1 is computed under. | |-----------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------| | [`SpanF1Metric`](_autosummary/ek.metrics.spans.html.md#ek.metrics.spans.SpanF1Metric)([scheme, tagging_scheme]) | Entity/slot precision/recall/F1 under an **explicit** [`MatchScheme`](_autosummary/ek.metrics.spans.html.md#ek.metrics.spans.MatchScheme). | ### *class* ek.metrics.spans.MatchScheme(\*values) Bases: [`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Enum`](https://docs.python.org/3/library/enum.html#enum.Enum) The (backend, scheme) contract a span/slot F1 is computed under. The two backends legitimately disagree (seqeval ignores other-type tags at the tag level; nervaluate includes them), so the scheme is always explicit. The `SEQEVAL_*` members consume BIO/IOBES *tag sequences*; the others consume SemEval-2013 *span lists* via nervaluate. #### EXACT *= 'exact'* exact span boundaries, type ignored. * **Type:** nervaluate exact #### PARTIAL *= 'partial'* boundary *overlap* (half credit), type ignored. * **Type:** nervaluate partial #### SEQEVAL_CONLL *= 'seqeval_conll'* seqeval, CoNLL `conlleval`-compatible entity F1 (the default seqeval mode). #### SEQEVAL_STRICT *= 'seqeval_strict'* span boundaries AND type must match, keyed to a scheme. * **Type:** seqeval strict mode #### STRICT *= 'strict'* exact span boundaries AND correct type. * **Type:** nervaluate SemEval-2013 strict #### TYPE *= 'type'* correct type with at least some span overlap. * **Type:** nervaluate type ### *class* ek.metrics.spans.SpanF1Metric(scheme=None, , tagging_scheme='IOB2') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Entity/slot precision/recall/F1 under an **explicit** [`MatchScheme`](_autosummary/ek.metrics.spans.html.md#ek.metrics.spans.MatchScheme). * **Parameters:** * **scheme** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`MatchScheme`](_autosummary/ek.metrics.spans.html.md#ek.metrics.spans.MatchScheme)]) – REQUIRED. The (backend, scheme) contract – there is deliberately no default, because seqeval and nervaluate disagree and a bare F1 is ambiguous (`misc/docs/ek_02` §2.2). Pass a [`MatchScheme`](_autosummary/ek.metrics.spans.html.md#ek.metrics.spans.MatchScheme). * **tagging_scheme** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – For [`MatchScheme.SEQEVAL_STRICT`](_autosummary/ek.metrics.spans.html.md#ek.metrics.spans.MatchScheme.SEQEVAL_STRICT) only: the BIO/IOBES tagging scheme name (e.g. `"IOB2"`, `"IOBES"`, `"BILOU"`) that strict-mode entity decoding is keyed to. Defaults to `"IOB2"`. * **Raises:** [**TypeError**](https://docs.python.org/3/builtins/exceptions.html#TypeError) – if `scheme` is omitted (the whole point of this metric). #### aggregate(scores) Micro-averaged F1 over the corpus (sum the raw counts, then divide once). seqeval schemes carry TP/FP/FN; nervaluate schemes carry COR/INC/PAR/MIS/SPU (PAR worth half). Both micro-average by summing counts globally – never a mean of per-document F1s. The two backends’ counts are not commensurable, so a corpus mixing them is rejected rather than silently dropping one. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) # _autosummary/ek.metrics.strings.html.md # ek.metrics.strings String metrics: Character/Word Error Rate (CER/WER). CER and WER are graded edit-distance metrics – the right baseline for OCR/ASR and free-text fields, far less brittle than exact match. Two correctness rules from the research drive the design: 1. **Accumulate globally.** A corpus error rate is `sum(edits) / sum(reference length)` over the whole corpus – never the mean of per-item rates (variable lengths make that mathematically wrong). So each [`Score`](_autosummary/ek.base.html.md#ek.base.Score) carries raw `edits` and `ref_len` in its `detail`, and [`StringMetric.aggregate()`](_autosummary/ek.metrics.strings.html.md#ek.metrics.strings.StringMetric.aggregate) sums them. [`evaluate()`](_autosummary/ek.facade.html.md#ek.facade.evaluate) uses this. 2. **CER/WER are error rates, not bounded similarities.** A hypothesis longer than the reference pushes the rate above 1.0 – itself a useful hallucination signal. `Score.value` is the error rate (lower is better; `detail["higher_is_better"] is False`). The canonical backend is `jiwer` (Apache-2.0, `rapidfuzz` C++ core), which also yields the substitution/deletion/insertion/hit breakdown. `ek` prefers it, falls back to `rapidfuzz` directly, then to a pure-Python edit distance, so the metric works even in a bare environment. ### Classes | [`StringMetric`](_autosummary/ek.metrics.strings.html.md#ek.metrics.strings.StringMetric)([mode, canonicalizer]) | CER (`mode='cer'`) or WER (`mode='wer'`) as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric). | |----------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------| ### *class* ek.metrics.strings.StringMetric(mode='cer', , canonicalizer=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) CER (`mode='cer'`) or WER (`mode='wer'`) as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric). * **Parameters:** * **mode** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – `"cer"` (character) or `"wer"` (word) error rate. * **canonicalizer** – Optional `str -> str` applied to both sides before scoring (see [`ek.canonicalize`](_autosummary/ek.canonicalize.html.md#module-ek.canonicalize)). Usually supplied by the facade instead. #### aggregate(scores) Corpus error rate = total edits / total reference length (the correct way). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) # _autosummary/ek.metrics.tables.html.md # ek.metrics.tables Table-structure metrics: TEDS / TEDS-Struct and GriTS (structure-only toggle). Two complementary table-recognition metrics from `misc/docs/ek_02` §3.1: - **TEDS** (Tree-Edit-Distance-based Similarity; Zhong et al., PubTabNet, IBM, [arXiv 1911.10683](https://arxiv.org/abs/1911.10683)) renders a table as an HTML tree and scores `1 - EditDist(Ta, Tb) / max(|Ta|, |Tb|)`. It captures both *structure* and *cell content*. **TEDS-Struct** (the `structure_only=True` toggle) ignores cell text to isolate structure, since “taking OCR errors into account may lead to an unfair comparison due to the different OCR models used by various TSR methods”. - **GriTS** (Grid Table Similarity; Smock et al., Microsoft, [arXiv 2203.12555](https://arxiv.org/abs/2203.12555)) scores the predicted table directly as a 2-D matrix and yields **precision/recall** over the most-similar common sub-grid. Its **GriTS-Top** (topology, structure-only) and **GriTS-Con** (content) variants share the `structure_only` toggle. Licensing note (`skills/ek-dev-licensing`): the obvious wrapper target, `table-recognition-metric`, pulls in the **GPL** `Levenshtein` package as a hard runtime dependency (`from Levenshtein import distance` in its source) – a scanner-invisible copyleft landmine that would fail the CI license gate. So TEDS is implemented **clean-room** here on the MIT `apted` engine (already in `[metrics]`), and GriTS is a dependency-free pure-Python implementation of the 2D-MSS heuristic. Both follow the precedent of reimplementing Krippendorff’s alpha in pure Python rather than quarantining a copyleft dependency. ## Input shapes `pred` and `gold` are [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table) objects, or anything the constructors accept: an HTML `str` (`
...
`; via [`Table.from_html()`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table.from_html)) or a 2-D grid – a list of rows of cell strings, where `None` marks a cell spanned over by a neighbour (via [`Table.from_grid()`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table.from_grid)). The facade and registry coerce HTML strings automatically. ### Example ```pycon >>> from ek.metrics.tables import TedsMetric, GritsMetric, Table >>> a = "
ab
" >>> b = "
ac
" >>> round(TedsMetric()(a, b).value, 3) # one cell-content edit 0.75 >>> TedsMetric(structure_only=True)(a, b).value # same structure -> 1.0 1.0 >>> GritsMetric(structure_only=True)(a, b).value # identical grid topology 1.0 ``` ### Classes | [`Cell`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Cell)([text, rowspan, colspan]) | One logical table cell: its text plus its row/column span (>=1 each). | |------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------| | [`GritsMetric`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.GritsMetric)(\*[, structure_only]) | GriTS-Top / GriTS-Con as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric) (pure-Python, P/R aware). | | [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table)([rows]) | A table as an ordered list of rows of [`Cell`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Cell) s (the logical cells). | | [`TedsMetric`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.TedsMetric)(\*[, structure_only]) | TEDS / TEDS-Struct as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric) (clean-room on `apted`). | ### *class* ek.metrics.tables.Cell(text='', rowspan=1, colspan=1) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One logical table cell: its text plus its row/column span (>=1 each). ### *class* ek.metrics.tables.GritsMetric(, structure_only=False) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) GriTS-Top / GriTS-Con as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric) (pure-Python, P/R aware). * **Parameters:** **structure_only** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – When `True`, score cell topology only (GriTS-Top); when `False` (default), average topology with cell-content similarity (GriTS-Con). #### aggregate(scores) Corpus GriTS F1: pool overlap and per-table cell counts, then divide once. GriTS precision = overlap / ``` |pred cells| ``` and recall = overlap / ``` |gold cells| ``` ; the corpus statistic micro-averages by summing the numerators and denominators globally – never a mean of per-table F1s. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.metrics.tables.Table(rows=()) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) A table as an ordered list of rows of [`Cell`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Cell) s (the logical cells). `rows[i]` is the i-th row’s logical cells in reading order; spans are carried on each [`Cell`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Cell). Use [`from_html()`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table.from_html) / [`from_grid()`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table.from_grid) to build one, and [`as_grid()`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table.as_grid) for the dense 2-D occupancy view GriTS scores. #### as_grid() Expand spans into a dense 2-D grid; a spanned position repeats its cell. Each physical grid position holds the (shared) [`Cell`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Cell) covering it, so two tables with the same topology have grids of the same shape – the view GriTS aligns position-by-position. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Cell`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Cell)]]] #### *classmethod* coerce(obj) Coerce a [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table), an HTML string, or a 2-D grid into a Table. * **Return type:** [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table) #### *classmethod* from_grid(grid) Build a Table from a 2-D grid of cell texts (`None` = spanned-over). * **Return type:** [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table) #### *classmethod* from_html(html) Parse an HTML `` (with `colspan`/`rowspan`) into a Table. * **Return type:** [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table) ### *class* ek.metrics.tables.TedsMetric(, structure_only=False) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) TEDS / TEDS-Struct as a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric) (clean-room on `apted`). * **Parameters:** **structure_only** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – When `True`, ignore cell text and score structure alone (TEDS-Struct), isolating table-structure accuracy from OCR noise. Defaults to `False` (full TEDS: structure + content). #### aggregate(scores) Corpus TEDS = **mean** of per-table TEDS (the PubTabNet/OmniDocBench convention). Each per-table TEDS is already node-count-normalized, so the reported corpus statistic is the average, not a pooled edit-rate. The pooled `1 - sum(dist)/sum(nodes)` is available from the per-item `detail` if a size-weighted variant is wanted instead. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) # _autosummary/ek.ocr.benchmark.html.md # ek.ocr.benchmark End-to-end OCR offline benchmark: run a fleet over gold, score, persist. This is `ek`’s first out-of-the-box experience – “how accurate is my OCR?”. It pairs each gold item with the engine’s output, scores text with a globally accumulated CER/WER (the correct corpus aggregation), optionally slices the report, and persists gold + per-item results + the run aggregate to the `dol` stores under `~/.local/share/ek/`. The runner is deliberately source-agnostic: `engine` is any `image -> OcrResult` callable (or an `ocracy` backend id, bridged for you), so the benchmark runs – and is fully testable – without any real OCR engine or cloud credentials. ### Example ```pycon >>> import tempfile; root = tempfile.mkdtemp() >>> gold = { ... "d1": {"image": "ignored", "reference_text": "cat", "slice": "easy"}, ... "d2": {"image": "ignored", "reference_text": "dog", "slice": "easy"}, ... } >>> # a stub engine that always reads "cat" (perfect for d1, wrong for d2): >>> engine = lambda image: type("R", (), {"text": "cat"})() >>> report = evaluate_ocr(engine, gold, metric="cer", rootdir=root) >>> report.n 2 >>> sorted(report.per_slice) ['easy'] ``` ### Functions | [`add_gold_item`](_autosummary/ek.ocr.benchmark.html.md#ek.ocr.benchmark.add_gold_item)(key, image, reference_text, \*) | Add one gold example to the `gold` store. | |--------------------------------------------------------------------------------------------------|------------------------------------------------| | [`evaluate_ocr`](_autosummary/ek.ocr.benchmark.html.md#ek.ocr.benchmark.evaluate_ocr)(engine, gold, \*[, metric, ...]) | Benchmark an OCR engine against a gold corpus. | ### ek.ocr.benchmark.add_gold_item(key, image, reference_text, , slice=None, rootdir=None, \*\*extra) Add one gold example to the `gold` store. * **Parameters:** * **key** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Unique id for the document. * **image** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – An image reference the engine can read (path, URL, …). Stored as a reference, not bytes – gold stores hold references + the expected text. * **reference_text** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The ground-truth transcription. * **slice** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – Optional stratification label (doc type, language, scan quality, …). * **rootdir** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – Override the data root (tests/ephemeral use). * **\*\*extra** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Any additional metadata to store on the record. * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) ### ek.ocr.benchmark.evaluate_ocr(engine, gold, , metric='cer', normalize=None, grammar=None, image_key='image', reference_key='reference_text', slice_key='slice', persist=False, run_id=None, rootdir=None) Benchmark an OCR engine against a gold corpus. * **Parameters:** * **engine** (`Union`[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)]) – An `image -> OcrResult` callable, or an `ocracy` backend id. * **gold** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)) – A mapping `key -> {image, reference_text, [slice], ...}` – a plain dict or a `ek` gold store. * **metric** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – `"cer"` (default) or `"wer"` (or any registered string metric). * **normalize** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Optional canonicalizer applied to both sides before scoring. * **grammar** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Optional Layer-A grammar (forwarded to the metric). * **image_key** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Field names within each gold record. * **reference_key** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Field names within each gold record. * **slice_key** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Field names within each gold record. * **persist** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – If true, write per-item results and the run aggregate to the `results` / `runs` stores. * **run_id** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – Identifier for this run (auto-generated from the clock if omitted). * **rootdir** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – Override the data root (tests/ephemeral use). * **Returns:** A [`Report`](_autosummary/ek.base.html.md#ek.base.Report) whose `detail['per_item']` maps each gold key to its prediction, reference, slice, score, and engine confidence/backend. # _autosummary/ek.ocr.bridge.html.md # ek.ocr.bridge The `ocracy` bridge: turn the OCR fleet into an `image -> OcrResult` callable. `ocracy` is a deliberately dependency-free facade over ~16 OCR engines whose single entry point `ocracy.ocr(image, *, backend=None, **kwargs)` returns a normalized `OcrResult` (`.text` plus `.blocks` carrying `[0, 1]`-normalized confidence and geometry). `ek` consumes that shape rather than reinventing engine integration. The bridge is intentionally tiny – it only adapts `ocracy`’s call into the [`OcrBackend`](_autosummary/ek.base.html.md#ek.base.OcrBackend) contract and guards the optional dependency with an actionable install hint. Null-safety reminder (see `misc/docs/ek_01`): VLM/markdown engines (Claude, GPT-4o, Mistral OCR) return text only – `blocks == []` and `confidence is None`. Any code reading confidence/geometry off the result must tolerate `None`. ### Functions | [`ocracy_backend`](_autosummary/ek.ocr.bridge.html.md#ek.ocr.bridge.ocracy_backend)([backend]) | Return an `image -> OcrResult` callable bound to one `ocracy` backend. | |----------------------------------------------------------------------------------|--------------------------------------------------------------------------| | [`read_text`](_autosummary/ek.ocr.bridge.html.md#ek.ocr.bridge.read_text)(image, \*[, backend]) | Convenience: OCR an image and return just the recognized text string. | ### ek.ocr.bridge.ocracy_backend(backend=None, \*\*default_kwargs) Return an `image -> OcrResult` callable bound to one `ocracy` backend. * **Parameters:** * **backend** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – An `ocracy` backend id (e.g. `"tesseract"`, `"google_vision"`); `None` uses `ocracy`’s first installed backend. * **\*\*default_kwargs** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Options forwarded to `ocracy.ocr` on every call (e.g. `languages=["en"]`); per-call kwargs override these. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] * **Returns:** A callable suitable for [`ek.ocr.evaluate_ocr()`](_autosummary/ek.ocr.html.md#ek.ocr.evaluate_ocr). ### ek.ocr.bridge.read_text(image, , backend=None, \*\*kwargs) Convenience: OCR an image and return just the recognized text string. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) # _autosummary/ek.ocr.html.md # ek.ocr OCR evaluation – `ek`’s first concrete instance, built on `ocracy`. OCR is the noisiest special case of information extraction. This subpackage adds the OCR-specific pieces on top of `ek`’s source-agnostic core: - [`ocracy_backend()`](_autosummary/ek.ocr.html.md#ek.ocr.ocracy_backend) / [`read_text()`](_autosummary/ek.ocr.html.md#ek.ocr.read_text) – a thin bridge that turns `ocracy`’s 16-engine fleet into an `image -> OcrResult` callable. The dependency arrow is strict: `ek -> ocracy` (via the `ek[ocr]` extra), never the reverse. `ek` core depends only on the `OcrResult` *shape* (`.text` / `.blocks` / `.mean_confidence`), so the benchmark below evaluates *any* `image -> OcrResult` callable – `ocracy` or your own. - [`profile()`](_autosummary/ek.ocr.html.md#ek.ocr.profile) – per-engine capability profiles (who emits real vs model-guessed geometry, who is calibrated, who yields table structure), distilled from `misc/docs/ek_01`. `ek` treats these as claims to verify empirically. - [`evaluate_ocr()`](_autosummary/ek.ocr.html.md#ek.ocr.evaluate_ocr) / [`add_gold_item()`](_autosummary/ek.ocr.html.md#ek.ocr.add_gold_item) – run a fleet over a gold corpus, score with CER/WER (globally accumulated), slice the report, and persist gold + results to the `dol` stores under `~/.local/share/ek/`. - [`table_from_ocr()`](_autosummary/ek.ocr.html.md#ek.ocr.table_from_ocr) / [`has_table_structure()`](_autosummary/ek.ocr.html.md#ek.ocr.has_table_structure) – recover the existing [`ek.metrics.tables.Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table) from an `OcrResult` (whose engine-specific table > data lives only in `.raw`/`.markdown`) so the table metrics (TEDS / GriTS) can > score OCR table output uniformly. The per-engine extractor is injected/registered > under the `"table_parsers"` namespace (open-closed); see [`ek.ocr.tables`](_autosummary/ek.ocr.tables.html.md#module-ek.ocr.tables). Heavy engine SDKs and credentials are never required to import this module; they are checked at call time via [`ek.registry.check_requirements()`](_autosummary/ek.registry.html.md#ek.registry.check_requirements). ### Functions | [`ocracy_backend`](_autosummary/ek.ocr.html.md#ek.ocr.ocracy_backend)([backend]) | Return an `image -> OcrResult` callable bound to one `ocracy` backend. | |------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------| | [`read_text`](_autosummary/ek.ocr.html.md#ek.ocr.read_text)(image, \*[, backend]) | Convenience: OCR an image and return just the recognized text string. | | [`evaluate_ocr`](_autosummary/ek.ocr.html.md#ek.ocr.evaluate_ocr)(engine, gold, \*[, metric, ...]) | Benchmark an OCR engine against a gold corpus. | | [`add_gold_item`](_autosummary/ek.ocr.html.md#ek.ocr.add_gold_item)(key, image, reference_text, \*) | Add one gold example to the `gold` store. | | [`profile`](_autosummary/ek.ocr.html.md#ek.ocr.profile)(engine) | Capability profile for an engine id (a safe default if unknown). | | [`table_from_ocr`](_autosummary/ek.ocr.html.md#ek.ocr.table_from_ocr)(ocr_result, \*[, parser]) | Extract a normalized [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table) from an `OcrResult`. | | [`has_table_structure`](_autosummary/ek.ocr.html.md#ek.ocr.has_table_structure)(ocr_result, \*[, parser]) | Whether [`table_from_ocr()`](_autosummary/ek.ocr.html.md#ek.ocr.table_from_ocr) recovers a non-empty table from `ocr_result`. | | [`resolve_table_parser`](_autosummary/ek.ocr.html.md#ek.ocr.resolve_table_parser)(ref) | Resolve a table parser reference (a registered name or a callable) to a callable. | | [`engine_yields_tables`](_autosummary/ek.ocr.html.md#ek.ocr.engine_yields_tables)(engine) | Whether `engine`'s capability profile claims scoreable table structure. | ### ek.ocr.add_gold_item(key, image, reference_text, , slice=None, rootdir=None, \*\*extra) Add one gold example to the `gold` store. * **Parameters:** * **key** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Unique id for the document. * **image** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – An image reference the engine can read (path, URL, …). Stored as a reference, not bytes – gold stores hold references + the expected text. * **reference_text** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The ground-truth transcription. * **slice** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – Optional stratification label (doc type, language, scan quality, …). * **rootdir** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – Override the data root (tests/ephemeral use). * **\*\*extra** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Any additional metadata to store on the record. * **Return type:** [`None`](https://docs.python.org/3/builtins/constants.html#None) ### ek.ocr.engine_yields_tables(engine) Whether `engine`’s capability profile claims scoreable table structure. A convenience over [`ek.ocr.profile()`](_autosummary/ek.ocr.html.md#ek.ocr.profile): returns the engine profile’s `tables` flag. ek treats the profile as a *prior* (Azure DI, AWS Textract, PaddleOCR and the markdown engines are expected to yield scoreable tables; Google Vision is not), to be verified empirically by actually running [`table_from_ocr()`](_autosummary/ek.ocr.html.md#ek.ocr.table_from_ocr). Use it to skip table scoring for engines that cannot emit cells in the first place. * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.ocr.evaluate_ocr(engine, gold, , metric='cer', normalize=None, grammar=None, image_key='image', reference_key='reference_text', slice_key='slice', persist=False, run_id=None, rootdir=None) Benchmark an OCR engine against a gold corpus. * **Parameters:** * **engine** (`Union`[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)]) – An `image -> OcrResult` callable, or an `ocracy` backend id. * **gold** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)) – A mapping `key -> {image, reference_text, [slice], ...}` – a plain dict or a `ek` gold store. * **metric** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – `"cer"` (default) or `"wer"` (or any registered string metric). * **normalize** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Optional canonicalizer applied to both sides before scoring. * **grammar** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Optional Layer-A grammar (forwarded to the metric). * **image_key** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Field names within each gold record. * **reference_key** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Field names within each gold record. * **slice_key** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Field names within each gold record. * **persist** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – If true, write per-item results and the run aggregate to the `results` / `runs` stores. * **run_id** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – Identifier for this run (auto-generated from the clock if omitted). * **rootdir** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – Override the data root (tests/ephemeral use). * **Returns:** A [`Report`](_autosummary/ek.base.html.md#ek.base.Report) whose `detail['per_item']` maps each gold key to its prediction, reference, slice, score, and engine confidence/backend. ### ek.ocr.has_table_structure(ocr_result, , parser=None) Whether [`table_from_ocr()`](_autosummary/ek.ocr.html.md#ek.ocr.table_from_ocr) recovers a non-empty table from `ocr_result`. A thin predicate over [`table_from_ocr()`](_autosummary/ek.ocr.html.md#ek.ocr.table_from_ocr) (same `parser` semantics): true iff a table with at least one cell can be recovered. Use it to route an OCR result to the table metrics vs the text-only CER/WER fallback. * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.ocr.ocracy_backend(backend=None, \*\*default_kwargs) Return an `image -> OcrResult` callable bound to one `ocracy` backend. * **Parameters:** * **backend** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – An `ocracy` backend id (e.g. `"tesseract"`, `"google_vision"`); `None` uses `ocracy`’s first installed backend. * **\*\*default_kwargs** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Options forwarded to `ocracy.ocr` on every call (e.g. `languages=["en"]`); per-call kwargs override these. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] * **Returns:** A callable suitable for [`ek.ocr.evaluate_ocr()`](_autosummary/ek.ocr.html.md#ek.ocr.evaluate_ocr). ### ek.ocr.profile(engine) Capability profile for an engine id (a safe default if unknown). Accepts `ocracy`’s canonical hyphenated ids and their underscore variants. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### ek.ocr.read_text(image, , backend=None, \*\*kwargs) Convenience: OCR an image and return just the recognized text string. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### ek.ocr.resolve_table_parser(ref) Resolve a table parser reference (a registered name or a callable) to a callable. Mirrors how other ek strategies resolve from the registry (see [`ek.registry.resolve()`](_autosummary/ek.registry.html.md#ek.registry.resolve)). A `str` is looked up under the `"table_parsers"` namespace; an already-callable parser is returned unchanged. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] ### ek.ocr.table_from_ocr(ocr_result, , parser=None) Extract a normalized [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table) from an `OcrResult`. The single seam ek’s table metrics (TEDS / GriTS) use to score OCR table output uniformly. Returns the recovered Table, or `None` when the result carries no recoverable table structure. * **Parameters:** * **ocr_result** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Any `OcrResult`-shaped object. ek reads only its `.raw` (the engine-specific payload), `.markdown` (VLM/markdown engines), and `.meta` – never ocracy types, so any `image -> OcrResult` callable’s output works. * **parser** (`Union`[[`None`](https://docs.python.org/3/builtins/constants.html#None), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]]) – How to turn the raw payload into a table (dependency injection / open-closed). One of: - `None` (default): try the built-in **safe heuristics** – `.raw` is already a [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table), a 2-D grid, or a `
...
` HTML string; or `.raw`/`.meta` is a mapping carrying one of `{tables, table, cells, grid, html}`; or `.markdown` contains an HTML ``. Returns `None` if none match – it never guesses wildly. - a `str`: the name of a parser registered under `"table_parsers"` (per-engine extractor), resolved from [`ek.registry`](_autosummary/ek.registry.html.md#module-ek.registry). - a callable `(raw) -> Table | grid | html | None`: an inline per-engine extractor. It receives the result’s `.raw` (falling back to the result itself when there is no `.raw`). * **through** (*Any non-None value an injected parser returns is fed*) * **Return type:** [*Table*](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table) | *None* :param `Table.coerce()`: :param so a parser may yield a Table: :param a 2-D grid: :param or HTML.: * **Return type:** [`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table)] * **Returns:** A non-empty [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table), or `None` when no table structure is recoverable. An empty Table (no rows/cells) is normalized to `None` so the predicate [`has_table_structure()`](_autosummary/ek.ocr.html.md#ek.ocr.has_table_structure) and downstream metrics see “no table”, not a degenerate one. ### Modules | [`benchmark`](_autosummary/ek.ocr.benchmark.html.md#module-ek.ocr.benchmark) | End-to-end OCR offline benchmark: run a fleet over gold, score, persist. | |--------------------------------------------------------------------------------------|------------------------------------------------------------------------------------| | [`bridge`](_autosummary/ek.ocr.bridge.html.md#module-ek.ocr.bridge) | The `ocracy` bridge: turn the OCR fleet into an `image -> OcrResult` callable. | | [`profiles`](_autosummary/ek.ocr.profiles.html.md#module-ek.ocr.profiles) | Per-engine OCR capability profiles (what QE signal each engine actually emits). | | [`tables`](_autosummary/ek.ocr.tables.html.md#module-ek.ocr.tables) | Recover a normalized table from an `OcrResult` so ek's table metrics can score it. | # _autosummary/ek.ocr.profiles.html.md # ek.ocr.profiles Per-engine OCR capability profiles (what QE signal each engine actually emits). Distilled from the 15-engine inventory in `misc/docs/ek_01`. These describe what an engine *claims* to surface; `ek` exists in part to verify such claims empirically, so treat them as defaults/priors, not ground truth. The fields that matter for evaluation: - `confidence_grain` – finest confidence granularity emitted: `"symbol"` > `"word"` > `"line"` > `"none"`. Genuine per-symbol confidence exists only in Google Cloud Vision and Tesseract; many VLMs emit nothing usable. - `has_real_provenance` – whether bounding boxes are real geometry (vs model-guessed pixels from Claude/OpenAI, or char-offset only from Mistral). - `calibrated` – whether the confidence is calibrated (Mathpix is the lone calibrated emitter). - `tables` – whether the engine emits scoreable table cell structure. ### Example ```pycon >>> profile("tesseract")["confidence_grain"] 'symbol' >>> profile("claude-vision")["has_real_provenance"] False >>> profile("google_vision")["confidence_grain"] # underscore form also accepted 'symbol' >>> profile("totally-unknown-engine")["confidence_grain"] 'unknown' ``` ### Module Attributes | [`ENGINE_PROFILES`](_autosummary/ek.ocr.profiles.html.md#ek.ocr.profiles.ENGINE_PROFILES) | Engine id -> capability profile. | |--------------------------------------------------------------------|-------------------------------------------| | [`DEFAULT_PROFILE`](_autosummary/ek.ocr.profiles.html.md#ek.ocr.profiles.DEFAULT_PROFILE) | Returned for any engine not in the table. | ### Functions | [`profile`](_autosummary/ek.ocr.profiles.html.md#ek.ocr.profiles.profile)(engine) | Capability profile for an engine id (a safe default if unknown). | |--------------------------------------------------------------------|--------------------------------------------------------------------| ### ek.ocr.profiles.DEFAULT_PROFILE *= {'calibrated': False, 'confidence_grain': 'unknown', 'has_real_provenance': False, 'tables': False}* Returned for any engine not in the table. ### ek.ocr.profiles.ENGINE_PROFILES *: [dict](https://docs.python.org/3/builtins/stdtypes.html#dict)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), [dict](https://docs.python.org/3/builtins/stdtypes.html#dict)]* *= {'aws-textract': {'calibrated': False, 'confidence_grain': 'word', 'has_real_provenance': True, 'tables': True}, 'azure-document-intelligence': {'calibrated': False, 'confidence_grain': 'word', 'has_real_provenance': True, 'tables': True}, 'claude-vision': {'calibrated': False, 'confidence_grain': 'none', 'has_real_provenance': False, 'tables': False}, 'doctr': {'calibrated': False, 'confidence_grain': 'word', 'has_real_provenance': True, 'tables': False}, 'easyocr': {'calibrated': False, 'confidence_grain': 'line', 'has_real_provenance': True, 'tables': False}, 'google-vision': {'calibrated': False, 'confidence_grain': 'symbol', 'has_real_provenance': True, 'tables': False}, 'gpt-4o-vision': {'calibrated': False, 'confidence_grain': 'none', 'has_real_provenance': False, 'tables': False}, 'mathpix': {'calibrated': True, 'confidence_grain': 'word', 'has_real_provenance': True, 'tables': False}, 'mistral-ocr': {'calibrated': False, 'confidence_grain': 'none', 'has_real_provenance': False, 'tables': False}, 'ocr-space': {'calibrated': False, 'confidence_grain': 'none', 'has_real_provenance': False, 'tables': False}, 'ocrmac': {'calibrated': False, 'confidence_grain': 'line', 'has_real_provenance': True, 'tables': False}, 'paddleocr': {'calibrated': False, 'confidence_grain': 'line', 'has_real_provenance': True, 'tables': True}, 'pix2tex-latex-ocr': {'calibrated': False, 'confidence_grain': 'none', 'has_real_provenance': False, 'tables': False}, 'rapidocr': {'calibrated': False, 'confidence_grain': 'line', 'has_real_provenance': True, 'tables': False}, 'tesseract': {'calibrated': False, 'confidence_grain': 'symbol', 'has_real_provenance': True, 'tables': False}, 'trocr': {'calibrated': False, 'confidence_grain': 'none', 'has_real_provenance': False, 'tables': False}, 'trocr-handwritten': {'calibrated': False, 'confidence_grain': 'none', 'has_real_provenance': False, 'tables': False}}* Engine id -> capability profile. Keys are `ocracy`’s canonical (hyphenated) backend ids; [`profile()`](_autosummary/ek.ocr.profiles.html.md#ek.ocr.profiles.profile) also accepts underscore variants. ### ek.ocr.profiles.profile(engine) Capability profile for an engine id (a safe default if unknown). Accepts `ocracy`’s canonical hyphenated ids and their underscore variants. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) # _autosummary/ek.ocr.tables.html.md # ek.ocr.tables Recover a normalized table from an `OcrResult` so ek’s table metrics can score it. `ocracy` exposes **no** normalized table/cell type: `TextBlock.level` stops at `block`, with no cell/row/column model, so OCR table structure survives only inside the engine-specific `OcrResult.raw` (Textract `Block` objects, PaddleOCR `pred_html`, Azure `tables[]`) or in the markdown/HTML an engine emits (`res.markdown` for VLM engines). ek therefore can’t uniformly score table structure (TEDS / GriTS) off the `OcrResult` shape alone. This module is the **ek-side seam** that closes that gap *without touching ocracy* (the dependency arrow stays `ek -> ocracy`, never the reverse). It turns an `OcrResult`-shaped object into the existing [`ek.metrics.tables.Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table) (the one TEDS/GriTS already score) – it does **not** define a new table type. Because every engine stores tables differently, the per-engine extractor is an **injected** callable (open-closed / dependency-injection): pass `parser=` a `(raw) -> Table | grid | html | None` function, or register one under the `"table_parsers"` namespace and pass its name. With no parser, [`table_from_ocr()`](_autosummary/ek.ocr.tables.html.md#ek.ocr.tables.table_from_ocr) falls back to a few **safe, documented** heuristics over the result’s `raw`/`markdown`/`meta` and returns `None` when no table structure is present – it never guesses wildly. Nothing here imports `ocracy` or any OCR engine: ek depends only on the `OcrResult` *shape* (`.text` / `.raw` / `.markdown` / `.meta`), so this works on any `image -> OcrResult` callable’s output. ### Example ```pycon >>> from ek.metrics.tables import Table >>> class R: # a minimal OcrResult-shaped object carrying a 2-D grid in .raw ... raw = {"tables": [[["a", "b"], ["c", "d"]]]} >>> t = table_from_ocr(R()) >>> [[c.text for c in row] for row in t.rows] [['a', 'b'], ['c', 'd']] >>> has_table_structure(R()) True >>> class Empty: # no table anywhere in the result ... raw = {"text": "just prose"} >>> table_from_ocr(Empty()) is None True ``` A registered third-party engine parser, used by name: ```pycon >>> from ek.registry import register >>> @register("table_parsers", "demo") ... def _demo(raw): ... return raw.get("cells") # this engine stores a grid under raw['cells'] >>> class D: ... raw = {"cells": [["x"]]} >>> table_from_ocr(D(), parser="demo").rows[0][0].text 'x' ``` ### Module Attributes | [`TABLE_PARSERS_NAMESPACE`](_autosummary/ek.ocr.tables.html.md#ek.ocr.tables.TABLE_PARSERS_NAMESPACE) | Registry namespace under which per-engine table parsers live. | |----------------------------------------------------------------------------|-----------------------------------------------------------------| ### Functions | [`engine_yields_tables`](_autosummary/ek.ocr.tables.html.md#ek.ocr.tables.engine_yields_tables)(engine) | Whether `engine`'s capability profile claims scoreable table structure. | |------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------| | [`has_table_structure`](_autosummary/ek.ocr.tables.html.md#ek.ocr.tables.has_table_structure)(ocr_result, \*[, parser]) | Whether [`table_from_ocr()`](_autosummary/ek.ocr.tables.html.md#ek.ocr.tables.table_from_ocr) recovers a non-empty table from `ocr_result`. | | [`resolve_table_parser`](_autosummary/ek.ocr.tables.html.md#ek.ocr.tables.resolve_table_parser)(ref) | Resolve a table parser reference (a registered name or a callable) to a callable. | | [`table_from_ocr`](_autosummary/ek.ocr.tables.html.md#ek.ocr.tables.table_from_ocr)(ocr_result, \*[, parser]) | Extract a normalized [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table) from an `OcrResult`. | ### ek.ocr.tables.TABLE_PARSERS_NAMESPACE *= 'table_parsers'* Registry namespace under which per-engine table parsers live. Register a parser with `ek.registry.register("table_parsers", "", fn)` (or an `ek.table_parsers` entry point) and resolve it by name via the `parser=` arg. ### ek.ocr.tables.engine_yields_tables(engine) Whether `engine`’s capability profile claims scoreable table structure. A convenience over [`ek.ocr.profile()`](_autosummary/ek.ocr.html.md#ek.ocr.profile): returns the engine profile’s `tables` flag. ek treats the profile as a *prior* (Azure DI, AWS Textract, PaddleOCR and the markdown engines are expected to yield scoreable tables; Google Vision is not), to be verified empirically by actually running [`table_from_ocr()`](_autosummary/ek.ocr.tables.html.md#ek.ocr.tables.table_from_ocr). Use it to skip table scoring for engines that cannot emit cells in the first place. * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.ocr.tables.has_table_structure(ocr_result, , parser=None) Whether [`table_from_ocr()`](_autosummary/ek.ocr.tables.html.md#ek.ocr.tables.table_from_ocr) recovers a non-empty table from `ocr_result`. A thin predicate over [`table_from_ocr()`](_autosummary/ek.ocr.tables.html.md#ek.ocr.tables.table_from_ocr) (same `parser` semantics): true iff a table with at least one cell can be recovered. Use it to route an OCR result to the table metrics vs the text-only CER/WER fallback. * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.ocr.tables.resolve_table_parser(ref) Resolve a table parser reference (a registered name or a callable) to a callable. Mirrors how other ek strategies resolve from the registry (see [`ek.registry.resolve()`](_autosummary/ek.registry.html.md#ek.registry.resolve)). A `str` is looked up under the `"table_parsers"` namespace; an already-callable parser is returned unchanged. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] ### ek.ocr.tables.table_from_ocr(ocr_result, , parser=None) Extract a normalized [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table) from an `OcrResult`. The single seam ek’s table metrics (TEDS / GriTS) use to score OCR table output uniformly. Returns the recovered Table, or `None` when the result carries no recoverable table structure. * **Parameters:** * **ocr_result** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Any `OcrResult`-shaped object. ek reads only its `.raw` (the engine-specific payload), `.markdown` (VLM/markdown engines), and `.meta` – never ocracy types, so any `image -> OcrResult` callable’s output works. * **parser** (`Union`[[`None`](https://docs.python.org/3/builtins/constants.html#None), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]]) – How to turn the raw payload into a table (dependency injection / open-closed). One of: - `None` (default): try the built-in **safe heuristics** – `.raw` is already a [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table), a 2-D grid, or a `
...
` HTML string; or `.raw`/`.meta` is a mapping carrying one of `{tables, table, cells, grid, html}`; or `.markdown` contains an HTML ``. Returns `None` if none match – it never guesses wildly. - a `str`: the name of a parser registered under `"table_parsers"` (per-engine extractor), resolved from [`ek.registry`](_autosummary/ek.registry.html.md#module-ek.registry). - a callable `(raw) -> Table | grid | html | None`: an inline per-engine extractor. It receives the result’s `.raw` (falling back to the result itself when there is no `.raw`). * **through** (*Any non-None value an injected parser returns is fed*) * **Return type:** [*Table*](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table) | *None* :param `Table.coerce()`: :param so a parser may yield a Table: :param a 2-D grid: :param or HTML.: * **Return type:** [`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table)] * **Returns:** A non-empty [`Table`](_autosummary/ek.metrics.tables.html.md#ek.metrics.tables.Table), or `None` when no table structure is recoverable. An empty Table (no rows/cells) is normalized to `None` so the predicate [`has_table_structure()`](_autosummary/ek.ocr.tables.html.md#ek.ocr.tables.has_table_structure) and downstream metrics see “no table”, not a degenerate one. # _autosummary/ek.qe.calibrate.html.md # ek.qe.calibrate Calibration: make a raw score *mean* a probability (the non-optional stage). A confidence of 0.9 should mean “correct 90% of the time.” Modern models violate this badly – they are systematically **overconfident** (Guo et al. 2017), and RLHF makes LLMs worse. So \*\*calibration is non-optional: never gate a raw posterior or a raw logprob\*\* (`misc/docs/ek_03` §2, Hard Rule 1). A [`Calibrator`](_autosummary/ek.base.html.md#ek.base.Calibrator) is fit on a labelled holdout of `(raw_score, field_correct?)` pairs and maps any later raw score to a calibrated probability. Three methods, by what input you have: - [`PlattCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.PlattCalibrator) – logistic fit on any scalar score (no logits needed): **the default** for aggregated OCR confidence or aggregated logprobs. - [`IsotonicCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.IsotonicCalibrator) – non-parametric monotonic fit; more flexible, needs more data, can overfit small sets. - [`TemperatureCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.TemperatureCalibrator) – a single scalar `T` on **logits**; use only when you have logits (it does not change the argmax). All three are pure-Python (stdlib only) so the calibration stage works with zero extra dependencies; [`sklearn_calibrator()`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.sklearn_calibrator) / `netcal_calibrator()` offer the library-backed equivalents behind the `ek[calibration]` extra. Measure calibration with [`expected_calibration_error()`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.expected_calibration_error) (+ a reliability curve). For per-field-type validity, wrap per group with [`GroupCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.GroupCalibrator) (Mondrian / class-conditional) – distribution-free *conditional* coverage is otherwise impossible (Hard Rule 2). When the raw signals are *heterogeneous* (a judge score next to a logprob next to a ROVER agreement), calibrate each signal family on its own with [`SignalFamilyCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.SignalFamilyCalibrator) and fuse the calibrated probabilities with a documented combiner ([`mean_probability()`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.mean_probability) by default, [`min_probability()`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.min_probability) for a conservative gate), rather than calibrating the mean of incomparable scores. Calibrate at the **granularity of the decision** (gate on fields -> calibrate a “field-correct?” target), and persist the fit ([`save_calibrator()`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.save_calibrator)); calibration is dataset-specific and decays, so re-fit on drift. ### Example ```pycon >>> # An overconfident raw signal, calibrated against ground truth. >>> raw = [0.95, 0.93, 0.92, 0.90, 0.55, 0.52, 0.51, 0.50] >>> correct = [True, True, False, False, True, False, False, False] >>> cal = PlattCalibrator().fit(raw, correct) >>> cal(0.95) < 0.95 # overconfidence pulled down True >>> 0.0 <= cal(0.5) <= 1.0 True ``` ### Module Attributes | [`DEFAULT_N_BINS`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.DEFAULT_N_BINS) | Default number of bins for [`expected_calibration_error()`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.expected_calibration_error) / reliability curves. | |-------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------| ### Functions | [`expected_calibration_error`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.expected_calibration_error)(probs, correct, \*) | Expected Calibration Error: weighted mean gap between confidence and accuracy. | |---------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------| | [`load_calibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.load_calibrator)(name, \*[, rootdir]) | Reconstruct a persisted calibrator by name (dispatched on its `kind`). | | [`mean_probability`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.mean_probability)(probs) | Fuse calibrated per-family probabilities by their mean (the default combiner). | | [`min_probability`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.min_probability)(probs) | Fuse calibrated per-family probabilities by their minimum (conservative). | | [`netcal_ece`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.netcal_ece)(probs, correct, \*[, bins]) | ECE via `netcal` (behind `ek[calibration]`); D-ECE for localized outputs lives there too. | | [`reliability_curve`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.reliability_curve)(probs, correct, \*[, n_bins]) | Per-bin `{confidence, accuracy, count}` for a reliability diagram. | | [`save_calibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.save_calibrator)(calibrator, name, \*[, rootdir]) | Persist a fitted calibrator's parameters to the `calibrators` store. | | [`sklearn_calibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.sklearn_calibrator)([method]) | A calibrator backed by scikit-learn (`method='sigmoid'` Platt or `'isotonic'`). | ### Classes | [`GroupCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.GroupCalibrator)([factory, by_group, pooled, ...]) | Per-group (Mondrian) calibration: one calibrator per `NodeType`/`FieldSpec`. | |------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------| | [`IsotonicCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.IsotonicCalibrator)([x, y, kind]) | Isotonic (monotonic non-decreasing) calibration via pool-adjacent-violators. | | [`PlattCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.PlattCalibrator)([a, b, max_iter, kind]) | Platt scaling: `sigmoid(a * score + b)`, fit by Newton/IRLS on labels. | | [`SignalFamilyCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.SignalFamilyCalibrator)([calibrators, ...]) | One calibrator per signal family, fused by a documented combination rule. | | [`TemperatureCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.TemperatureCalibrator)([T, kind, t_min, ...]) | Temperature scaling: `sigmoid(logit / T)` with one `T` fit on a holdout. | ### ek.qe.calibrate.DEFAULT_N_BINS *= 10* Default number of bins for [`expected_calibration_error()`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.expected_calibration_error) / reliability curves. ### *class* ek.qe.calibrate.GroupCalibrator(factory=, by_group=, pooled=None, kind='group') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Per-group (Mondrian) calibration: one calibrator per `NodeType`/`FieldSpec`. Distribution-free *conditional* (per-field-type) coverage is impossible in general (Barber et al. 2019); calibrating separately per group restores it approximately. `fit` takes a parallel `groups` sequence; `__call__` routes by group key, falling back to a pooled calibrator for unseen groups. * **Parameters:** **factory** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – Zero-arg callable producing a fresh per-group calibrator (default [`PlattCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.PlattCalibrator)). #### factory alias of [`PlattCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.PlattCalibrator) #### fit(scores, correct, , groups) Fit one calibrator per distinct group key, plus a pooled fallback. * **Return type:** [`GroupCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.GroupCalibrator) #### to_dict() Serialize each per-group calibrator (keyed by stringified group) plus the pooled fallback, so a fitted Mondrian calibrator round-trips like the others. Group keys are stringified (they are `NodeType`/`FieldSpec` names). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### *class* ek.qe.calibrate.IsotonicCalibrator(x=, y=, kind='isotonic') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Isotonic (monotonic non-decreasing) calibration via pool-adjacent-violators. More flexible than Platt; needs more calibration data and can overfit small sets. Predicts by linear interpolation between fitted points, clipped at the ends. #### fit(scores, correct) Fit the monotonic step function to `(score, correct)` pairs. * **Return type:** [`IsotonicCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.IsotonicCalibrator) ### *class* ek.qe.calibrate.PlattCalibrator(a=1.0, b=0.0, max_iter=100, kind='platt') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Platt scaling: `sigmoid(a * score + b)`, fit by Newton/IRLS on labels. The default calibrator: works on any scalar (aggregated confidence/logprob), no logits required. Uses Platt’s target smoothing so it does not overfit small calibration sets. * **Parameters:** **max_iter** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Newton iterations (converges in a handful for a 2-parameter model). #### fit(scores, correct) Fit `a, b` to maximise the likelihood of `correct` given `scores`. * **Return type:** [`PlattCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.PlattCalibrator) ### *class* ek.qe.calibrate.SignalFamilyCalibrator(calibrators=, combine=, factory=, fusion=None, kind='signal_family') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One calibrator per signal family, fused by a documented combination rule. Averaging heterogeneous raw signals (an LLM-judge score, a token logprob, a ROVER agreement) and calibrating the mean discards exactly what a calibrator needs: each family has its own scale, reliability and direction of miscalibration. This calibrates every signal *by name* with its own calibrator, then fuses the calibrated probabilities with `combine`. Pass it (or simply a `Mapping[str, Calibrator]`) as `estimate_quality(..., calibrator=...)`. **The fused value must itself be calibrated before it is gated.** Fusing calibrated probabilities (by mean, min, …) does not yield a calibrated probability: agreeing families make the mean under-confident, a weak family drags it toward the base rate. So [`fit()`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.SignalFamilyCalibrator.fit) also learns a second-stage `fusion` calibrator on `combine`’s output, and `__call__` applies it. Without a `fusion` calibrator (e.g. a hand-built mapping), [`ek.estimate_quality()`](_autosummary/ek.html.md#ek.estimate_quality) refuses to gate a value fused from two or more families unless told `assume_calibrated=True` (Hard Rule 1). A value from a single family is that family’s calibrated probability and gates as is. A field’s own upstream `confidence` is just one more family here, named `"confidence"`: include it in the training rows under that name. * **Parameters:** * **calibrators** ([`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)) – `signal name -> Calibrator`. A `"*"` entry is the fallback for any signal without its own entry. A signal with neither is refused (`ValueError`) rather than silently dropped or passed through raw (Hard Rule 1). * **combine** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`float`](https://docs.python.org/3/builtins/functions.html#float)]], [`float`](https://docs.python.org/3/builtins/functions.html#float)]) – `Mapping[str, float] -> float` fusing the calibrated values; [`mean_probability()`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.mean_probability) (default) or [`min_probability()`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.min_probability), or any callable of that shape (e.g. a learned stacker). * **factory** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – Zero-arg callable producing a fresh calibrator, used by [`fit()`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.SignalFamilyCalibrator.fit). * **fusion** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Second-stage calibrator applied to `combine`’s output (learned by [`fit()`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.SignalFamilyCalibrator.fit)); `None` leaves the fused value uncalibrated. ### Example ```pycon >>> rows = [{"judge": j, "logprob": l} for j, l in ... [(0.9, -0.1), (0.8, -0.2), (0.85, -2.0), (0.3, -0.1), (0.2, -2.5), (0.7, -1.8)]] >>> correct = [True, True, False, True, False, False] >>> cal = SignalFamilyCalibrator().fit(rows, correct) >>> sorted(cal.calibrators) ['judge', 'logprob'] >>> 0.0 <= cal({"judge": 0.9, "logprob": -0.1}) <= 1.0 True ``` #### calibrate_signals(raw_signals, , group=None) `{name: calibrated probability}` for each raw signal (before fusion). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) #### combine() Fuse calibrated per-family probabilities by their mean (the default combiner). Each family’s calibrated value estimates the *same* event (“this field is correct”), so the mean is a defensible, assumption-light pooling rule. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ```pycon >>> mean_probability({"judge": 0.9, "logprob": 0.7}) 0.8 ``` #### factory alias of [`PlattCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.PlattCalibrator) #### fit(signals, correct) Fit one calibrator per signal name, then the `fusion` calibrator. Each family is fit over the rows where it appears with a finite value. A family with fewer than two such rows, or only one label class, cannot be calibrated and is refused (`ValueError`) rather than turned into a constant that would then dilute the fusion. * **Return type:** [`SignalFamilyCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.SignalFamilyCalibrator) #### fuse(calibrated) Fuse per-family calibrated probabilities: `combine`, then `fusion`. A single family’s value is returned as is: it is already calibrated, and `fusion` was learned on fused values. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) #### to_dict() Serialize every family’s calibrator and the (registered) combiner’s name. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### *class* ek.qe.calibrate.TemperatureCalibrator(T=1.0, kind='temperature', t_min=0.05, t_max=10.0, max_iter=60) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Temperature scaling: `sigmoid(logit / T)` with one `T` fit on a holdout. Use **only when you have logits**: `__call__` expects a *logit*, not a probability. `T > 1` softens overconfidence; the argmax is unchanged. #### fit(logits, correct) Fit `T` by minimising NLL with a bounded 1-D search. * **Return type:** [`TemperatureCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.TemperatureCalibrator) #### t_min *: [float](https://docs.python.org/3/builtins/functions.html#float)* *= 0.05* Golden-section search bounds and iteration count (config, not magic numbers). ### ek.qe.calibrate.expected_calibration_error(probs, correct, , n_bins=10) Expected Calibration Error: weighted mean gap between confidence and accuracy. Bins predictions by confidence into `n_bins` equal-width bins and averages `|mean_confidence - accuracy|` weighted by bin population. `0` is perfect. Non-finite probs are skipped; out-of-range probs are clamped into `[0, 1]`. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.qe.calibrate.load_calibrator(name, , rootdir=None) Reconstruct a persisted calibrator by name (dispatched on its `kind`). Validates the stored record so a malformed or unknown `kind` fails with an actionable error rather than a raw `KeyError` or a load-then-crash-later. * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) ### ek.qe.calibrate.mean_probability(probs) Fuse calibrated per-family probabilities by their mean (the default combiner). Each family’s calibrated value estimates the *same* event (“this field is correct”), so the mean is a defensible, assumption-light pooling rule. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ```pycon >>> mean_probability({"judge": 0.9, "logprob": 0.7}) 0.8 ``` ### ek.qe.calibrate.min_probability(probs) Fuse calibrated per-family probabilities by their minimum (conservative). Use when any one family flagging a problem should be enough to lower the gate. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ```pycon >>> min_probability({"judge": 0.9, "logprob": 0.7}) 0.7 ``` ### ek.qe.calibrate.netcal_ece(probs, correct, , bins=10) ECE via `netcal` (behind `ek[calibration]`); D-ECE for localized outputs lives there too. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.qe.calibrate.reliability_curve(probs, correct, , n_bins=10) Per-bin `{confidence, accuracy, count}` for a reliability diagram. * **Return type:** [`List`](https://docs.python.org/3/library/typing.html#typing.List)[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)] ### ek.qe.calibrate.save_calibrator(calibrator, name, , rootdir=None) Persist a fitted calibrator’s parameters to the `calibrators` store. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### ek.qe.calibrate.sklearn_calibrator(method='sigmoid') A calibrator backed by scikit-learn (`method='sigmoid'` Platt or `'isotonic'`). Behind `ek[calibration]`. Returns an object satisfying the [`Calibrator`](_autosummary/ek.base.html.md#ek.base.Calibrator) protocol that wraps sklearn’s calibration. The pure-Python [`PlattCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.PlattCalibrator)/[`IsotonicCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.IsotonicCalibrator) are the dependency-free defaults; use this for parity with an sklearn-centric stack. # _autosummary/ek.qe.decide.html.md # ek.qe.decide Selective prediction: turn a calibrated probability into accept / flag / block. Calibration makes a score *mean* something; a **decision policy** turns it into an action with a guarantee (`misc/docs/ek_03` §3-4). This is the formal home of “accept / flag / block” – a selective predictor that abstains (routes to a human) rather than emit a low-confidence answer. Every policy here reads \*\*only calibrated probabilities\*\* (Hard Rule 1): a raw score must already have passed through a [`Calibrator`](_autosummary/ek.base.html.md#ek.base.Calibrator). Three policies, by what you want to guarantee: - [`CostSensitiveGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.CostSensitiveGate) – **the default**. The accept threshold falls out of the cost ratio `rho = c_FN / c_FP` on the calibrated probability; no labelled set needed once you know your costs. - [`ConformalGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.ConformalGate) – split (inductive) conformal: a finite-sample, distribution-free **marginal** guarantee that at most `alpha` of truly-correct items are flagged. Per-field-type validity needs the Mondrian ([`GroupConformalGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.GroupConformalGate)) variant – conditional coverage is otherwise impossible (Barber et al. 2019). - [`RiskControlGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.RiskControlGate) – conformal risk control: bound a monotone loss (the rate of accepted-yet-wrong items) at a target level. [`risk_coverage_curve()`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.risk_coverage_curve) exposes the trade-off so stakeholders pick the operating point explicitly. The pure-Python gates need no extra dependencies; the `ek[calibration]` extra (MAPIE/crepes) offers library-backed conformal for an sklearn-centric stack. Exchangeability is the load-bearing assumption – track realised coverage and re-fit on drift. ### Example ```pycon >>> from ek.base import Decision >>> gate = CostSensitiveGate(rho=9.0) # a false-negative costs 9x a review >>> gate(0.95) is Decision.ACCEPT # tau = rho/(1+rho) = 9/10 = 0.9 True >>> gate(0.5) is Decision.FLAG True ``` ### Module Attributes | [`DEFAULT_ALPHA`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.DEFAULT_ALPHA) | Default target miscoverage for conformal gates (flag at most ~alpha of correct items). | |----------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------| | [`DEFAULT_RISK_TARGET`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.DEFAULT_RISK_TARGET) | Default risk bound for [`RiskControlGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.RiskControlGate) (accepted-error rate ceiling). | ### Functions | [`crepes_mondrian_gate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.crepes_mondrian_gate)(\*args, \*\*kwargs) | A Mondrian (class-conditional) conformal gate backed by `crepes` (`ek[calibration]`). | |-------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------| | [`risk_coverage_curve`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.risk_coverage_curve)(probs, correct, \*[, ...]) | The risk-coverage trade-off: `{threshold, coverage, selective_risk}` per point. | | [`split_conformal_quantile`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.split_conformal_quantile)(scores, alpha) | The split-conformal threshold: the `ceil((n+1)(1-alpha)) / n` empirical quantile. | ### Classes | [`ConformalGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.ConformalGate)([alpha, nonconformity, q, \_cal]) | Split-conformal accept/flag with a finite-sample **marginal** guarantee. | |----------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------| | [`CostSensitiveGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.CostSensitiveGate)([rho, block_threshold, ...]) | Accept/flag/block from the cost ratio `rho = c_FN / c_FP` on calibrated `p`. | | [`GroupConformalGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.GroupConformalGate)([alpha, nonconformity, ...]) | Mondrian (class-conditional) conformal: one [`ConformalGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.ConformalGate) per group. | | [`RiskControlGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.RiskControlGate)([target, loss, loss_bound, lam]) | Conformal Risk Control: accept `p >= lambda` bounding the accepted-error rate. | ### *class* ek.qe.decide.ConformalGate(alpha=0.1, nonconformity=>, q=inf, \_cal=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Split-conformal accept/flag with a finite-sample **marginal** guarantee. Fit on `(calibrated_prob, correct)`: the nonconformity scores of the *correct* calibration items set a threshold `q` such that, among truly-correct test items exchangeable with them, at most `alpha` are flagged. The guarantee is *marginal*, not per-field-type – use [`GroupConformalGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.GroupConformalGate) for that. * **Parameters:** * **alpha** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Target false-flag rate on correct items (e.g. `0.1`). * **nonconformity** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`float`](https://docs.python.org/3/builtins/functions.html#float)], [`float`](https://docs.python.org/3/builtins/functions.html#float)]) – Maps calibrated confidence -> nonconformity (default `1 - confidence`: low confidence is nonconforming). #### fit(probs, correct) Fit `q` from the nonconformity scores of the *correct* calibration items. Non-finite confidences are dropped (a single `NaN` would otherwise sort to the end and corrupt the quantile, silently breaking the coverage guarantee). * **Return type:** [`ConformalGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.ConformalGate) #### p_value(confidence) Per-instance conformal p-value (small = atypical of correct items). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.qe.decide.CostSensitiveGate(rho=1.0, block_threshold=0.0, accept_threshold=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Accept/flag/block from the cost ratio `rho = c_FN / c_FP` on calibrated `p`. Accept when the expected cost of accepting is below that of routing to a human. Accepting risks an undetected error: cost `(1 - p) * c_FN`. Flagging a value that was actually *correct* wastes a review: cost `p * c_FP` (the false-positive cost is incurred only with probability `p`, not unconditionally). So accept iff `(1 - p) * c_FN <= p * c_FP`, i.e. the Bayes-optimal threshold `p >= c_FN / (c_FN + c_FP) = rho / (1 + rho)`. A larger `rho` (misses much costlier than reviews) raises the bar to auto-accept (`tau -> 1` as `rho -> inf`); `rho = 1` gives the symmetric `tau = 0.5`. `block_threshold` adds an optional hard-fail band at the bottom. * **Parameters:** * **rho** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Cost ratio `c_FN / c_FP` (>= 0). The single lever; no magic numbers. * **block_threshold** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Calibrated probability at or below which to `block` rather than `flag` (default `0` -> never block on probability alone; a failed verifier is what forces a block, upstream). * **accept_threshold** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`float`](https://docs.python.org/3/builtins/functions.html#float)]) – Override the derived accept threshold explicitly. #### *property* tau *: [float](https://docs.python.org/3/builtins/functions.html#float)* The accept threshold on calibrated probability (derived from `rho`). ### ek.qe.decide.DEFAULT_ALPHA *= 0.1* Default target miscoverage for conformal gates (flag at most ~alpha of correct items). ### ek.qe.decide.DEFAULT_RISK_TARGET *= 0.05* Default risk bound for [`RiskControlGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.RiskControlGate) (accepted-error rate ceiling). ### *class* ek.qe.decide.GroupConformalGate(alpha=0.1, nonconformity=>, by_group=, pooled=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Mondrian (class-conditional) conformal: one [`ConformalGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.ConformalGate) per group. Restores approximate per-field-type validity by calibrating separately per group key (e.g. `NodeType`/`FieldSpec`), with a pooled fallback for unseen groups. ### *class* ek.qe.decide.RiskControlGate(target=0.05, loss=>, loss_bound=1.0, lam=1.0) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Conformal Risk Control: accept `p >= lambda` bounding the accepted-error rate. Picks the *smallest* threshold `lambda` (the most coverage) whose finite-sample risk bound is at or below `target`. The controlled quantity is `E[ loss(item) * 1(accepted) ]` – with the default 0/1 loss, the population rate of accepted-yet-wrong items. * **Parameters:** * **target** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Upper bound on the accepted-error rate (e.g. `0.05`). * **loss** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`bool`](https://docs.python.org/3/builtins/functions.html#bool)], [`float`](https://docs.python.org/3/builtins/functions.html#float)]) – `correct -> loss` in `[0, loss_bound]`; default `0` if correct else `1`. Any monotone loss bounded by `loss_bound` works. * **loss_bound** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – The loss’s upper bound `B` – the CRC finite-sample slack is `+B/(n+1)`. Must match `loss`’s range or the guarantee is invalid; keep the default `1.0` for the 0/1 loss. #### fit(probs, correct) Find the smallest accept threshold whose risk bound is <= `target`. * **Return type:** [`RiskControlGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.RiskControlGate) ### ek.qe.decide.crepes_mondrian_gate(\*args, \*\*kwargs) A Mondrian (class-conditional) conformal gate backed by `crepes` (`ek[calibration]`). The pure-Python [`GroupConformalGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.GroupConformalGate) is the dependency-free default; use this for parity with a crepes/sklearn conformal stack (full CDFs, normalised variants). Raises an actionable install hint if `crepes` is absent. ### ek.qe.decide.risk_coverage_curve(probs, correct, , thresholds=None) The risk-coverage trade-off: `{threshold, coverage, selective_risk}` per point. Coverage is the fraction auto-accepted at each threshold; selective risk is the error rate *among the accepted*. Lower-left is better; pick the operating point against a target risk or coverage. * **Return type:** [`List`](https://docs.python.org/3/library/typing.html#typing.List)[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)] ### ek.qe.decide.split_conformal_quantile(scores, alpha) The split-conformal threshold: the `ceil((n+1)(1-alpha)) / n` empirical quantile. `scores` are nonconformity scores on an exchangeable calibration set. Returns `+inf` when `n` is too small to guarantee `1 - alpha` (so nothing is flagged – the honest behaviour at that sample size), and `-inf` at the `alpha -> 1` limit (flag everything). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) # _autosummary/ek.qe.html.md # ek.qe Reference-free quality estimation: the `estimate_quality()` side of `ek`. This package implements the online (no-gold) half of evaluation as the strict pipeline `signal -> calibrate -> validate -> decide` (`misc/docs/ek_03`): - **signals** (cheapest first): the deterministic [`VerifierSignal`](_autosummary/ek.qe.verifiers.html.md#ek.qe.verifiers.VerifierSignal) (schema/checksum/cross-field, cost tier 1, runs always), free intrinsic [`IntrinsicConfidenceSignal`](_autosummary/ek.qe.signals.html.md#ek.qe.signals.IntrinsicConfidenceSignal) / [`LogprobSignal`](_autosummary/ek.qe.signals.html.md#ek.qe.signals.LogprobSignal) > (tier 2), and the flagship ROVER [`AgreementSignal`](_autosummary/ek.qe.html.md#ek.qe.AgreementSignal) multi-engine > vote (tier 3); - **calibrate** – [`PlattCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.PlattCalibrator) (default) / isotonic / temperature, with Mondrian grouping and ECE diagnostics: \*non-optional before any gate\*; - **decide** – [`CostSensitiveGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.CostSensitiveGate) (default), [`ConformalGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.ConformalGate), and [`RiskControlGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.RiskControlGate), > reading only calibrated probabilities. Everything is pure-Python and dependency-free by default; the `ek[calibration]` (netcal/sklearn/MAPIE/crepes) and `ek[agreement]` (uqlm) extras supply library-backed equivalents, imported lazily so `import ek` stays light. Importing this package registers the built-in strategies (signals, aggregators, checks, calibrators, policies) by name in [`ek.registry`](_autosummary/ek.registry.html.md#module-ek.registry). ### Functions | [`rover`](_autosummary/ek.qe.html.md#ek.qe.rover)(hypotheses, \*[, use_confidence, ...]) | Align N hypotheses, vote per slot, and emit consensus + per-position agreement. | |----------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------| | [`geo_mean`](_autosummary/ek.qe.html.md#ek.qe.geo_mean)(logps) | Geometric mean of token probabilities, `exp(mean log p)` (perplexity-inverse). | | [`length_normalized`](_autosummary/ek.qe.html.md#ek.qe.length_normalized)(logps, \*[, alpha]) | Length-normalised score `exp(Σ log p / T**alpha)` (tunable length penalty). | | [`min_prob`](_autosummary/ek.qe.html.md#ek.qe.min_prob)(logps) | Weakest-token probability `exp(min log p)` -- catches one bad character. | | [`mean_prob`](_autosummary/ek.qe.html.md#ek.qe.mean_prob)(logps) | Arithmetic mean of token probabilities `mean(exp(log p))`. | | [`mean_probability`](_autosummary/ek.qe.html.md#ek.qe.mean_probability)(probs) | Fuse calibrated per-family probabilities by their mean (the default combiner). | | [`min_probability`](_autosummary/ek.qe.html.md#ek.qe.min_probability)(probs) | Fuse calibrated per-family probabilities by their minimum (conservative). | | [`expected_calibration_error`](_autosummary/ek.qe.html.md#ek.qe.expected_calibration_error)(probs, correct, \*) | Expected Calibration Error: weighted mean gap between confidence and accuracy. | | [`reliability_curve`](_autosummary/ek.qe.html.md#ek.qe.reliability_curve)(probs, correct, \*[, n_bins]) | Per-bin `{confidence, accuracy, count}` for a reliability diagram. | | [`save_calibrator`](_autosummary/ek.qe.html.md#ek.qe.save_calibrator)(calibrator, name, \*[, rootdir]) | Persist a fitted calibrator's parameters to the `calibrators` store. | | [`load_calibrator`](_autosummary/ek.qe.html.md#ek.qe.load_calibrator)(name, \*[, rootdir]) | Reconstruct a persisted calibrator by name (dispatched on its `kind`). | | [`sklearn_calibrator`](_autosummary/ek.qe.html.md#ek.qe.sklearn_calibrator)([method]) | A calibrator backed by scikit-learn (`method='sigmoid'` Platt or `'isotonic'`). | | [`risk_coverage_curve`](_autosummary/ek.qe.html.md#ek.qe.risk_coverage_curve)(probs, correct, \*[, ...]) | The risk-coverage trade-off: `{threshold, coverage, selective_risk}` per point. | | [`split_conformal_quantile`](_autosummary/ek.qe.html.md#ek.qe.split_conformal_quantile)(scores, alpha) | The split-conformal threshold: the `ceil((n+1)(1-alpha)) / n` empirical quantile. | | [`checksum_validator`](_autosummary/ek.qe.html.md#ek.qe.checksum_validator)([kind, field_name, layer]) | A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a value failing a named checksum. | | [`regex_validator`](_autosummary/ek.qe.html.md#ek.qe.regex_validator)(pattern, \*[, field_name, layer]) | A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a value not fully matching `pattern`. | | [`range_validator`](_autosummary/ek.qe.html.md#ek.qe.range_validator)(lo, hi, \*[, field_name, layer]) | A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a number outside `[lo, hi]`. | | [`enum_validator`](_autosummary/ek.qe.html.md#ek.qe.enum_validator)(members, \*[, field_name, layer]) | A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a value not in an allowed set. | | [`schema_validator`](_autosummary/ek.qe.html.md#ek.qe.schema_validator)(value, \*[, spec]) | A spec-driven [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator): type and `domain` checks from a FieldSpec. | | [`totals_consistent`](_autosummary/ek.qe.html.md#ek.qe.totals_consistent)(record, \*, total_key, ...) | Flag when `record[total_key]` does not equal the sum of `item_keys` (±tol). | | [`luhn_check`](_autosummary/ek.qe.html.md#ek.qe.luhn_check)(number) | Luhn (mod-10) checksum: credit cards, IMEIs, some national IDs. | | [`iban_check`](_autosummary/ek.qe.html.md#ek.qe.iban_check)(iban) | IBAN validity via the ISO 13616 / ISO 7064 mod-97 rule (must equal 1). | | [`isbn_check`](_autosummary/ek.qe.html.md#ek.qe.isbn_check)(value) | Validate an ISBN-10 or ISBN-13 by length (digits/hyphens/spaces ignored). | ### Classes | [`RoverConsensus`](_autosummary/ek.qe.html.md#ek.qe.RoverConsensus)([tokens, slots, agreement, ...]) | The result of a ROVER pass over N hypotheses. | |----------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------| | [`RoverSlot`](_autosummary/ek.qe.html.md#ek.qe.RoverSlot)([entries, winner, score, vote_share]) | One branch point of the transition network: each engine's token (or NULL). | | [`AgreementSignal`](_autosummary/ek.qe.html.md#ek.qe.AgreementSignal)([cost_tier, use_confidence, ...]) | ROVER multi-engine agreement as a reference-free [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal). | | [`VerifierSignal`](_autosummary/ek.qe.html.md#ek.qe.VerifierSignal)([validators, cost_tier]) | Deterministic verifier evidence as a reference-free [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal). | | [`LogprobSignal`](_autosummary/ek.qe.html.md#ek.qe.LogprobSignal)([aggregator, alpha, cost_tier]) | Aggregate token log-probabilities into one field score (a [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal)). | | [`IntrinsicConfidenceSignal`](_autosummary/ek.qe.html.md#ek.qe.IntrinsicConfidenceSignal)([pool, cost_tier]) | Aggregate an extractor's per-unit confidences into a field score. | | [`PlattCalibrator`](_autosummary/ek.qe.html.md#ek.qe.PlattCalibrator)([a, b, max_iter, kind]) | Platt scaling: `sigmoid(a * score + b)`, fit by Newton/IRLS on labels. | | [`IsotonicCalibrator`](_autosummary/ek.qe.html.md#ek.qe.IsotonicCalibrator)([x, y, kind]) | Isotonic (monotonic non-decreasing) calibration via pool-adjacent-violators. | | [`TemperatureCalibrator`](_autosummary/ek.qe.html.md#ek.qe.TemperatureCalibrator)([T, kind, t_min, ...]) | Temperature scaling: `sigmoid(logit / T)` with one `T` fit on a holdout. | | [`GroupCalibrator`](_autosummary/ek.qe.html.md#ek.qe.GroupCalibrator)([factory, by_group, pooled, ...]) | Per-group (Mondrian) calibration: one calibrator per `NodeType`/`FieldSpec`. | | [`SignalFamilyCalibrator`](_autosummary/ek.qe.html.md#ek.qe.SignalFamilyCalibrator)([calibrators, ...]) | One calibrator per signal family, fused by a documented combination rule. | | [`CostSensitiveGate`](_autosummary/ek.qe.html.md#ek.qe.CostSensitiveGate)([rho, block_threshold, ...]) | Accept/flag/block from the cost ratio `rho = c_FN / c_FP` on calibrated `p`. | | [`ConformalGate`](_autosummary/ek.qe.html.md#ek.qe.ConformalGate)([alpha, nonconformity, q, \_cal]) | Split-conformal accept/flag with a finite-sample **marginal** guarantee. | | [`GroupConformalGate`](_autosummary/ek.qe.html.md#ek.qe.GroupConformalGate)([alpha, nonconformity, ...]) | Mondrian (class-conditional) conformal: one [`ConformalGate`](_autosummary/ek.qe.html.md#ek.qe.ConformalGate) per group. | | [`RiskControlGate`](_autosummary/ek.qe.html.md#ek.qe.RiskControlGate)([target, loss, loss_bound, lam]) | Conformal Risk Control: accept `p >= lambda` bounding the accepted-error rate. | ### *class* ek.qe.AgreementSignal(cost_tier=3, use_confidence=True, conf_weight=0.5, null_conf=0.7, tokenize=None, max_tokens=5000) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) ROVER multi-engine agreement as a reference-free [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal). Cost tier 3 (`N` engine runs): try the deterministic verifier layer and any free intrinsic confidence first. Called on a collection of hypotheses, it runs [`rover()`](_autosummary/ek.qe.html.md#ek.qe.rover) and returns the mean per-position agreement as the raw signal – *uncalibrated*, like every signal, so it must pass through a [`Calibrator`](_autosummary/ek.base.html.md#ek.base.Calibrator) before any gate reads it. ### Example ```pycon >>> sig = AgreementSignal() >>> round(sig(["the cat sat", "the cat sit", "the bat sat"]), 3) 0.778 ``` ### *class* ek.qe.ConformalGate(alpha=0.1, nonconformity=>, q=inf, \_cal=) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Split-conformal accept/flag with a finite-sample **marginal** guarantee. Fit on `(calibrated_prob, correct)`: the nonconformity scores of the *correct* calibration items set a threshold `q` such that, among truly-correct test items exchangeable with them, at most `alpha` are flagged. The guarantee is *marginal*, not per-field-type – use [`GroupConformalGate`](_autosummary/ek.qe.html.md#ek.qe.GroupConformalGate) for that. * **Parameters:** * **alpha** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Target false-flag rate on correct items (e.g. `0.1`). * **nonconformity** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`float`](https://docs.python.org/3/builtins/functions.html#float)], [`float`](https://docs.python.org/3/builtins/functions.html#float)]) – Maps calibrated confidence -> nonconformity (default `1 - confidence`: low confidence is nonconforming). #### fit(probs, correct) Fit `q` from the nonconformity scores of the *correct* calibration items. Non-finite confidences are dropped (a single `NaN` would otherwise sort to the end and corrupt the quantile, silently breaking the coverage guarantee). * **Return type:** [`ConformalGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.ConformalGate) #### p_value(confidence) Per-instance conformal p-value (small = atypical of correct items). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.qe.CostSensitiveGate(rho=1.0, block_threshold=0.0, accept_threshold=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Accept/flag/block from the cost ratio `rho = c_FN / c_FP` on calibrated `p`. Accept when the expected cost of accepting is below that of routing to a human. Accepting risks an undetected error: cost `(1 - p) * c_FN`. Flagging a value that was actually *correct* wastes a review: cost `p * c_FP` (the false-positive cost is incurred only with probability `p`, not unconditionally). So accept iff `(1 - p) * c_FN <= p * c_FP`, i.e. the Bayes-optimal threshold `p >= c_FN / (c_FN + c_FP) = rho / (1 + rho)`. A larger `rho` (misses much costlier than reviews) raises the bar to auto-accept (`tau -> 1` as `rho -> inf`); `rho = 1` gives the symmetric `tau = 0.5`. `block_threshold` adds an optional hard-fail band at the bottom. * **Parameters:** * **rho** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Cost ratio `c_FN / c_FP` (>= 0). The single lever; no magic numbers. * **block_threshold** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Calibrated probability at or below which to `block` rather than `flag` (default `0` -> never block on probability alone; a failed verifier is what forces a block, upstream). * **accept_threshold** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`float`](https://docs.python.org/3/builtins/functions.html#float)]) – Override the derived accept threshold explicitly. #### *property* tau *: [float](https://docs.python.org/3/builtins/functions.html#float)* The accept threshold on calibrated probability (derived from `rho`). ### *class* ek.qe.GroupCalibrator(factory=, by_group=, pooled=None, kind='group') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Per-group (Mondrian) calibration: one calibrator per `NodeType`/`FieldSpec`. Distribution-free *conditional* (per-field-type) coverage is impossible in general (Barber et al. 2019); calibrating separately per group restores it approximately. `fit` takes a parallel `groups` sequence; `__call__` routes by group key, falling back to a pooled calibrator for unseen groups. * **Parameters:** **factory** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – Zero-arg callable producing a fresh per-group calibrator (default [`PlattCalibrator`](_autosummary/ek.qe.html.md#ek.qe.PlattCalibrator)). #### factory alias of [`PlattCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.PlattCalibrator) #### fit(scores, correct, , groups) Fit one calibrator per distinct group key, plus a pooled fallback. * **Return type:** [`GroupCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.GroupCalibrator) #### to_dict() Serialize each per-group calibrator (keyed by stringified group) plus the pooled fallback, so a fitted Mondrian calibrator round-trips like the others. Group keys are stringified (they are `NodeType`/`FieldSpec` names). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### *class* ek.qe.GroupConformalGate(alpha=0.1, nonconformity=>, by_group=, pooled=None) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Mondrian (class-conditional) conformal: one [`ConformalGate`](_autosummary/ek.qe.html.md#ek.qe.ConformalGate) per group. Restores approximate per-field-type validity by calibrating separately per group key (e.g. `NodeType`/`FieldSpec`), with a pooled fallback for unseen groups. ### *class* ek.qe.IntrinsicConfidenceSignal(pool='min', cost_tier=2) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Aggregate an extractor’s per-unit confidences into a field score. * **Parameters:** * **pool** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – `"min"` (weakest unit), `"mean"`, or `"geo_mean"` over the per-unit confidences (already probabilities in `[0, 1]`). * **cost_tier** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – `2` – free (the posteriors are emitted at inference). ### Example ```pycon >>> IntrinsicConfidenceSignal(pool="min")([0.99, 0.4, 0.95]) 0.4 ``` ### *class* ek.qe.IsotonicCalibrator(x=, y=, kind='isotonic') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Isotonic (monotonic non-decreasing) calibration via pool-adjacent-violators. More flexible than Platt; needs more calibration data and can overfit small sets. Predicts by linear interpolation between fitted points, clipped at the ends. #### fit(scores, correct) Fit the monotonic step function to `(score, correct)` pairs. * **Return type:** [`IsotonicCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.IsotonicCalibrator) ### *class* ek.qe.LogprobSignal(aggregator=, alpha=0.6, cost_tier=2) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Aggregate token log-probabilities into one field score (a [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal)). * **Parameters:** * **aggregator** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – A logprob aggregator (name registered under `aggregators` or a callable). Defaults to [`geo_mean()`](_autosummary/ek.qe.html.md#ek.qe.geo_mean). * **alpha** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Length penalty passed to [`length_normalized()`](_autosummary/ek.qe.html.md#ek.qe.length_normalized) only. * **cost_tier** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – `2` – free intrinsic signal (already computed at inference). #### aggregator() Geometric mean of token probabilities, `exp(mean log p)` (perplexity-inverse). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### *class* ek.qe.PlattCalibrator(a=1.0, b=0.0, max_iter=100, kind='platt') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Platt scaling: `sigmoid(a * score + b)`, fit by Newton/IRLS on labels. The default calibrator: works on any scalar (aggregated confidence/logprob), no logits required. Uses Platt’s target smoothing so it does not overfit small calibration sets. * **Parameters:** **max_iter** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Newton iterations (converges in a handful for a 2-parameter model). #### fit(scores, correct) Fit `a, b` to maximise the likelihood of `correct` given `scores`. * **Return type:** [`PlattCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.PlattCalibrator) ### *class* ek.qe.RiskControlGate(target=0.05, loss=>, loss_bound=1.0, lam=1.0) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Conformal Risk Control: accept `p >= lambda` bounding the accepted-error rate. Picks the *smallest* threshold `lambda` (the most coverage) whose finite-sample risk bound is at or below `target`. The controlled quantity is `E[ loss(item) * 1(accepted) ]` – with the default 0/1 loss, the population rate of accepted-yet-wrong items. * **Parameters:** * **target** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Upper bound on the accepted-error rate (e.g. `0.05`). * **loss** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`bool`](https://docs.python.org/3/builtins/functions.html#bool)], [`float`](https://docs.python.org/3/builtins/functions.html#float)]) – `correct -> loss` in `[0, loss_bound]`; default `0` if correct else `1`. Any monotone loss bounded by `loss_bound` works. * **loss_bound** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – The loss’s upper bound `B` – the CRC finite-sample slack is `+B/(n+1)`. Must match `loss`’s range or the guarantee is invalid; keep the default `1.0` for the 0/1 loss. #### fit(probs, correct) Find the smallest accept threshold whose risk bound is <= `target`. * **Return type:** [`RiskControlGate`](_autosummary/ek.qe.decide.html.md#ek.qe.decide.RiskControlGate) ### *class* ek.qe.RoverConsensus(tokens=, slots=, agreement=, n_engines=0) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The result of a ROVER pass over N hypotheses. #### *property* mean_agreement *: [float](https://docs.python.org/3/builtins/functions.html#float)* Mean per-position agreement over consensus tokens, a reference-free confidence in `[0, 1]`: `1.0` means every engine agreed at every emitted position; lower means at least one engine dissented. An **empty** consensus (no token survived voting – NULL won every slot) means the engines agreed on *nothing*, so this returns `0.0` when 2+ engines were fused, NOT a misleading `1.0` (which would feed a maximal raw confidence to the calibrator/gate). A single engine – or none – is vacuously `1.0`. #### *property* text *: [str](https://docs.python.org/3/builtins/stdtypes.html#str)* The consensus transcription (space-joined winning tokens). ### *class* ek.qe.RoverSlot(entries=, winner=None, score=0.0, vote_share=0.0) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One branch point of the transition network: each engine’s token (or NULL). ### *class* ek.qe.SignalFamilyCalibrator(calibrators=, combine=, factory=, fusion=None, kind='signal_family') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) One calibrator per signal family, fused by a documented combination rule. Averaging heterogeneous raw signals (an LLM-judge score, a token logprob, a ROVER agreement) and calibrating the mean discards exactly what a calibrator needs: each family has its own scale, reliability and direction of miscalibration. This calibrates every signal *by name* with its own calibrator, then fuses the calibrated probabilities with `combine`. Pass it (or simply a `Mapping[str, Calibrator]`) as `estimate_quality(..., calibrator=...)`. **The fused value must itself be calibrated before it is gated.** Fusing calibrated probabilities (by mean, min, …) does not yield a calibrated probability: agreeing families make the mean under-confident, a weak family drags it toward the base rate. So [`fit()`](_autosummary/ek.qe.html.md#ek.qe.SignalFamilyCalibrator.fit) also learns a second-stage `fusion` calibrator on `combine`’s output, and `__call__` applies it. Without a `fusion` calibrator (e.g. a hand-built mapping), [`ek.estimate_quality()`](_autosummary/ek.html.md#ek.estimate_quality) refuses to gate a value fused from two or more families unless told `assume_calibrated=True` (Hard Rule 1). A value from a single family is that family’s calibrated probability and gates as is. A field’s own upstream `confidence` is just one more family here, named `"confidence"`: include it in the training rows under that name. * **Parameters:** * **calibrators** ([`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)) – `signal name -> Calibrator`. A `"*"` entry is the fallback for any signal without its own entry. A signal with neither is refused (`ValueError`) rather than silently dropped or passed through raw (Hard Rule 1). * **combine** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`float`](https://docs.python.org/3/builtins/functions.html#float)]], [`float`](https://docs.python.org/3/builtins/functions.html#float)]) – `Mapping[str, float] -> float` fusing the calibrated values; [`mean_probability()`](_autosummary/ek.qe.html.md#ek.qe.mean_probability) (default) or [`min_probability()`](_autosummary/ek.qe.html.md#ek.qe.min_probability), or any callable of that shape (e.g. a learned stacker). * **factory** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – Zero-arg callable producing a fresh calibrator, used by [`fit()`](_autosummary/ek.qe.html.md#ek.qe.SignalFamilyCalibrator.fit). * **fusion** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Second-stage calibrator applied to `combine`’s output (learned by [`fit()`](_autosummary/ek.qe.html.md#ek.qe.SignalFamilyCalibrator.fit)); `None` leaves the fused value uncalibrated. ### Example ```pycon >>> rows = [{"judge": j, "logprob": l} for j, l in ... [(0.9, -0.1), (0.8, -0.2), (0.85, -2.0), (0.3, -0.1), (0.2, -2.5), (0.7, -1.8)]] >>> correct = [True, True, False, True, False, False] >>> cal = SignalFamilyCalibrator().fit(rows, correct) >>> sorted(cal.calibrators) ['judge', 'logprob'] >>> 0.0 <= cal({"judge": 0.9, "logprob": -0.1}) <= 1.0 True ``` #### calibrate_signals(raw_signals, , group=None) `{name: calibrated probability}` for each raw signal (before fusion). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) #### combine() Fuse calibrated per-family probabilities by their mean (the default combiner). Each family’s calibrated value estimates the *same* event (“this field is correct”), so the mean is a defensible, assumption-light pooling rule. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ```pycon >>> mean_probability({"judge": 0.9, "logprob": 0.7}) 0.8 ``` #### factory alias of [`PlattCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.PlattCalibrator) #### fit(signals, correct) Fit one calibrator per signal name, then the `fusion` calibrator. Each family is fit over the rows where it appears with a finite value. A family with fewer than two such rows, or only one label class, cannot be calibrated and is refused (`ValueError`) rather than turned into a constant that would then dilute the fusion. * **Return type:** [`SignalFamilyCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.SignalFamilyCalibrator) #### fuse(calibrated) Fuse per-family calibrated probabilities: `combine`, then `fusion`. A single family’s value is returned as is: it is already calibrated, and `fusion` was learned on fused values. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) #### to_dict() Serialize every family’s calibrator and the (registered) combiner’s name. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### *class* ek.qe.TemperatureCalibrator(T=1.0, kind='temperature', t_min=0.05, t_max=10.0, max_iter=60) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Temperature scaling: `sigmoid(logit / T)` with one `T` fit on a holdout. Use **only when you have logits**: `__call__` expects a *logit*, not a probability. `T > 1` softens overconfidence; the argmax is unchanged. #### fit(logits, correct) Fit `T` by minimising NLL with a bounded 1-D search. * **Return type:** [`TemperatureCalibrator`](_autosummary/ek.qe.calibrate.html.md#ek.qe.calibrate.TemperatureCalibrator) #### t_min *: [float](https://docs.python.org/3/builtins/functions.html#float)* *= 0.05* Golden-section search bounds and iteration count (config, not magic numbers). ### *class* ek.qe.VerifierSignal(validators=, cost_tier=1) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Deterministic verifier evidence as a reference-free [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal). Cost tier 1 – the free, first-to-run layer. Runs a list of [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) s on a value and returns the **fraction that passed** in `[0, 1]` as the raw signal (`1.0` = every check passed). It also exposes the [`Finding`](_autosummary/ek.base.html.md#ek.base.Finding) s it produced via [`findings()`](_autosummary/ek.qe.html.md#ek.qe.VerifierSignal.findings), so the same object feeds both the score path (-> calibrate) and the audit path (-> review). Like every signal it is uncalibrated: a verifier failing is strong evidence, not a probability, so a [`Calibrator`](_autosummary/ek.base.html.md#ek.base.Calibrator) still runs before any gate. ### Example ```pycon >>> v = VerifierSignal([checksum_validator("luhn")]) >>> v("79927398713") # valid -> all checks pass 1.0 >>> v("79927398710") # invalid -> the one check fails 0.0 ``` #### findings(value, , spec=None) All findings the validators produce for `value` (empty == all clear). * **Return type:** [`List`](https://docs.python.org/3/library/typing.html#typing.List)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)] ### ek.qe.checksum_validator(kind='luhn', , field_name='', layer='checksum') A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a value failing a named checksum. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)]] ### ek.qe.enum_validator(members, , field_name='', layer='enum') A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a value not in an allowed set. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)]] ### ek.qe.expected_calibration_error(probs, correct, , n_bins=10) Expected Calibration Error: weighted mean gap between confidence and accuracy. Bins predictions by confidence into `n_bins` equal-width bins and averages `|mean_confidence - accuracy|` weighted by bin population. `0` is perfect. Non-finite probs are skipped; out-of-range probs are clamped into `[0, 1]`. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.qe.geo_mean(logps) Geometric mean of token probabilities, `exp(mean log p)` (perplexity-inverse). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.qe.iban_check(iban) IBAN validity via the ISO 13616 / ISO 7064 mod-97 rule (must equal 1). * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.qe.isbn_check(value) Validate an ISBN-10 or ISBN-13 by length (digits/hyphens/spaces ignored). * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.qe.length_normalized(logps, , alpha=0.6) Length-normalised score `exp(Σ log p / T**alpha)` (tunable length penalty). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.qe.load_calibrator(name, , rootdir=None) Reconstruct a persisted calibrator by name (dispatched on its `kind`). Validates the stored record so a malformed or unknown `kind` fails with an actionable error rather than a raw `KeyError` or a load-then-crash-later. * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) ### ek.qe.luhn_check(number) Luhn (mod-10) checksum: credit cards, IMEIs, some national IDs. * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.qe.mean_prob(logps) Arithmetic mean of token probabilities `mean(exp(log p))`. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.qe.mean_probability(probs) Fuse calibrated per-family probabilities by their mean (the default combiner). Each family’s calibrated value estimates the *same* event (“this field is correct”), so the mean is a defensible, assumption-light pooling rule. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ```pycon >>> mean_probability({"judge": 0.9, "logprob": 0.7}) 0.8 ``` ### ek.qe.min_prob(logps) Weakest-token probability `exp(min log p)` – catches one bad character. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.qe.min_probability(probs) Fuse calibrated per-family probabilities by their minimum (conservative). Use when any one family flagging a problem should be enough to lower the gate. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ```pycon >>> min_probability({"judge": 0.9, "logprob": 0.7}) 0.7 ``` ### ek.qe.range_validator(lo, hi, , field_name='', layer='range') A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a number outside `[lo, hi]`. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)]] ### ek.qe.regex_validator(pattern, , field_name='', layer='format') A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a value not fully matching `pattern`. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)]] ### ek.qe.reliability_curve(probs, correct, , n_bins=10) Per-bin `{confidence, accuracy, count}` for a reliability diagram. * **Return type:** [`List`](https://docs.python.org/3/library/typing.html#typing.List)[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)] ### ek.qe.risk_coverage_curve(probs, correct, , thresholds=None) The risk-coverage trade-off: `{threshold, coverage, selective_risk}` per point. Coverage is the fraction auto-accepted at each threshold; selective risk is the error rate *among the accepted*. Lower-left is better; pick the operating point against a target risk or coverage. * **Return type:** [`List`](https://docs.python.org/3/library/typing.html#typing.List)[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)] ### ek.qe.rover(hypotheses, , use_confidence=True, conf_weight=0.5, null_conf=0.7, tokenize=None, max_tokens=5000) Align N hypotheses, vote per slot, and emit consensus + per-position agreement. * **Parameters:** * **hypotheses** ([`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – The recognizer/extractor outputs to fuse. Each may be a string, an `OcrResult`-shaped object (`.text`/`.mean_confidence`), a list of token strings, or a list of `(token, confidence)` pairs. * **use_confidence** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Blend confidence into the vote (`True`) or vote purely by frequency (`False`, which forces `conf_weight` to `0`). * **conf_weight** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Weight on average confidence vs vote frequency in the slot score (Fiscus’s `1 - alpha`); `0` is frequency-only, `1` is confidence-only. Ignored when `use_confidence` is `False`. * **null_conf** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Confidence credited to a NULL vote (an engine that produced no token at a slot) – the lever for how readily a deletion wins. * **tokenize** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`List`](https://docs.python.org/3/library/typing.html#typing.List)[[`Tuple`](https://docs.python.org/3/library/typing.html#typing.Tuple)[[`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`float`](https://docs.python.org/3/builtins/functions.html#float)]]]]) – Optional `hypothesis -> [(token, confidence), ...]` override; by default `_as_units()` handles strings/OcrResults/token lists. * **max_tokens** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`int`](https://docs.python.org/3/builtins/functions.html#int)]) – Reject any hypothesis longer than this (the aligner is `O(N*l*L*L')`; an unbounded input is a quadratic time/memory DoS). `None` disables the guard. * **Return type:** [`RoverConsensus`](_autosummary/ek.qe.html.md#ek.qe.RoverConsensus) * **Returns:** A [`RoverConsensus`](_autosummary/ek.qe.html.md#ek.qe.RoverConsensus) with the consensus `tokens`/`text`, the per-slot breakdown, and the per-consensus-token `agreement` usable as a raw signal. ### ek.qe.save_calibrator(calibrator, name, , rootdir=None) Persist a fitted calibrator’s parameters to the `calibrators` store. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### ek.qe.schema_validator(value, , spec=None) A spec-driven [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator): type and `domain` checks from a FieldSpec. Reads the [`FieldSpec`](_autosummary/ek.base.html.md#ek.base.FieldSpec) `type` and `domain` – a `(lo, hi)` numeric range, an enum tuple, or a regex string – and applies the matching check. With no spec it is a no-op, so it composes safely in any validator list. * **Return type:** [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)] ### ek.qe.sklearn_calibrator(method='sigmoid') A calibrator backed by scikit-learn (`method='sigmoid'` Platt or `'isotonic'`). Behind `ek[calibration]`. Returns an object satisfying the [`Calibrator`](_autosummary/ek.base.html.md#ek.base.Calibrator) protocol that wraps sklearn’s calibration. The pure-Python [`PlattCalibrator`](_autosummary/ek.qe.html.md#ek.qe.PlattCalibrator)/[`IsotonicCalibrator`](_autosummary/ek.qe.html.md#ek.qe.IsotonicCalibrator) are the dependency-free defaults; use this for parity with an sklearn-centric stack. ### ek.qe.split_conformal_quantile(scores, alpha) The split-conformal threshold: the `ceil((n+1)(1-alpha)) / n` empirical quantile. `scores` are nonconformity scores on an exchangeable calibration set. Returns `+inf` when `n` is too small to guarantee `1 - alpha` (so nothing is flagged – the honest behaviour at that sample size), and `-inf` at the `alpha -> 1` limit (flag everything). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.qe.totals_consistent(record, , total_key, item_keys, tol=0.01, layer='cross_field') Flag when `record[total_key]` does not equal the sum of `item_keys` (±tol). The canonical cross-field check (invoice total = Σ line items). Skipped (rather than raising or flagging) when the total or *all* item keys are absent or non-numeric, so it is safe on partial records. * **Return type:** [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)] ### Modules | [`calibrate`](_autosummary/ek.qe.calibrate.html.md#module-ek.qe.calibrate) | Calibration: make a raw score *mean* a probability (the non-optional stage). | |-------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------| | [`decide`](_autosummary/ek.qe.decide.html.md#module-ek.qe.decide) | Selective prediction: turn a calibrated probability into accept / flag / block. | | [`signals`](_autosummary/ek.qe.signals.html.md#module-ek.qe.signals) | Intrinsic confidence signals: token logprobs and per-unit posteriors (cost tier 2). | | [`verifiers`](_autosummary/ek.qe.verifiers.html.md#module-ek.qe.verifiers) | Deterministic verifier signals -- the cheapest, first-to-run QE layer. | # _autosummary/ek.qe.signals.html.md # ek.qe.signals Intrinsic confidence signals: token logprobs and per-unit posteriors (cost tier 2). When the extractor already emits something usable – token log-probabilities (generative locals like TrOCR/pix2tex, GPT-4o, Mistral OCR) or per-unit posteriors (Google Cloud Vision, Azure, Tesseract) – the cheapest reliable signal is *free*: it was computed during inference. This module turns those raw numbers into a single field-level score (`misc/docs/ek_03` §1a). Two modelling facts drive the design: 1. **Raw summed logprob is length-biased** – longer fields accumulate more negative terms and look worse. The fix is length normalisation; the aggregation is a *modelling choice to validate on held-out data*, so it stays **pluggable** (no hardcoded pooling). Use `min` to catch the single weakest token (a transposed digit in an amount); use the geometric mean for overall field plausibility (the family Mistral OCR reports as `average_page_confidence_score`). 2. **It is uncalibrated** – and for RLHF’d LLMs *systematically overconfident*. Every signal here is a raw score; a [`Calibrator`](_autosummary/ek.base.html.md#ek.base.Calibrator) must run before any gate reads it. Aggregators are registered under the `aggregators` namespace so a caller (or a third party) can swap pooling by name. ### Example ```pycon >>> import math >>> logps = [math.log(0.9), math.log(0.8), math.log(0.95)] >>> round(geo_mean(logps), 4) # exp(mean log p) 0.8811 >>> round(min_prob(logps), 4) # weakest token 0.8 ``` ### Module Attributes | [`DEFAULT_LENGTH_ALPHA`](_autosummary/ek.qe.signals.html.md#ek.qe.signals.DEFAULT_LENGTH_ALPHA) | Length-penalty exponent for [`length_normalized()`](_autosummary/ek.qe.signals.html.md#ek.qe.signals.length_normalized) (Wu et al. 2016 found `alpha` in `[0.6, 0.7]` best); a keyword default, never a magic constant. | |-------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`DEFAULT_PROB_FLOOR`](_autosummary/ek.qe.signals.html.md#ek.qe.signals.DEFAULT_PROB_FLOOR) | Smallest probability treated as non-zero. | ### Functions | [`geo_mean`](_autosummary/ek.qe.signals.html.md#ek.qe.signals.geo_mean)(logps) | Geometric mean of token probabilities, `exp(mean log p)` (perplexity-inverse). | |----------------------------------------------------------------------------------------|----------------------------------------------------------------------------------| | [`length_normalized`](_autosummary/ek.qe.signals.html.md#ek.qe.signals.length_normalized)(logps, \*[, alpha]) | Length-normalised score `exp(Σ log p / T**alpha)` (tunable length penalty). | | [`mean_prob`](_autosummary/ek.qe.signals.html.md#ek.qe.signals.mean_prob)(logps) | Arithmetic mean of token probabilities `mean(exp(log p))`. | | [`min_prob`](_autosummary/ek.qe.signals.html.md#ek.qe.signals.min_prob)(logps) | Weakest-token probability `exp(min log p)` -- catches one bad character. | ### Classes | [`IntrinsicConfidenceSignal`](_autosummary/ek.qe.signals.html.md#ek.qe.signals.IntrinsicConfidenceSignal)([pool, cost_tier]) | Aggregate an extractor's per-unit confidences into a field score. | |-------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------| | [`LogprobSignal`](_autosummary/ek.qe.signals.html.md#ek.qe.signals.LogprobSignal)([aggregator, alpha, cost_tier]) | Aggregate token log-probabilities into one field score (a [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal)). | ### ek.qe.signals.DEFAULT_LENGTH_ALPHA *= 0.6* Length-penalty exponent for [`length_normalized()`](_autosummary/ek.qe.signals.html.md#ek.qe.signals.length_normalized) (Wu et al. 2016 found `alpha` in `[0.6, 0.7]` best); a keyword default, never a magic constant. ### ek.qe.signals.DEFAULT_PROB_FLOOR *= 1e-12* Smallest probability treated as non-zero. `log(p=0) = -inf` clamps to `log(DEFAULT_PROB_FLOOR)` (mirrors the confidence floor) so a p=0 token still drives min_prob/geo_mean toward 0 instead of being silently discarded. ### *class* ek.qe.signals.IntrinsicConfidenceSignal(pool='min', cost_tier=2) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Aggregate an extractor’s per-unit confidences into a field score. * **Parameters:** * **pool** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – `"min"` (weakest unit), `"mean"`, or `"geo_mean"` over the per-unit confidences (already probabilities in `[0, 1]`). * **cost_tier** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – `2` – free (the posteriors are emitted at inference). ### Example ```pycon >>> IntrinsicConfidenceSignal(pool="min")([0.99, 0.4, 0.95]) 0.4 ``` ### *class* ek.qe.signals.LogprobSignal(aggregator=, alpha=0.6, cost_tier=2) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Aggregate token log-probabilities into one field score (a [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal)). * **Parameters:** * **aggregator** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – A logprob aggregator (name registered under `aggregators` or a callable). Defaults to [`geo_mean()`](_autosummary/ek.qe.signals.html.md#ek.qe.signals.geo_mean). * **alpha** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Length penalty passed to [`length_normalized()`](_autosummary/ek.qe.signals.html.md#ek.qe.signals.length_normalized) only. * **cost_tier** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – `2` – free intrinsic signal (already computed at inference). #### aggregator() Geometric mean of token probabilities, `exp(mean log p)` (perplexity-inverse). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.qe.signals.geo_mean(logps) Geometric mean of token probabilities, `exp(mean log p)` (perplexity-inverse). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.qe.signals.length_normalized(logps, , alpha=0.6) Length-normalised score `exp(Σ log p / T**alpha)` (tunable length penalty). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.qe.signals.mean_prob(logps) Arithmetic mean of token probabilities `mean(exp(log p))`. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.qe.signals.min_prob(logps) Weakest-token probability `exp(min log p)` – catches one bad character. * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) # _autosummary/ek.qe.verifiers.html.md # ek.qe.verifiers Deterministic verifier signals – the cheapest, first-to-run QE layer. Reference-free quality estimation should try the \*\*free, label-free, deterministic checks first\*\* (`misc/docs/ek_03` §1d, §5; `ek_04` Layers 0-3): they cost nothing, need no training data, and catch exactly the errors a confidence signal misses – a value the extractor was *confident* about but that is structurally impossible (a checksum that does not validate, a total that does not reconcile, an enum value out of range). Only escalate to intrinsic confidence, agreement (ROVER), or LLM signals on the residual that verifiers cannot settle. The check primitives here (Luhn / IBAN / ISBN checksums, type/range/enum/regex, cross-field reconciliation) are pure-Python and dependency-free *by design*: they are reused both as reference-free [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal) evidence (this module, the [`ek.estimate_quality()`](_autosummary/ek.html.md#ek.estimate_quality) side) and – later – as the L1-L2 layers of the flag-vs-correct validation pipeline (`ek/validate.py`, issue #7). Implementing the checksums directly (rather than depending on the LGPL `python-stdnum`) keeps the permissive-core promise intact. A verifier emits raw evidence – a pass/fail [`Finding`](_autosummary/ek.base.html.md#ek.base.Finding) and a fraction-passed score – never a calibrated probability or a decision. ### Example ```pycon >>> luhn_check("79927398713") # a classic Luhn-valid number True >>> luhn_check("79927398710") False >>> isbn_check("0-306-40615-2") # ISBN-10 True ``` ### Module Attributes | [`DEFAULT_TOTALS_TOL`](_autosummary/ek.qe.verifiers.html.md#ek.qe.verifiers.DEFAULT_TOTALS_TOL) | Default absolute tolerance for the cross-field totals reconciliation check. | |-----------------------------------------------------------------------|---------------------------------------------------------------------------------| | [`CHECKSUMS`](_autosummary/ek.qe.verifiers.html.md#ek.qe.verifiers.CHECKSUMS) | Built-in checksum checks, by name (register more under the `checks` namespace). | ### Functions | [`checksum_validator`](_autosummary/ek.qe.verifiers.html.md#ek.qe.verifiers.checksum_validator)([kind, field_name, layer]) | A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a value failing a named checksum. | |----------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------| | [`enum_validator`](_autosummary/ek.qe.verifiers.html.md#ek.qe.verifiers.enum_validator)(members, \*[, field_name, layer]) | A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a value not in an allowed set. | | [`iban_check`](_autosummary/ek.qe.verifiers.html.md#ek.qe.verifiers.iban_check)(iban) | IBAN validity via the ISO 13616 / ISO 7064 mod-97 rule (must equal 1). | | [`isbn10_check`](_autosummary/ek.qe.verifiers.html.md#ek.qe.verifiers.isbn10_check)(value) | ISBN-10 checksum (weighted mod-11; a trailing `X` counts as 10). | | [`isbn13_check`](_autosummary/ek.qe.verifiers.html.md#ek.qe.verifiers.isbn13_check)(value) | ISBN-13 / EAN-13 checksum (alternating 1/3 weights, mod-10). | | [`isbn_check`](_autosummary/ek.qe.verifiers.html.md#ek.qe.verifiers.isbn_check)(value) | Validate an ISBN-10 or ISBN-13 by length (digits/hyphens/spaces ignored). | | [`luhn_check`](_autosummary/ek.qe.verifiers.html.md#ek.qe.verifiers.luhn_check)(number) | Luhn (mod-10) checksum: credit cards, IMEIs, some national IDs. | | [`range_validator`](_autosummary/ek.qe.verifiers.html.md#ek.qe.verifiers.range_validator)(lo, hi, \*[, field_name, layer]) | A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a number outside `[lo, hi]`. | | [`regex_validator`](_autosummary/ek.qe.verifiers.html.md#ek.qe.verifiers.regex_validator)(pattern, \*[, field_name, layer]) | A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a value not fully matching `pattern`. | | [`schema_validator`](_autosummary/ek.qe.verifiers.html.md#ek.qe.verifiers.schema_validator)(value, \*[, spec]) | A spec-driven [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator): type and `domain` checks from a FieldSpec. | | [`totals_consistent`](_autosummary/ek.qe.verifiers.html.md#ek.qe.verifiers.totals_consistent)(record, \*, total_key, ...) | Flag when `record[total_key]` does not equal the sum of `item_keys` (±tol). | ### Classes | [`VerifierSignal`](_autosummary/ek.qe.verifiers.html.md#ek.qe.verifiers.VerifierSignal)([validators, cost_tier]) | Deterministic verifier evidence as a reference-free [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal). | |--------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------| ### ek.qe.verifiers.CHECKSUMS *: [dict](https://docs.python.org/3/builtins/stdtypes.html#dict)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), [Callable](https://docs.python.org/3/library/typing.html#typing.Callable)[[[Any](https://docs.python.org/3/library/typing.html#typing.Any)], [bool](https://docs.python.org/3/builtins/functions.html#bool)]]* *= {'iban': , 'isbn': , 'isbn10': , 'isbn13': , 'luhn': }* Built-in checksum checks, by name (register more under the `checks` namespace). ### ek.qe.verifiers.DEFAULT_TOTALS_TOL *= 0.01* Default absolute tolerance for the cross-field totals reconciliation check. ### *class* ek.qe.verifiers.VerifierSignal(validators=, cost_tier=1) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Deterministic verifier evidence as a reference-free [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal). Cost tier 1 – the free, first-to-run layer. Runs a list of [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) s on a value and returns the **fraction that passed** in `[0, 1]` as the raw signal (`1.0` = every check passed). It also exposes the [`Finding`](_autosummary/ek.base.html.md#ek.base.Finding) s it produced via [`findings()`](_autosummary/ek.qe.verifiers.html.md#ek.qe.verifiers.VerifierSignal.findings), so the same object feeds both the score path (-> calibrate) and the audit path (-> review). Like every signal it is uncalibrated: a verifier failing is strong evidence, not a probability, so a [`Calibrator`](_autosummary/ek.base.html.md#ek.base.Calibrator) still runs before any gate. ### Example ```pycon >>> v = VerifierSignal([checksum_validator("luhn")]) >>> v("79927398713") # valid -> all checks pass 1.0 >>> v("79927398710") # invalid -> the one check fails 0.0 ``` #### findings(value, , spec=None) All findings the validators produce for `value` (empty == all clear). * **Return type:** [`List`](https://docs.python.org/3/library/typing.html#typing.List)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)] ### ek.qe.verifiers.checksum_validator(kind='luhn', , field_name='', layer='checksum') A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a value failing a named checksum. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)]] ### ek.qe.verifiers.enum_validator(members, , field_name='', layer='enum') A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a value not in an allowed set. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)]] ### ek.qe.verifiers.iban_check(iban) IBAN validity via the ISO 13616 / ISO 7064 mod-97 rule (must equal 1). * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.qe.verifiers.isbn10_check(value) ISBN-10 checksum (weighted mod-11; a trailing `X` counts as 10). * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.qe.verifiers.isbn13_check(value) ISBN-13 / EAN-13 checksum (alternating 1/3 weights, mod-10). * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.qe.verifiers.isbn_check(value) Validate an ISBN-10 or ISBN-13 by length (digits/hyphens/spaces ignored). * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.qe.verifiers.luhn_check(number) Luhn (mod-10) checksum: credit cards, IMEIs, some national IDs. * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.qe.verifiers.range_validator(lo, hi, , field_name='', layer='range') A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a number outside `[lo, hi]`. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)]] ### ek.qe.verifiers.regex_validator(pattern, , field_name='', layer='format') A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that flags a value not fully matching `pattern`. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)]] ### ek.qe.verifiers.schema_validator(value, , spec=None) A spec-driven [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator): type and `domain` checks from a FieldSpec. Reads the [`FieldSpec`](_autosummary/ek.base.html.md#ek.base.FieldSpec) `type` and `domain` – a `(lo, hi)` numeric range, an enum tuple, or a regex string – and applies the matching check. With no spec it is a no-op, so it composes safely in any validator list. * **Return type:** [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)] ### ek.qe.verifiers.totals_consistent(record, , total_key, item_keys, tol=0.01, layer='cross_field') Flag when `record[total_key]` does not equal the sum of `item_keys` (±tol). The canonical cross-field check (invoice total = Σ line items). Skipped (rather than raising or flagging) when the total or *all* item keys are absent or non-numeric, so it is safe on partial records. * **Return type:** [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)] # _autosummary/ek.registry.html.md # ek.registry Strategy registry, optional-dependency guarding, and requirement checks. `ek` is open-closed: every swappable behaviour (a [`Metric`](_autosummary/ek.base.html.md#ek.base.Metric), a [`Signal`](_autosummary/ek.base.html.md#ek.base.Signal), a calibrator, an OCR backend, …) is registered under a *namespace* and resolved by name. Built-in strategies register themselves at import time; third parties add their own either by calling [`register()`](_autosummary/ek.registry.html.md#ek.registry.register) or by declaring an entry point in the `ek.` group – no edit to `ek` needed. Two more concerns live here so they stay out of the core logic: - [`requires_extra()`](_autosummary/ek.registry.html.md#ek.registry.requires_extra) – a decorator that turns a missing optional dependency into an *actionable* error (`pip install ek[ocr]`) instead of an opaque `ImportError` raised deep inside a call. - [`check_requirements()`](_autosummary/ek.registry.html.md#ek.registry.check_requirements) – a helper that reports, without raising, what an engine/feature needs (Python packages, system binaries, credentials), and defers to `ocracy`’s own doctor for OCR engines when it is installed. ### Example ```pycon >>> @register("metrics", "shout") ... def shout(pred, gold, *, grammar=None): ... return pred == gold >>> get("metrics", "shout")("a", "a") True >>> "shout" in names("metrics") True ``` ### Functions | [`check_requirements`](_autosummary/ek.registry.html.md#ek.registry.check_requirements)(\*[, engine, extra]) | Report what an OCR engine or feature extra needs, without raising. | |--------------------------------------------------------------------------------------------|-------------------------------------------------------------------------| | [`get`](_autosummary/ek.registry.html.md#ek.registry.get)(namespace, name) | Resolve a registered strategy by name (discovering entry points first). | | [`names`](_autosummary/ek.registry.html.md#ek.registry.names)(namespace) | All registered names in a namespace (after entry-point discovery). | | [`register`](_autosummary/ek.registry.html.md#ek.registry.register)(namespace, name[, obj]) | Register `obj` under `namespace`/`name` (usable as a decorator). | | [`requires_extra`](_autosummary/ek.registry.html.md#ek.registry.requires_extra)(extra, \*[, packages]) | Decorator: fail with an actionable install hint if an extra is missing. | | [`resolve`](_autosummary/ek.registry.html.md#ek.registry.resolve)(namespace, ref, \*[, default]) | Coerce a strategy reference to a callable. | ### Exceptions | [`MissingExtraError`](_autosummary/ek.registry.html.md#ek.registry.MissingExtraError) | Raised when an optional feature is used without its extra installed. | |----------------------------------------------------------------------|------------------------------------------------------------------------| ### *exception* ek.registry.MissingExtraError Bases: [`ImportError`](https://docs.python.org/3/builtins/exceptions.html#ImportError) Raised when an optional feature is used without its extra installed. ### ek.registry.check_requirements(, engine=None, extra=None) Report what an OCR engine or feature extra needs, without raising. For an OCR `engine` and an installed `ocracy`, defers to `ocracy`’s own doctor (which knows each backend’s pip package, system binaries and credentials). Otherwise returns a generic hint. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) * **Returns:** A dict with at least `{"ok": bool, "hint": str}`. ### ek.registry.get(namespace, name) Resolve a registered strategy by name (discovering entry points first). * **Raises:** [**KeyError**](https://docs.python.org/3/builtins/exceptions.html#KeyError) – with the list of available names, if `name` is unknown. * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) ### ek.registry.names(namespace) All registered names in a namespace (after entry-point discovery). * **Return type:** [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] ### ek.registry.register(namespace, name, obj=None) Register `obj` under `namespace`/`name` (usable as a decorator). * **Parameters:** * **namespace** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – A strategy family, e.g. `"metrics"`, `"signals"`, `"calibrators"`, `"policies"`, `"normalizers"`, `"ocr"`. * **name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The lookup key within the namespace. * **obj** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – The object to register; omit to use as a decorator. * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) * **Returns:** `obj` (so it can be used as a decorator). ### ek.registry.requires_extra(extra, , packages=None) Decorator: fail with an actionable install hint if an extra is missing. The wrapped callable runs only once every package in `packages` imports; the common case (everything installed) adds a single cheap import check. * **Parameters:** * **extra** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The extra name, used in the hint `pip install ek[]`. * **packages** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]]) – Import names to probe (defaults to `[extra]`). * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable) ### Example ```pycon >>> @requires_extra("ocr", packages=["definitely_not_installed_pkg"]) ... def run(): ... return "ran" >>> try: ... run() ... except MissingExtraError as e: ... print("ek[ocr]" in str(e)) True ``` ### ek.registry.resolve(namespace, ref, , default=None) Coerce a strategy reference to a callable. `ref` may be a registered name (`str`), an already-resolved object, or `None` (in which case `default`, itself a name or object, is used). * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) # _autosummary/ek.stores.html.md # ek.stores Local-file persistence for `ek`, as plain `MutableMapping` stores. Everything `ek` persists – gold corpora, evaluation results, fitted calibrators, human corrections, run logs – lives under the per-user data directory `~/.local/share/ek/` (resolved cross-platform by `config2py.AppData`), one subfolder per *kind*. Each store is a JSON key-value store from `dol`: `store[key] = value` writes `/.json`, `store[key]` reads it back, iteration lists keys – so callers get a dict-like facade rather than a bespoke god-class. Progressive disclosure: ```pycon >>> import tempfile; root = tempfile.mkdtemp() >>> s = json_store("gold", rootdir=root) >>> s["doc-001"] = {"reference_text": "hello world"} >>> s["doc-001"]["reference_text"] 'hello world' >>> list(s) ['doc-001'] ``` A [`mall()`](_autosummary/ek.stores.html.md#ek.stores.mall) groups every kind into one stores-of-stores mapping, with optional `mall["gold", "doc-001"]` tuple-key access. Tests and ephemeral use can redirect the root via the `rootdir` argument or the `EK_DATA_HOME` environment variable, so nothing touches the real user folder. ### Module Attributes | [`cache_this`](_autosummary/ek.stores.html.md#ek.stores.cache_this)([func, cache, key, pre_cache, ...]) | Unified caching decorator for properties and methods with persistent storage support. | |---------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------| | [`KINDS`](_autosummary/ek.stores.html.md#ek.stores.KINDS) | The kinds of artifact `ek` persists; each becomes a subfolder + a store. | ### Functions | [`app_folder`](_autosummary/ek.stores.html.md#ek.stores.app_folder)() | The root `ek` data folder (e.g. `~/.local/share/ek`). | |-----------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------| | [`data_dir`](_autosummary/ek.stores.html.md#ek.stores.data_dir)(kind, \*[, rootdir]) | Filesystem directory backing `kind` (under the data root or `rootdir`). | | [`json_store`](_autosummary/ek.stores.html.md#ek.stores.json_store)(kind, \*[, rootdir]) | A JSON `MutableMapping` for one `kind` of artifact. | | [`mall`](_autosummary/ek.stores.html.md#ek.stores.mall)(\*[, rootdir]) | The full stores-of-stores mall (one JSON store per [`KINDS`](_autosummary/ek.stores.html.md#ek.stores.KINDS)). | | [`persistent_cache`](_autosummary/ek.stores.html.md#ek.stores.persistent_cache)([func, kind, key, rootdir]) | Memoize a plain function's result in a `kind` JSON store, across sessions. | ### Classes | [`Mall`](_autosummary/ek.stores.html.md#ek.stores.Mall)(\*[, kinds, rootdir]) | A read-only mapping of `kind -> json_store(kind)` (a dol "mall"). | |-------------------------------------------------------------------------------|---------------------------------------------------------------------| ### ek.stores.KINDS *= ('gold', 'results', 'calibrators', 'corrections', 'runs', 'baselines')* The kinds of artifact `ek` persists; each becomes a subfolder + a store. ### *class* ek.stores.Mall(, kinds=('gold', 'results', 'calibrators', 'corrections', 'runs', 'baselines'), rootdir=None) Bases: [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping) A read-only mapping of `kind -> json_store(kind)` (a dol “mall”). Supports both `mall["gold"]` (the store) and `mall["gold", key]` (the value), the latter being sugar for `mall["gold"][key]`. ### ek.stores.app_folder() The root `ek` data folder (e.g. `~/.local/share/ek`). * **Return type:** [`Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path) ### ek.stores.cache_this(func=None, , cache=None, key=None, pre_cache=False, as_property=None, ignore=None, serialize=None, deserialize=None) Re-export for the idiomatic class-property persisted cache, e.g. `@cache_this(cache=lambda self: json_store("runs"), key=...)` on a method. ### ek.stores.data_dir(kind, , rootdir=None) Filesystem directory backing `kind` (under the data root or `rootdir`). * **Return type:** [`Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path) ### ek.stores.json_store(kind, , rootdir=None) A JSON `MutableMapping` for one `kind` of artifact. Keys are plain names (no extension); values are JSON-serializable objects. * **Return type:** [`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping) ### ek.stores.mall(, rootdir=None) The full stores-of-stores mall (one JSON store per [`KINDS`](_autosummary/ek.stores.html.md#ek.stores.KINDS)). * **Return type:** [`Mall`](_autosummary/ek.stores.html.md#ek.stores.Mall) ### ek.stores.persistent_cache(func=None, , kind='runs', key=None, rootdir=None) Memoize a plain function’s result in a `kind` JSON store, across sessions. Use as `@persistent_cache(kind="runs", key=...)`. Returned values must be JSON-serializable. `key` may be a constant or a callable receiving the call arguments; it defaults to the function’s qualified name (so provide a `key` for functions that take arguments). For caching a *class property/method*, prefer [`cache_this`](_autosummary/ek.stores.html.md#ek.stores.cache_this) (dol’s descriptor) instead. ### Example ```pycon >>> import tempfile; root = tempfile.mkdtemp() >>> calls = [] >>> @persistent_cache(kind="runs", key="answer", rootdir=root) ... def compute(): ... calls.append(1) ... return {"answer": 42} >>> compute(), compute(), len(calls) ({'answer': 42}, {'answer': 42}, 1) ``` # _autosummary/ek.tools.html.md # ek.tools Small, CLI-friendly functions over the `ek` core (the dispatch surface). These are thin, string-in/value-out wrappers designed for triple dispatch (CLI today, HTTP/UI later). The single `_dispatch_funcs` list is the SSOT of what `ek.__main__` exposes, so there is no duplicated command registration, and each function’s *signature* is its command-line interface. Run them from the shell: ```default python -m ek cer "hello wrld" "hello world" python -m ek where python -m ek check tesseract ``` ### Functions | [`cer`](_autosummary/ek.tools.html.md#ek.tools.cer)(prediction, gold, \*[, normalize]) | Character Error Rate between a prediction and a gold string (lower is better). | |-------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------| | [`check`](_autosummary/ek.tools.html.md#ek.tools.check)(engine) | Report what an OCR engine needs to run (delegates to ocracy when installed). | | [`cost_per_success`](_autosummary/ek.tools.html.md#ek.tools.cost_per_success)(total_dollars, successes, \*) | Cost-of-Pass: dollars per *successfully completed* task (`inf` if none succeeded). | | [`engines`](_autosummary/ek.tools.html.md#ek.tools.engines)() | List installed OCR backends (requires the `ek[ocr]` extra). | | [`pass_k`](_autosummary/ek.tools.html.md#ek.tools.pass_k)(n, c[, k]) | pass@k (capability) and pass^k (reliability) from `c` successes in `n` trials. | | [`rover`](_autosummary/ek.tools.html.md#ek.tools.rover)(\*hypotheses[, confidence]) | ROVER-fuse 2+ transcriptions into a consensus + mean agreement (reference-free). | | [`version`](_autosummary/ek.tools.html.md#ek.tools.version)() | Print the installed `ek` version. | | [`wer`](_autosummary/ek.tools.html.md#ek.tools.wer)(prediction, gold, \*[, normalize]) | Word Error Rate between a prediction and a gold string (lower is better). | | [`where`](_autosummary/ek.tools.html.md#ek.tools.where)() | Print the local data folder where `ek` persists gold/results/runs. | ### ek.tools.cer(prediction, gold, , normalize=None) Character Error Rate between a prediction and a gold string (lower is better). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.tools.check(engine) Report what an OCR engine needs to run (delegates to ocracy when installed). * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### ek.tools.cost_per_success(total_dollars, successes, , attempts=None) Cost-of-Pass: dollars per *successfully completed* task (`inf` if none succeeded). The unit that matters for an agent – tokens spent on a failed episode are pure waste. Example: ```default python -m ek cost-per-success 12.50 5 --attempts 20 ``` * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### ek.tools.engines() List installed OCR backends (requires the `ek[ocr]` extra). * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list) ### ek.tools.pass_k(n, c, k=1) pass@k (capability) and pass^k (reliability) from `c` successes in `n` trials. `pass@k` asks “can it *ever* do this?”; `pass^k` asks “does it do this *every* time?” – and only the second is a production number. Example: ```default python -m ek pass-k 10 9 --k 8 # a 90%-reliable agent, judged over 8 trials ``` * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### ek.tools.rover(\*hypotheses, confidence=True) ROVER-fuse 2+ transcriptions into a consensus + mean agreement (reference-free). Example: ```default python -m ek rover "the cat sat" "the cat sit" "the bat sat" ``` * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### ek.tools.version() Print the installed `ek` version. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### ek.tools.wer(prediction, gold, , normalize=None) Word Error Rate between a prediction and a gold string (lower is better). * **Return type:** [`float`](https://docs.python.org/3/builtins/functions.html#float) ### ek.tools.where() Print the local data folder where `ek` persists gold/results/runs. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) # _autosummary/ek.validate.html.md # ek.validate Reference-free post-extraction validation & correction (issue #7). A layered, **cheapest -> most-expensive** pipeline with an explicit **FLAG-vs-CORRECT** distinction (`misc/docs/ek_04`). Every method estimates `argmax_v P(v|o)` without a gold reference: the prior `P(v)` strengthens in layers – schema/type/range (cheapest, strongest), then lexicon/gazetteer, then language-model surprisal (softest) – with cross-field consistency and statistical anomaly checks on top. The single most important architectural line is **FLAG vs CORRECT**: - A **Validator** only *flags*: it yields [`Finding`](_autosummary/ek.base.html.md#ek.base.Finding) s with `severity=FLAG` and never rewrites the value. The deterministic verifier layer (`ek.qe.verifiers`: checksums, regex, range, enum, schema, totals) is re-exported here so a caller composes the whole stack from one place. - A **Corrector** can also *correct*: it yields a `severity=CORRECT` Finding whose `suggestion` is the proposed replacement. Only a narrow, safe, deterministic class corrects automatically – canonicalization (L0) and closed-set lexicon resolution (L2); free invention is reserved for the gated L5 neural corrector. Both share one callable shape – `(value, *, spec) -> Iterable[Finding]` – so [`validation_pipeline()`](_autosummary/ek.validate.html.md#ek.validate.validation_pipeline) composes them uniformly. A CORRECT finding’s suggestion is applied to the value *before the next layer runs* (the noisy-channel chain: a cheap deterministic fix short-circuits the expensive layers), and an injectable `stop_when` policy ends the pass early (e.g. once a value is corrected or clean). The deterministic layers here need no extra dependencies (`rapidfuzz` is core). The softer/heavier layers ship as **dependency-injected** layers: the layer is dep-free and you inject the model/LLM call (your in-domain LM scorer, your LLM client), so the expensive backend is your deployment choice, not a default dependency. - **L3 LM surprisal** (FLAG): [`lm_surprisal_validator()`](_autosummary/ek.validate.html.md#ek.validate.lm_surprisal_validator) takes an injected `(str) -> float` surprisal scorer (n-gram / masked-LM PLL, e.g. `minicons`) and flags low-probability spans. Statistical anomaly without an LM: [`benford_findings()`](_autosummary/ek.validate.html.md#ek.validate.benford_findings) (leading-digit) and [`zscore_anomaly_findings()`](_autosummary/ek.validate.html.md#ek.validate.zscore_anomaly_findings) > (robust median/MAD magnitude outliers), both dep-free. - **L5 neural/LLM correction** (CORRECT, gated): [`llm_corrector()`](_autosummary/ek.validate.html.md#ek.validate.llm_corrector) takes an injected `(str) -> Optional[str]` correction call; it is the only layer that invents content, so it fires only on already-flagged values (`only_flagged`), reads the pipeline’s findings, and keeps the original for audit. - **L4 constrained generation** is a *generation*-time concern (it constrains a generative extractor, not post-hoc output – ek_04 separates “constrain-on-generate” from “post-hoc validation”), so it lives on the extraction side (`ek[constrained]`: `outlines`/XGrammar/`instructor`), not in this pipeline. ### Example ```pycon >>> from ek.validate import validation_pipeline, lexicon_corrector, stop_on_correction >>> from ek.base import Severity >>> fix = validation_pipeline( ... lexicon_corrector(["France", "Germany", "Spain"]), ... stop_when=stop_on_correction, ... ) >>> r = fix("Frnace") # one transposition from a closed enum >>> r.value 'France' >>> r.corrected True >>> r.findings[0].severity is Severity.CORRECT True ``` ### Module Attributes | [`Layer`](_autosummary/ek.validate.html.md#ek.validate.Layer) | a value (and optional spec) -> zero+ Findings. | |----------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------| | [`DEFAULT_LEXICON_THRESHOLD`](_autosummary/ek.validate.html.md#ek.validate.DEFAULT_LEXICON_THRESHOLD) | Default minimum similarity (`rapidfuzz` ratio in [0, 1]) for a closed-set lexicon match to be auto-applied as a correction rather than flagged. | | [`BENFORD_EXPECTED`](_autosummary/ek.validate.html.md#ek.validate.BENFORD_EXPECTED) | Benford's law expected first-significant-digit frequencies, `P(d)=log10(1+1/d)`. | | [`DEFAULT_BENFORD_TOL`](_autosummary/ek.validate.html.md#ek.validate.DEFAULT_BENFORD_TOL) | Default max absolute per-digit deviation from Benford before a field is flagged. | | [`DEFAULT_BENFORD_MIN_N`](_autosummary/ek.validate.html.md#ek.validate.DEFAULT_BENFORD_MIN_N) | Default minimum sample size before applying Benford (the law is asymptotic). | | [`DEFAULT_ZSCORE_THRESHOLD`](_autosummary/ek.validate.html.md#ek.validate.DEFAULT_ZSCORE_THRESHOLD) | Modified (median/MAD) z-score above which a value is flagged an outlier (Iglewicz & Hoaglin recommend 3.5). | ### Functions | [`benford_findings`](_autosummary/ek.validate.html.md#ek.validate.benford_findings)(numbers, \*[, field, tol, ...]) | FLAG a numeric field whose first-digit distribution deviates from Benford's law. | |-----------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------| | [`canonicalize_corrector`](_autosummary/ek.validate.html.md#ek.validate.canonicalize_corrector)(normalize, \*[, ...]) | L0: fold a value to canonical form, emitting a CORRECT finding when it changes. | | [`cross_field_validator`](_autosummary/ek.validate.html.md#ek.validate.cross_field_validator)(predicate, \*, message) | FLAG a record when `predicate(record)` is False (a general cross-field check). | | [`lexicon_corrector`](_autosummary/ek.validate.html.md#ek.validate.lexicon_corrector)(vocabulary, \*[, ...]) | L2: resolve a value against a **closed** vocabulary by fuzzy match (`rapidfuzz`). | | [`llm_corrector`](_autosummary/ek.validate.html.md#ek.validate.llm_corrector)(correct_fn, \*[, only_flagged, ...]) | L5: a **gated** neural/LLM corrector -- the only layer that invents content. | | [`lm_surprisal_validator`](_autosummary/ek.validate.html.md#ek.validate.lm_surprisal_validator)(scorer, \*, threshold) | L3: FLAG a value whose language-model surprisal crosses `threshold`. | | [`ordering_validator`](_autosummary/ek.validate.html.md#ek.validate.ordering_validator)(keys, \*[, strict, layer]) | FLAG a record whose comparable values at `keys` are not ascending. | | [`stop_on_correction`](_autosummary/ek.validate.html.md#ek.validate.stop_on_correction)(findings) | Stop policy: end the pass as soon as any layer proposes a correction. | | [`stop_on_flag`](_autosummary/ek.validate.html.md#ek.validate.stop_on_flag)(findings) | Stop policy: end the pass at the first FLAG (fail fast). | | [`validate`](_autosummary/ek.validate.html.md#ek.validate.validate)(value, \*[, layers, spec, ...]) | Run one value through a validation/correction pipeline (progressive disclosure). | | [`validation_pipeline`](_autosummary/ek.validate.html.md#ek.validate.validation_pipeline)(\*layers[, ...]) | Compose validators/correctors into a cheapest -> most-expensive pipeline. | | [`zscore_anomaly_findings`](_autosummary/ek.validate.html.md#ek.validate.zscore_anomaly_findings)(numbers, \*[, field, ...]) | FLAG numeric values that are robust-z-score outliers (median + MAD based). | ### Classes | [`Corrector`](_autosummary/ek.validate.html.md#ek.validate.Corrector)(\*args, \*\*kwargs) | A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that may also CORRECT. | |------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------| | [`ValidationResult`](_autosummary/ek.validate.html.md#ek.validate.ValidationResult)(original, value[, findings]) | The outcome of running a value through a [`validation_pipeline()`](_autosummary/ek.validate.html.md#ek.validate.validation_pipeline). | ### ek.validate.BENFORD_EXPECTED *= {1: 0.3010299956639812, 2: 0.17609125905568124, 3: 0.12493873660829992, 4: 0.09691001300805642, 5: 0.07918124604762482, 6: 0.06694678963061322, 7: 0.05799194697768673, 8: 0.05115252244738129, 9: 0.04575749056067514}* Benford’s law expected first-significant-digit frequencies, `P(d)=log10(1+1/d)`. ### *class* ek.validate.Corrector(\*args, \*\*kwargs) Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol) A [`Validator`](_autosummary/ek.base.html.md#ek.base.Validator) that may also CORRECT. Same callable shape as a Validator, but it can emit a `severity=CORRECT` [`Finding`](_autosummary/ek.base.html.md#ek.base.Finding) whose `suggestion` is the proposed replacement value. The `layer` attribute names its place on the cheap->expensive spine. ### ek.validate.DEFAULT_BENFORD_MIN_N *= 30* Default minimum sample size before applying Benford (the law is asymptotic). ### ek.validate.DEFAULT_BENFORD_TOL *= 0.15* Default max absolute per-digit deviation from Benford before a field is flagged. ### ek.validate.DEFAULT_LEXICON_THRESHOLD *= 0.8* Default minimum similarity (`rapidfuzz` ratio in [0, 1]) for a closed-set lexicon match to be auto-applied as a correction rather than flagged. 0.8 catches typical single-edit and transposition typos (“Frnace”->”France” ~0.83) while the wide gap to an unrelated word (~0.3) keeps false corrections rare. ### ek.validate.DEFAULT_ZSCORE_THRESHOLD *= 3.5* Modified (median/MAD) z-score above which a value is flagged an outlier (Iglewicz & Hoaglin recommend 3.5). ### ek.validate.Layer a value (and optional spec) -> zero+ Findings. * **Type:** A Validator/Corrector callable alias of `Callable`[[…], [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)]] ### *class* ek.validate.ValidationResult(original, value, findings=()) Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) The outcome of running a value through a [`validation_pipeline()`](_autosummary/ek.validate.html.md#ek.validate.validation_pipeline). `value` is the final value after any applied corrections; `original` is the input; `findings` is the full audit trail (FLAG and CORRECT), in layer order. #### *property* clean *: [bool](https://docs.python.org/3/builtins/functions.html#bool)* No findings at all – the value passed every layer untouched. #### *property* corrected *: [bool](https://docs.python.org/3/builtins/functions.html#bool)* Whether any correction changed the value. #### *property* flagged *: [bool](https://docs.python.org/3/builtins/functions.html#bool)* Whether any layer raised a FLAG (something a human should look at). ### ek.validate.benford_findings(numbers, , field='', tol=0.15, min_n=30, layer='anomaly') FLAG a numeric field whose first-digit distribution deviates from Benford’s law. A reference-free anomaly check for naturally-occurring magnitudes (amounts, populations, counts): the first significant digit should follow `P(d)=log10(1+1/d)`. A large deviation flags fabricated or systematically-wrong data. **Corpus-level** (takes the whole column of values) and **FLAG-only** – it routes a field to review, never auto-edits, and can false-positive on legitimately bounded/skewed fields. Skipped (returns `[]`) below `min_n` usable values, since the law is asymptotic. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list) ### ek.validate.canonicalize_corrector(normalize, , field_name='', layer='canonicalize') L0: fold a value to canonical form, emitting a CORRECT finding when it changes. `normalize` is anything [`ek.canonicalize.resolve_canonicalizer()`](_autosummary/ek.canonicalize.html.md#ek.canonicalize.resolve_canonicalizer) accepts (a registered name, a callable, a step list). The narrowest, safest corrector – run it first so later layers compare canonical forms. * **Return type:** [`Corrector`](_autosummary/ek.validate.html.md#ek.validate.Corrector) ### ek.validate.cross_field_validator(predicate, , message, fields=(), layer='cross_field') FLAG a record when `predicate(record)` is False (a general cross-field check). Skips silently when `predicate` raises on a partial record (a missing field is not itself a cross-field violation). * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)]] ### ek.validate.lexicon_corrector(vocabulary, , threshold=0.8, scorer=None, field_name='', layer='lexicon', flag_unmatched=True) L2: resolve a value against a **closed** vocabulary by fuzzy match (`rapidfuzz`). The safest corrector in the stack when the candidate set is closed (country codes, SKUs, enums): an exact member passes untouched; a single close match (similarity `>= threshold`) is applied as a CORRECT finding; otherwise the value is FLAGged as out-of-vocabulary (when `flag_unmatched`). Against an *open* vocabulary, set `flag_unmatched=False` and treat matches as suggestions only. * **Parameters:** * **vocabulary** ([`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – The closed set of valid values. * **threshold** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Minimum `rapidfuzz` similarity in `[0, 1]` to auto-correct. * **scorer** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – A `rapidfuzz` scorer (default `fuzz.ratio`). * **flag_unmatched** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – FLAG a value with no close match (default `True`). * **Return type:** [`Corrector`](_autosummary/ek.validate.html.md#ek.validate.Corrector) ### ek.validate.llm_corrector(correct_fn, , only_flagged=True, field_name='', layer='llm_correct') L5: a **gated** neural/LLM corrector – the only layer that invents content. `correct_fn` is an **injected** `(value) -> Optional[str]` (bring your own LLM/seq2seq call; return a corrected string, or `None` to leave the value unchanged). It is the most expensive and stochastic layer, so by default (`only_flagged`) it fires \*\*only on a value that a cheaper layer already FLAGged\*\* – gate it to the residual, never let it silently rewrite clean fields (this layer reads the pipeline’s accumulated findings via `wants_findings`). Emits a CORRECT finding; the pipeline applies the rewrite and keeps the original for audit. Always verify its output downstream – it can make text worse. Recipe (Anthropic): `correct_fn=lambda v: client.messages.create(model=..., messages=[{"role": "user", "content": prompt(v)}]).content[0].text.strip()` with `client = anthropic.Anthropic()` (MIT). Constrain/verify the result and keep an audit trail. * **Return type:** [`Corrector`](_autosummary/ek.validate.html.md#ek.validate.Corrector) ### ek.validate.lm_surprisal_validator(scorer, , threshold, higher_is_worse=True, field_name='', layer='lm_prior') L3: FLAG a value whose language-model surprisal crosses `threshold`. `scorer` is an **injected** `(str) -> float` returning a surprisal / perplexity / negative-log-likelihood (dependency injection: bring your own in-domain n-gram or masked-LM scorer – an *in-domain* prior is what makes this useful). FLAG-only on its own; pair it with a candidate generator (`lexicon_corrector` / `llm_corrector`) to actually correct. Set `higher_is_worse=False` if your scorer returns a probability (higher = better). Recipe (masked-LM pseudo-log-likelihood): wrap `minicons` – `from minicons import scorer; m = scorer.MaskedLMScorer("bert-base-uncased", "cpu")` – and pass `scorer=lambda s: -m.sequence_score([s])[0]` (surprisal = negative PLL). `minicons` is MIT; install your own LM backend (it pulls torch). * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)]] ### ek.validate.ordering_validator(keys, , strict=True, layer='cross_field') FLAG a record whose comparable values at `keys` are not ascending. The canonical date/sequence check (`start <= end`, issue before due). Only the keys actually present and mutually comparable are checked, so it is safe on partial records. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Finding`](_autosummary/ek.base.html.md#ek.base.Finding)]] ### ek.validate.stop_on_correction(findings) Stop policy: end the pass as soon as any layer proposes a correction. * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.validate.stop_on_flag(findings) Stop policy: end the pass at the first FLAG (fail fast). * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### ek.validate.validate(value, \*, layers=(), spec=None, apply_corrections=True, stop_when=) Run one value through a validation/correction pipeline (progressive disclosure). The simple call `validate(v, layers=[...])` Just Works; tune the correction behaviour with `apply_corrections`/`stop_when`. Compose `layers` from the factories in this module (`canonicalize_corrector`, `lexicon_corrector`, the re-exported verifier validators) in cheapest-first order. * **Return type:** [`ValidationResult`](_autosummary/ek.validate.html.md#ek.validate.ValidationResult) ### ek.validate.validation_pipeline(\*layers, apply_corrections=True, stop_when=) Compose validators/correctors into a cheapest -> most-expensive pipeline. `layers` run in the given order (the order encodes the cost spine). Each layer is a `(value, *, spec) -> Iterable[Finding]` callable. When `apply_corrections` (default), the first `CORRECT` finding a layer emits is applied to the value before the next layer runs – so a cheap deterministic fix is what the expensive layers then see (the noisy-channel chain). `stop_when` is an injectable early-exit policy over the findings accumulated so far (see [`stop_on_correction()`](_autosummary/ek.validate.html.md#ek.validate.stop_on_correction) / [`stop_on_flag()`](_autosummary/ek.validate.html.md#ek.validate.stop_on_flag)); the default runs every layer. * **Return type:** [`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`ValidationResult`](_autosummary/ek.validate.html.md#ek.validate.ValidationResult)] ### ek.validate.zscore_anomaly_findings(numbers, , field='', threshold=3.5, min_n=30, layer='anomaly') FLAG numeric values that are robust-z-score outliers (median + MAD based). A reference-free, **dependency-free** anomaly check that complements [`benford_findings()`](_autosummary/ek.validate.html.md#ek.validate.benford_findings) (which checks the leading-digit *distribution*; this checks individual *magnitudes*). For a numeric column it computes each value’s **modified z-score** – `0.6745 * (x - median) / MAD` – which is robust to the very outliers it is looking for, and FLAGs those whose absolute score exceeds `threshold`. One FLAG per outlying value (carrying its index); **FLAG-only** – it routes a value to review, never edits. Skipped below `min_n` (robust statistics on a handful of points are noise). When the MAD is 0 (a *near*-constant column, e.g. many identical values plus one outlier) it falls back to the mean-absolute-deviation scale (Iglewicz & Hoaglin), so a lone outlier is still caught; a *truly* constant column (no spread at all) yields no findings. For *multivariate* outliers, plug an isolation forest via `pyod` (already in `ek[validation]`) as a custom validator. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list) # about-this-build.html.md # About this build This documentation was built on **2026-10-14 03:58 UTC** from commit 3c8454e on branch main, for **ek 0.1.11** (from pyproject.toml). #### WARNING The documentation and the package may be misaligned: - The documented version (0.1.11) is behind the latest release on PyPI (0.1.12): `pip install ek` gives newer code than these docs describe. ## Source | | | |---------------------|------------------------------------------------------------------------------------------------------------------------------------------------------| | Commit | 3c8454e725c57eb49c22b27db1f2b21838f6dcff | | Branch | main | | Tags at this commit | none | | Working tree | clean | | Remote | https://github.com/thorwhalen/ek | ## Continuous integration | | | |--------------|--------------------------------------------------------------------------------------------| | Repository | thorwhalen/ek | | Run | 38508869172 | | Ref | refs/heads/main | | Event commit | 3c8454e725c57eb49c22b27db1f2b21838f6dcff (in the history of the built commit) | ## Tools | | | |----------|---------| | epythet | 0.2.17 | | Sphinx | 9.1.0 | | docutils | 0.22.4 | | Python | 3.12.15 | ## Configuration as resolved | | | |---------------|-------------------------------------------------------------------| | theme | auto (Sphinx theme pydata_sphinx_theme) | | accent | #006d6b | | api_generator | autosummary | | ignore | tests/, scrap/, examples/ | | agent_outputs | true | | aggregates | md | | ai_artifacts | true | ## Package on PyPI Latest release: 0.1.12, newer than the documented version (0.1.11). ## Reproduce ```bash git clone https://github.com/thorwhalen/ek && cd ek git checkout 3c8454e725c57eb49c22b27db1f2b21838f6dcff pip install "epythet==0.2.17" epythet quickstart . --ignore tests/ scrap/ examples/ ``` The same data, for machines: build_info.json (schema version 1). # ai-agents.html.md # For AI agents `ek` ships artifacts for coding agents alongside its code. This page lists them, says where each lives in the repository, and points at the machine-readable copies of this documentation. ## Skills Skills are folders holding a `SKILL.md` (the [Agent Skills](https://agentskills.io) format): a description that tells an agent when to use it and a body with the procedure. Install one into your agent with `gh skill` (any host: `--agent claude-code`, `copilot`, `cursor`, `codex`, `gemini`), or use the copy bundled in the wheel. ### `ek-dev-add-metric` How to add a reference-based (offline) metric to ek’s score() side — the output-type to metric to library decision table (CER/WER via jiwer, ANLS\*/anls_star, span-F1 via seqeval/nervaluate, TEDS/GriTS for tables, cost-weighted typed-graph GED via networkx/apted/zss, coref via metametric), the Metric Protocol **call**(pred, gold, \*, grammar=None) -> float, the Report/Score decomposition object, how to wrap a library behind the Protocol and register it, and the flagship cost-weighted typed-graph must-build whose weights come from Layer-A importance. Use when implementing, wrapping, registering, choosing, or debugging an offline/reference-based metric, scoring pred vs gold, slot/entity F1, table TEDS, graph edit distance, ANLS, WER/CER, canonicalization-before-scoring, match schemes, or per-slice/per-path TP/FP/FN. For online/reference-free confidence, calibration, and decisions use the estimate_quality side instead. ```bash gh skill install thorwhalen/ek ek-dev-add-metric --agent claude-code ``` Source: [`skills/ek-dev-add-metric`](https://github.com/thorwhalen/ek/tree/HEAD/skills/ek-dev-add-metric). ### `ek-dev-add-signal` How to add a reference-free quality-estimation component to ek’s online estimate_quality() side: a new Signal (deterministic verifier / intrinsic-confidence / logprob / agreement-ROVER / auxiliary-QE / LLM-judge), a Calibrator (Platt/Temperature/Isotonic), or a DecisionPolicy/SelectivePolicy (ConformalGate / RiskControlGate / CostSensitiveGate). Use when building or extending the signal->calibrate->decide pipeline, wiring a Strategy Protocol into the registry, adding confidence/uncertainty/agreement signals, ROVER voting, conformal prediction (MAPIE/crepes/TorchCP), calibration (netcal/sklearn), ECE/D-ECE, risk-coverage curves, or selective prediction / accept-flag-block gating for extraction or OCR quality. Not for offline reference-based metrics (that is the score()/add-metric side). ```bash gh skill install thorwhalen/ek ek-dev-add-signal --agent claude-code ``` Source: [`skills/ek-dev-add-signal`](https://github.com/thorwhalen/ek/tree/HEAD/skills/ek-dev-add-signal). ### `ek-dev-agents` How to build and extend ek’s AGENT/ASSISTANT evaluation instance (ek/agents/) — the second concrete instance after ek.ocr. Covers the episode object (Episode/Trajectory/Step/Cost/RunProvenance as Layer B) and the task/tool grammar (ToolSpec/TaskSpec building a GraphGrammar as Layer A, where FieldSpec.importance is the cost of a wrong tool argument), the agent metrics (TaskSuccessMetric with an injected Checker oracle, CostPerSuccessMetric implementing Cost-of-Pass E[cost]/P(success), ToolCallMetric doing BFCL AST match over a multiset of calls, TrajectoryMetric doing a Needleman-Wunsch sequence edit distance), the reliability estimators (pass@k capability vs pass^k reliability — pure functions, NOT Metrics), the LLM-as-judge seams (JudgeSignal reference-free vs JudgeMetric reference-based vs pairwise_judge, plus judge_validation reusing Krippendorff/Cohen IAA), the agent harness (run_suite with k trials, and the variance-aware agent_regression_gate using Wilson/bootstrap intervals), and the duck-typed bridges (trajectory_from_messages, from_inspect_sample). Use when implementing, debugging, or extending anything under ek/agents/, adding an agent metric/signal/checker, scoring an episode or trajectory or tool call, computing pass^k or cost per successful task, evaluating an LLM judge, or wiring an external agent harness (Inspect AI, DeepEval, Ragas). Read BEFORE touching ek/agents/. ```bash gh skill install thorwhalen/ek ek-dev-agents --agent claude-code ``` Source: [`skills/ek-dev-agents`](https://github.com/thorwhalen/ek/tree/HEAD/skills/ek-dev-agents). ### `ek-dev-architecture` Master orientation map for developers BUILDING ek (a framework for building Knowledge Evaluation systems that evaluate information-extraction outputs; OCR is the noisiest special case). Read this FIRST before touching any ek component. Covers the two-need 2x2 (reference-based vs reference-free, instance vs system), the two-layer data model (Layer A GraphGrammar schema/cost SSOT + Layer B AnnotatedExtraction metadata), the facades score()/evaluate()/estimate_quality(), the strategy-Protocol + registry + dependency-injection + requires_extra open-closed pattern, the module layout (base, registry, stores, metrics, canonicalize, qe/rover, validate, harness, ocr, tools, **main**), the config2py.AppData + dol persistence model, the must-builds (two-layer object, cost-weighted typed-graph distance, ROVER), and a which-report-for-which-component table. Use when deciding where code goes, what a component is named, how facades share the Layer-A object, how strategies plug in, or which ek_0N report to read. ```bash gh skill install thorwhalen/ek ek-dev-architecture --agent claude-code ``` Source: [`skills/ek-dev-architecture`](https://github.com/thorwhalen/ek/tree/HEAD/skills/ek-dev-architecture). ### `ek-dev-licensing` Licensing guardrail for ek developers: consult BEFORE adding ANY dependency. Covers the license-tiering policy (lean permissive core vs opt-in extras), the register of 6 license landmines (GPL Levenshtein/python-Levenshtein, LGPL TorchCP, non-commercial surya-ocr, GPL Potato, proprietary Prodigy) with their permissive replacements, scanner-invisible traps where license terms live in repo files not PyPI metadata (zss BSD-3, mistralai namespace pkg, PubTabNet CDLA), the rule that nothing copyleft or non-commercial is ever a default, and the CI license gate (pip-licenses/reuse) that fails the build on GPL/AGPL/non-commercial in core..hitl. Use when picking a metric/calibration/validation/OCR/agreement/HITL backend, adding to pyproject extras, choosing an edit-distance or conformal or annotation library, hitting a “GPL”/”non-commercial”/”RAIL”/”copyleft” license, deciding which extra a dep belongs in, or whether a dep needs import-only quarantine. ```bash gh skill install thorwhalen/ek ek-dev-licensing --agent claude-code ``` Source: [`skills/ek-dev-licensing`](https://github.com/thorwhalen/ek/tree/HEAD/skills/ek-dev-licensing). ### `ek-dev-ocr` Build and maintain ek’s OCR evaluation layer on top of the ocracy package — the concrete instance of ek’s reference-based score() and reference-free estimate_quality() facades applied to image->OcrResult engines. Use when working on the ek[ocr] optional extra, wiring or running the OCR offline benchmark (CER/WER/ANLS per-slice over a gold corpus), mapping OcrResult/TextBlock into AnnotatedExtraction/FieldEstimate, deciding which ocracy backend to evaluate, handling null-safe VLM engines (Claude/GPT-4o/Mistral/Mathpix returning text-only, blocks=[], confidence=None), reading per-engine capability profiles (who emits real vs model-guessed geometry, who is calibrated, who emits TEDS-able tables), gating remote engines that make paid API calls, dealing with the missing normalized table type, or anything touching ek<->ocracy. Enforces the dependency-direction rule: ek depends on ocracy via an extra; never ocracy->ek; ek core depends only on the OcrResult shape so it evaluates any image->OcrResult callable. ```bash gh skill install thorwhalen/ek ek-dev-ocr --agent claude-code ``` Source: [`skills/ek-dev-ocr`](https://github.com/thorwhalen/ek/tree/HEAD/skills/ek-dev-ocr). ## Instruction files Files agents read before working in this repository. - [`.claude/CLAUDE.md`](https://github.com/thorwhalen/ek/tree/HEAD/.claude/CLAUDE.md): read by Claude Code - [`AGENTS.md`](https://github.com/thorwhalen/ek/tree/HEAD/AGENTS.md): read by Codex, Copilot, Cursor and other agents ## Machine-readable documentation This site publishes the same documentation in forms that fit an agent’s context window: - [`llms.txt`](https://thorwhalen.github.io/ek/llms.txt): an index of every page with a one-line description ([llms.txt](https://llmstxt.org) format) - [`ek.md`](https://thorwhalen.github.io/ek/ek.md): the whole documentation as one Markdown file - `.html.md`: a rendered Markdown twin of every page, advertised from each page’s `` with `` - [`objects.inv`](https://thorwhalen.github.io/ek/objects.inv): the Sphinx inventory: a symbol-to-URL index (`sphobjinv convert plain objects.inv -`) # api.html.md # API reference | [`ek`](_autosummary/ek.html.md#module-ek) | `ek` -- a framework for building Knowledge Evaluation systems. | |-----------------------------------------------------------------|------------------------------------------------------------------|