ek.facade#

The two top-level facades: score() (offline) and estimate_quality() (online).

Both operate on the same Layer-A object and follow progressive disclosure – the simple call Just Works, every strategy is replaceable by keyword:

  • score() – reference-based: compare one prediction to one gold reference, the metric chosen by output type (string -> CER, record -> field-F1) unless you name or pass one.

  • evaluate() – reference-based at corpus scale: aggregate many comparisons correctly (global error-rate accumulation, micro-F1), with optional per-slice cuts. This is what the OCR benchmark and the regression harness build on.

  • estimate_quality() – reference-free: gather signals -> calibrate -> validate -> decide, returning a QualityReport. The heavy signal families (ROVER, conformal gates) are injected; this facade composes whatever you give it and ships sensible no-op defaults.

Example

>>> score("hello wrld", "hello world").metric
'cer'
>>> round(score("hello wrld", "hello world", metric="wer").value, 3)
0.5
>>> r = evaluate([("ct", "cat"), ("dg", "dog")], metric="cer")
>>> r.n, round(r.aggregate, 3)
(2, 0.333)

Functions

estimate_quality(extraction, *[, sources, ...])

Estimate the quality of an extraction with no gold reference.

evaluate(cases, *[, metric, grammar, ...])

Aggregate many comparisons into a Report (corpus level).

score(pred, gold, *[, grammar, metric, ...])

Score one prediction against one gold reference (reference-based).
ek.facade.estimate_quality(extraction, *, sources=(), signals=(), calibrator=None, validators=(), policy=None, agreement=True, assume_calibrated=False)[source]#

Estimate the quality of an extraction with no gold reference.

Composes the strict signal -> calibrate -> validate -> decide pipeline (misc/docs/ek_03) over a single value, a FieldEstimate, or a whole AnnotatedExtraction (scored per field, with per-field specs feeding the validators and the node type as the Mondrian group key). The simple call estimate_quality(value) Just Works; every stage is injectable.

Parameters:

  • extraction (Any) – A raw value, a FieldEstimate, or an AnnotatedExtraction.

  • sources (Iterable) – Additional hypotheses of the same content (strings or OcrResult-shaped objects) to fuse with ROVER – their mean agreement becomes a raw_signals["agreement"] entry (the flagship online signal).

  • signals (Iterable) – Explicit Signal callables target -> float | Mapping producing further raw signals.

  • calibrator – A Calibrator mapping raw score -> probability, applied to the field’s confidence (else the mean of its raw signals). When the signals are heterogeneous (e.g. an LLM-judge score next to a logprob), pass a Mapping[str, Calibrator] keyed by signal name (a "*" entry is the fallback) or a SignalFamilyCalibrator: each signal family is then calibrated on its own and the calibrated probabilities are fused by its combine rule (mean by default) followed by its learned fusion calibrator. A field’s own confidence is the "confidence" family. The per-family values are reported in provenance["calibrated_signals"]. Calibration is non-optional before gating (Hard Rule 1): passing a policy with no calibrator and assume_calibrated false raises – a raw, uncalibrated score must never reach a DecisionPolicy.

  • validators (Iterable) – Validator callables yielding findings. A flat iterable runs on every field (use spec-driven validators like schema_validator()). For per-field scoping pass a Mapping keyed by field name or node type ("*" runs on all).

  • policy – A DecisionPolicy producing accept/flag/block from the calibrated confidence.

  • agreement (bool) – Auto-run ROVER over sources when any are given (default true).

  • assume_calibrated (bool) – Treat the incoming confidence as already calibrated (silences the uncalibrated-gating warning). With a per-family calibrator it also waives the check that a value fused from two or more families went through a learned fusion calibrator.

Return type:

QualityReport

ek.facade.evaluate(cases, *, metric=None, grammar=None, normalize=None, weights=None)[source]#

Aggregate many comparisons into a Report (corpus level).

Parameters:

  • cases (Iterable) – Iterable of (pred, gold) or (pred, gold, slice_label) tuples.

  • metric (Union[None, str, Metric]) – As in score() (resolved once from the first case’s types).

  • grammar (Optional[GraphGrammar]) – Optional Layer-A grammar passed to every comparison.

  • normalize (Any) – Optional canonicalizer applied before every comparison.

Return type:

Report

Returns:

A Report whose aggregate is computed by the metric’s own aggregator (e.g. globally accumulated CER/WER, micro-F1) – never a naive mean.

ek.facade.score(pred, gold, *, grammar=None, metric=None, normalize=None, weights=None)[source]#

Score one prediction against one gold reference (reference-based).

Parameters:

  • pred (Any) – The predicted output (string, record dict, or anything with .text).

  • gold (Any) – The gold reference, same shape as pred.

  • grammar (Optional[GraphGrammar]) – Optional Layer-A GraphGrammar (carries cost weights).

  • metric (Union[None, str, Metric]) – A registered metric name ("cer", "wer", "fields", …), a callable Metric, or None to dispatch by type.

  • normalize (Any) – Optional canonicalizer (name, callable, step list, or Canonicalizer) applied before comparison.

  • weights (Any) – A CostWeight for cost-weighted metrics (the typed-graph distance); overrides the schema’s importance weights.

Return type:

Score

Returns:

A Score.