ek.facade

The two top-level facades: score() (offline) and estimate_quality() (online).

Both operate on the same Layer-A object and follow progressive disclosure – the simple call Just Works, every strategy is replaceable by keyword:

  • score() – reference-based: compare one prediction to one gold reference, the metric chosen by output type (string -> CER, record -> field-F1) unless you name or pass one.

  • evaluate() – reference-based at corpus scale: aggregate many comparisons correctly (global error-rate accumulation, micro-F1), with optional per-slice cuts. This is what the OCR benchmark and the regression harness build on.

  • estimate_quality() – reference-free: gather signals -> calibrate -> validate -> decide, returning a QualityReport. The heavy signal families (ROVER, conformal gates) are injected; this facade composes whatever you give it and ships sensible no-op defaults.

Example

>>> score("hello wrld", "hello world").metric
'cer'
>>> round(score("hello wrld", "hello world", metric="wer").value, 3)
0.5
>>> r = evaluate([("ct", "cat"), ("dg", "dog")], metric="cer")
>>> r.n, round(r.aggregate, 3)
(2, 0.333)
ek.facade.estimate_quality(extraction: Any, *, sources: Iterable = (), signals: Iterable = (), calibrator=None, validators: Iterable = (), policy=None, agreement: bool = True, assume_calibrated: bool = False) → QualityReport[source]

Estimate the quality of an extraction with no gold reference.

Composes the strict signal -> calibrate -> validate -> decide pipeline (misc/docs/ek_03) over a single value, a FieldEstimate, or a whole AnnotatedExtraction (scored per field, with per-field specs feeding the validators and the node type as the Mondrian group key). The simple call estimate_quality(value) Just Works; every stage is injectable.

Parameters:

  • extraction – A raw value, a FieldEstimate, or an AnnotatedExtraction.

  • sources – Additional hypotheses of the same content (strings or OcrResult-shaped objects) to fuse with ROVER – their mean agreement becomes a raw_signals["agreement"] entry (the flagship online signal).

  • signals – Explicit Signal callables target -> float | Mapping producing further raw signals.

  • calibrator – A Calibrator mapping raw score -> probability. Calibration is non-optional before gating (Hard Rule 1): passing a policy with no calibrator and assume_calibrated false raises – a raw, uncalibrated score must never reach a DecisionPolicy.

  • validators – Validator callables yielding findings. A flat iterable runs on every field (use spec-driven validators like schema_validator()). For per-field scoping pass a Mapping keyed by field name or node type ("*" runs on all).

  • policy – A DecisionPolicy producing accept/flag/block from the calibrated confidence.

  • agreement – Auto-run ROVER over sources when any are given (default true).

  • assume_calibrated – Treat the incoming confidence as already calibrated (silences the uncalibrated-gating warning).

ek.facade.evaluate(cases: Iterable, *, metric: None | str | Metric = None, grammar: GraphGrammar | None = None, normalize: Any = None, weights: Any = None) → Report[source]

Aggregate many comparisons into a Report (corpus level).

Parameters:

  • cases – Iterable of (pred, gold) or (pred, gold, slice_label) tuples.

  • metric – As in score() (resolved once from the first case’s types).

  • grammar – Optional Layer-A grammar passed to every comparison.

  • normalize – Optional canonicalizer applied before every comparison.

Returns:

A Report whose aggregate is computed by the metric’s own aggregator (e.g. globally accumulated CER/WER, micro-F1) – never a naive mean.

ek.facade.score(pred: Any, gold: Any, *, grammar: GraphGrammar | None = None, metric: None | str | Metric = None, normalize: Any = None, weights: Any = None) → Score[source]

Score one prediction against one gold reference (reference-based).

Parameters:

  • pred – The predicted output (string, record dict, or anything with .text).

  • gold – The gold reference, same shape as pred.

  • grammar – Optional Layer-A GraphGrammar (carries cost weights).

  • metric – A registered metric name ("cer", "wer", "fields", …), a callable Metric, or None to dispatch by type.

  • normalize – Optional canonicalizer (name, callable, step list, or Canonicalizer) applied before comparison.

  • weights – A CostWeight for cost-weighted metrics (the typed-graph distance); overrides the schema’s importance weights.

Returns:

A Score.