ek.qe.calibrate

Calibration: make a raw score mean a probability (the non-optional stage).

A confidence of 0.9 should mean “correct 90% of the time.” Modern models violate this badly – they are systematically overconfident (Guo et al. 2017), and RLHF makes LLMs worse. So calibration is non-optional: never gate a raw posterior or a raw logprob (misc/docs/ek_03 §2, Hard Rule 1). A Calibrator is fit on a labelled holdout of (raw_score, field_correct?) pairs and maps any later raw score to a calibrated probability.

Three methods, by what input you have:

• PlattCalibrator – logistic fit on any scalar score (no logits needed): the default for aggregated OCR confidence or aggregated logprobs.

• IsotonicCalibrator – non-parametric monotonic fit; more flexible, needs more data, can overfit small sets.

• TemperatureCalibrator – a single scalar T on logits; use only when you have logits (it does not change the argmax).

All three are pure-Python (stdlib only) so the calibration stage works with zero extra dependencies; sklearn_calibrator() / netcal_calibrator() offer the library-backed equivalents behind the ek[calibration] extra. Measure calibration with expected_calibration_error() (+ a reliability curve). For per-field-type validity, wrap per group with GroupCalibrator (Mondrian / class-conditional) – distribution-free conditional coverage is otherwise impossible (Hard Rule 2).

Calibrate at the granularity of the decision (gate on fields -> calibrate a “field-correct?” target), and persist the fit (save_calibrator()); calibration is dataset-specific and decays, so re-fit on drift.

Example

>>> # An overconfident raw signal, calibrated against ground truth.
>>> raw =     [0.95, 0.93, 0.92, 0.90, 0.55, 0.52, 0.51, 0.50]
>>> correct = [True, True, False, False, True, False, False, False]
>>> cal = PlattCalibrator().fit(raw, correct)
>>> cal(0.95) < 0.95            # overconfidence pulled down
True
>>> 0.0 <= cal(0.5) <= 1.0
True

ek.qe.calibrate.DEFAULT_N_BINS = 10

Default number of bins for expected_calibration_error() / reliability curves.

class ek.qe.calibrate.GroupCalibrator(factory: Callable[[], ~typing.Any]=<class 'ek.qe.calibrate.PlattCalibrator'>, by_group: dict = <factory>, pooled: Any = None, kind: str = 'group')

Per-group (Mondrian) calibration: one calibrator per NodeType/FieldSpec.

Distribution-free conditional (per-field-type) coverage is impossible in general (Barber et al. 2019); calibrating separately per group restores it approximately. fit takes a parallel groups sequence; __call__ routes by group key, falling back to a pooled calibrator for unseen groups.

Parameters:

factory – Zero-arg callable producing a fresh per-group calibrator (default PlattCalibrator).

factory

alias of PlattCalibrator

fit(scores: Sequence[float], correct: Sequence[bool], *, groups: Sequence[Any]) → GroupCalibrator

Fit one calibrator per distinct group key, plus a pooled fallback.

to_dict() → dict

Serialize each per-group calibrator (keyed by stringified group) plus the pooled fallback, so a fitted Mondrian calibrator round-trips like the others. Group keys are stringified (they are NodeType/FieldSpec names).

class ek.qe.calibrate.IsotonicCalibrator(x: List[float] = <factory>, y: List[float] = <factory>, kind: str = 'isotonic')

Isotonic (monotonic non-decreasing) calibration via pool-adjacent-violators.

More flexible than Platt; needs more calibration data and can overfit small sets. Predicts by linear interpolation between fitted points, clipped at the ends.

fit(scores: Sequence[float], correct: Sequence[bool]) → IsotonicCalibrator

Fit the monotonic step function to (score, correct) pairs.

class ek.qe.calibrate.PlattCalibrator(a: float = 1.0, b: float = 0.0, max_iter: int = 100, kind: str = 'platt')

Platt scaling: sigmoid(a * score + b), fit by Newton/IRLS on labels.

The default calibrator: works on any scalar (aggregated confidence/logprob), no logits required. Uses Platt’s target smoothing so it does not overfit small calibration sets.

Parameters:

max_iter – Newton iterations (converges in a handful for a 2-parameter model).

fit(scores: Sequence[float], correct: Sequence[bool]) → PlattCalibrator

Fit a, b to maximise the likelihood of correct given scores.

class ek.qe.calibrate.TemperatureCalibrator(T: float = 1.0, kind: str = 'temperature', t_min: float = 0.05, t_max: float = 10.0, max_iter: int = 60)

Temperature scaling: sigmoid(logit / T) with one T fit on a holdout.

Use only when you have logits: __call__ expects a logit, not a probability. T > 1 softens overconfidence; the argmax is unchanged.

fit(logits: Sequence[float], correct: Sequence[bool]) → TemperatureCalibrator

Fit T by minimising NLL with a bounded 1-D search.

t_min: float = 0.05

Golden-section search bounds and iteration count (config, not magic numbers).

ek.qe.calibrate.expected_calibration_error(probs: Sequence[float], correct: Sequence[bool], *, n_bins: int = 10) → float

Expected Calibration Error: weighted mean gap between confidence and accuracy.

Bins predictions by confidence into n_bins equal-width bins and averages |mean_confidence - accuracy| weighted by bin population. 0 is perfect. Non-finite probs are skipped; out-of-range probs are clamped into [0, 1].

ek.qe.calibrate.load_calibrator(name: str, *, rootdir: str | None = None) → Any

Reconstruct a persisted calibrator by name (dispatched on its kind).

Validates the stored record so a malformed or unknown kind fails with an actionable error rather than a raw KeyError or a load-then-crash-later.

ek.qe.calibrate.netcal_ece(probs: Sequence[float], correct: Sequence[bool], *, bins: int = 10) → float

ECE via netcal (behind ek[calibration]); D-ECE for localized outputs lives there too.

ek.qe.calibrate.reliability_curve(probs: Sequence[float], correct: Sequence[bool], *, n_bins: int = 10) → List[dict]

Per-bin {confidence, accuracy, count} for a reliability diagram.

ek.qe.calibrate.save_calibrator(calibrator: Any, name: str, *, rootdir: str | None = None) → dict

Persist a fitted calibrator’s parameters to the calibrators store.

ek.qe.calibrate.sklearn_calibrator(method: str = 'sigmoid')

A calibrator backed by scikit-learn (method='sigmoid' Platt or 'isotonic').

Behind ek[calibration]. Returns an object satisfying the Calibrator protocol that wraps sklearn’s calibration. The pure-Python PlattCalibrator/IsotonicCalibrator are the dependency-free defaults; use this for parity with an sklearn-centric stack.