ek.qe.rover

ROVER: multi-engine consensus + per-position agreement (the online flagship).

NIST’s Recognizer Output Voting Error Reduction (Fiscus, 1997) combines the outputs of several recognizers/extractors when no reference is available. It runs in two stages: an alignment stage that merges the hypotheses into a single word transition network (WTN) via iterative dynamic-programming alignments, and a voting stage that picks, per branch point, the best-scoring word – by vote frequency alone, or blended with confidence. There is no maintained, permissively licensed Python ROVER to depend on, so ek builds it on the edit-distance primitives it already ships (pure-Python here for null-safety and zero new deps).

Two outputs come out of one pass:

• a consensus transcription (often lower error than any single engine), and

• a per-position agreement score in [0, 1] – the fraction of engines that voted for the winning token at each slot. Positions where engines disagree are exactly the positions to flag, so this doubles as a reference-free confidence signal (an AgreementSignal for ek.estimate_quality()).

ek depends only on the OcrResult shape (.text / .blocks / .mean_confidence), so ROVER fuses any image -> OcrResult callable – or plain strings, token lists, or (token, confidence) lists. The alignment is O(N * l * L * L') in the engine count and sequence lengths, hence designed for a handful of engines (its historical regime). Alignment is incremental and therefore order-dependent – a documented property of ROVER, not a bug.

See misc/docs/ek_03 (and ek_04 for cross-source corroboration).

Example

>>> c = rover(["the cat sat", "the cat sit", "the bat sat"])
>>> c.text
'the cat sat'
>>> [round(a, 3) for a in c.agreement]            # per consensus token
[1.0, 0.667, 0.667]
>>> round(c.mean_agreement, 3)
0.778

class ek.qe.rover.AgreementSignal(cost_tier: int = 3, use_confidence: bool = True, conf_weight: float = 0.5, null_conf: float = 0.7, tokenize: Callable[[Any], List[Tuple[str | None, float]]] | None = None, max_tokens: int | None = 5000)

ROVER multi-engine agreement as a reference-free Signal.

Cost tier 3 (N engine runs): try the deterministic verifier layer and any free intrinsic confidence first. Called on a collection of hypotheses, it runs rover() and returns the mean per-position agreement as the raw signal – uncalibrated, like every signal, so it must pass through a Calibrator before any gate reads it.

Example

>>> sig = AgreementSignal()
>>> round(sig(["the cat sat", "the cat sit", "the bat sat"]), 3)
0.778

ek.qe.rover.DEFAULT_CONF_WEIGHT = 0.5

Default blend of average confidence vs vote frequency in the slot score (Fiscus’s 1 - alpha); 0 is frequency-only, 1 is confidence-only.

ek.qe.rover.DEFAULT_MAX_TOKENS = 5000

Default per-hypothesis token cap for the O(N*l*L*L’) aligner (DoS guard).

ek.qe.rover.DEFAULT_NULL_CONF = 0.7

Default confidence credited to a NULL vote (an engine that emitted no token at a slot) – the lever for how readily a deletion wins.

ek.qe.rover.DEFAULT_TOKEN_CONF = 1.0

Confidence assigned to a present token whose source reports no usable confidence (e.g. a VLM/markdown OcrResult with confidence is None).

class ek.qe.rover.RoverConsensus(tokens: List[str] = <factory>, slots: List[RoverSlot] = <factory>, agreement: List[float] = <factory>, n_engines: int = 0)

The result of a ROVER pass over N hypotheses.

property mean_agreement: float

Mean per-position agreement over consensus tokens, a reference-free confidence in [0, 1]: 1.0 means every engine agreed at every emitted position; lower means at least one engine dissented.

An empty consensus (no token survived voting – NULL won every slot) means the engines agreed on nothing, so this returns 0.0 when 2+ engines were fused, NOT a misleading 1.0 (which would feed a maximal raw confidence to the calibrator/gate). A single engine – or none – is vacuously 1.0.

property text: str

The consensus transcription (space-joined winning tokens).

class ek.qe.rover.RoverSlot(entries: Tuple[str | None, float]]=<factory>, winner: str | None = None, score: float = 0.0, vote_share: float = 0.0)

One branch point of the transition network: each engine’s token (or NULL).

ek.qe.rover.rover(hypotheses: Iterable[Any], *, use_confidence: bool = True, conf_weight: float = 0.5, null_conf: float = 0.7, tokenize: Callable[[Any], List[Tuple[str | None, float]]] | None = None, max_tokens: int | None = 5000) → RoverConsensus

Align N hypotheses, vote per slot, and emit consensus + per-position agreement.

Parameters:

• hypotheses – The recognizer/extractor outputs to fuse. Each may be a string, an OcrResult-shaped object (.text/.mean_confidence), a list of token strings, or a list of (token, confidence) pairs.

• use_confidence – Blend confidence into the vote (True) or vote purely by frequency (False, which forces conf_weight to 0).

• conf_weight – Weight on average confidence vs vote frequency in the slot score (Fiscus’s 1 - alpha); 0 is frequency-only, 1 is confidence-only. Ignored when use_confidence is False.

• null_conf – Confidence credited to a NULL vote (an engine that produced no token at a slot) – the lever for how readily a deletion wins.

• tokenize – Optional hypothesis -> [(token, confidence), ...] override; by default _as_units() handles strings/OcrResults/token lists.

• max_tokens – Reject any hypothesis longer than this (the aligner is O(N*l*L*L'); an unbounded input is a quadratic time/memory DoS). None disables the guard.

Returns:

A RoverConsensus with the consensus tokens/text, the per-slot breakdown, and the per-consensus-token agreement usable as a raw signal.