# For AI agents `ek` ships artifacts for coding agents alongside its code. This page lists them, says where each lives in the repository, and points at the machine-readable copies of this documentation. ## Skills Skills are folders holding a `SKILL.md` (the [Agent Skills](https://agentskills.io) format): a description that tells an agent when to use it and a body with the procedure. Install one into your agent with `gh skill` (any host: `--agent claude-code`, `copilot`, `cursor`, `codex`, `gemini`), or use the copy bundled in the wheel. ### `ek-dev-add-metric` How to add a reference-based (offline) metric to ek’s score() side — the output-type to metric to library decision table (CER/WER via jiwer, ANLS\*/anls_star, span-F1 via seqeval/nervaluate, TEDS/GriTS for tables, cost-weighted typed-graph GED via networkx/apted/zss, coref via metametric), the Metric Protocol **call**(pred, gold, \*, grammar=None) -> float, the Report/Score decomposition object, how to wrap a library behind the Protocol and register it, and the flagship cost-weighted typed-graph must-build whose weights come from Layer-A importance. Use when implementing, wrapping, registering, choosing, or debugging an offline/reference-based metric, scoring pred vs gold, slot/entity F1, table TEDS, graph edit distance, ANLS, WER/CER, canonicalization-before-scoring, match schemes, or per-slice/per-path TP/FP/FN. For online/reference-free confidence, calibration, and decisions use the estimate_quality side instead. ```bash gh skill install thorwhalen/ek ek-dev-add-metric --agent claude-code ``` Source: [`skills/ek-dev-add-metric`](https://github.com/thorwhalen/ek/tree/HEAD/skills/ek-dev-add-metric). ### `ek-dev-add-signal` How to add a reference-free quality-estimation component to ek’s online estimate_quality() side: a new Signal (deterministic verifier / intrinsic-confidence / logprob / agreement-ROVER / auxiliary-QE / LLM-judge), a Calibrator (Platt/Temperature/Isotonic), or a DecisionPolicy/SelectivePolicy (ConformalGate / RiskControlGate / CostSensitiveGate). Use when building or extending the signal->calibrate->decide pipeline, wiring a Strategy Protocol into the registry, adding confidence/uncertainty/agreement signals, ROVER voting, conformal prediction (MAPIE/crepes/TorchCP), calibration (netcal/sklearn), ECE/D-ECE, risk-coverage curves, or selective prediction / accept-flag-block gating for extraction or OCR quality. Not for offline reference-based metrics (that is the score()/add-metric side). ```bash gh skill install thorwhalen/ek ek-dev-add-signal --agent claude-code ``` Source: [`skills/ek-dev-add-signal`](https://github.com/thorwhalen/ek/tree/HEAD/skills/ek-dev-add-signal). ### `ek-dev-agents` How to build and extend ek’s AGENT/ASSISTANT evaluation instance (ek/agents/) — the second concrete instance after ek.ocr. Covers the episode object (Episode/Trajectory/Step/Cost/RunProvenance as Layer B) and the task/tool grammar (ToolSpec/TaskSpec building a GraphGrammar as Layer A, where FieldSpec.importance is the cost of a wrong tool argument), the agent metrics (TaskSuccessMetric with an injected Checker oracle, CostPerSuccessMetric implementing Cost-of-Pass E[cost]/P(success), ToolCallMetric doing BFCL AST match over a multiset of calls, TrajectoryMetric doing a Needleman-Wunsch sequence edit distance), the reliability estimators (pass@k capability vs pass^k reliability — pure functions, NOT Metrics), the LLM-as-judge seams (JudgeSignal reference-free vs JudgeMetric reference-based vs pairwise_judge, plus judge_validation reusing Krippendorff/Cohen IAA), the agent harness (run_suite with k trials, and the variance-aware agent_regression_gate using Wilson/bootstrap intervals), and the duck-typed bridges (trajectory_from_messages, from_inspect_sample). Use when implementing, debugging, or extending anything under ek/agents/, adding an agent metric/signal/checker, scoring an episode or trajectory or tool call, computing pass^k or cost per successful task, evaluating an LLM judge, or wiring an external agent harness (Inspect AI, DeepEval, Ragas). Read BEFORE touching ek/agents/. ```bash gh skill install thorwhalen/ek ek-dev-agents --agent claude-code ``` Source: [`skills/ek-dev-agents`](https://github.com/thorwhalen/ek/tree/HEAD/skills/ek-dev-agents). ### `ek-dev-architecture` Master orientation map for developers BUILDING ek (a framework for building Knowledge Evaluation systems that evaluate information-extraction outputs; OCR is the noisiest special case). Read this FIRST before touching any ek component. Covers the two-need 2x2 (reference-based vs reference-free, instance vs system), the two-layer data model (Layer A GraphGrammar schema/cost SSOT + Layer B AnnotatedExtraction metadata), the facades score()/evaluate()/estimate_quality(), the strategy-Protocol + registry + dependency-injection + requires_extra open-closed pattern, the module layout (base, registry, stores, metrics, canonicalize, qe/rover, validate, harness, ocr, tools, **main**), the config2py.AppData + dol persistence model, the must-builds (two-layer object, cost-weighted typed-graph distance, ROVER), and a which-report-for-which-component table. Use when deciding where code goes, what a component is named, how facades share the Layer-A object, how strategies plug in, or which ek_0N report to read. ```bash gh skill install thorwhalen/ek ek-dev-architecture --agent claude-code ``` Source: [`skills/ek-dev-architecture`](https://github.com/thorwhalen/ek/tree/HEAD/skills/ek-dev-architecture). ### `ek-dev-licensing` Licensing guardrail for ek developers: consult BEFORE adding ANY dependency. Covers the license-tiering policy (lean permissive core vs opt-in extras), the register of 6 license landmines (GPL Levenshtein/python-Levenshtein, LGPL TorchCP, non-commercial surya-ocr, GPL Potato, proprietary Prodigy) with their permissive replacements, scanner-invisible traps where license terms live in repo files not PyPI metadata (zss BSD-3, mistralai namespace pkg, PubTabNet CDLA), the rule that nothing copyleft or non-commercial is ever a default, and the CI license gate (pip-licenses/reuse) that fails the build on GPL/AGPL/non-commercial in core..hitl. Use when picking a metric/calibration/validation/OCR/agreement/HITL backend, adding to pyproject extras, choosing an edit-distance or conformal or annotation library, hitting a “GPL”/”non-commercial”/”RAIL”/”copyleft” license, deciding which extra a dep belongs in, or whether a dep needs import-only quarantine. ```bash gh skill install thorwhalen/ek ek-dev-licensing --agent claude-code ``` Source: [`skills/ek-dev-licensing`](https://github.com/thorwhalen/ek/tree/HEAD/skills/ek-dev-licensing). ### `ek-dev-ocr` Build and maintain ek’s OCR evaluation layer on top of the ocracy package — the concrete instance of ek’s reference-based score() and reference-free estimate_quality() facades applied to image->OcrResult engines. Use when working on the ek[ocr] optional extra, wiring or running the OCR offline benchmark (CER/WER/ANLS per-slice over a gold corpus), mapping OcrResult/TextBlock into AnnotatedExtraction/FieldEstimate, deciding which ocracy backend to evaluate, handling null-safe VLM engines (Claude/GPT-4o/Mistral/Mathpix returning text-only, blocks=[], confidence=None), reading per-engine capability profiles (who emits real vs model-guessed geometry, who is calibrated, who emits TEDS-able tables), gating remote engines that make paid API calls, dealing with the missing normalized table type, or anything touching ek<->ocracy. Enforces the dependency-direction rule: ek depends on ocracy via an extra; never ocracy->ek; ek core depends only on the OcrResult shape so it evaluates any image->OcrResult callable. ```bash gh skill install thorwhalen/ek ek-dev-ocr --agent claude-code ``` Source: [`skills/ek-dev-ocr`](https://github.com/thorwhalen/ek/tree/HEAD/skills/ek-dev-ocr). ## Instruction files Files agents read before working in this repository. - [`.claude/CLAUDE.md`](https://github.com/thorwhalen/ek/tree/HEAD/.claude/CLAUDE.md): read by Claude Code - [`AGENTS.md`](https://github.com/thorwhalen/ek/tree/HEAD/AGENTS.md): read by Codex, Copilot, Cursor and other agents ## Machine-readable documentation This site publishes the same documentation in forms that fit an agent’s context window: - [`llms.txt`](https://thorwhalen.github.io/ek/llms.txt): an index of every page with a one-line description ([llms.txt](https://llmstxt.org) format) - [`ek.md`](https://thorwhalen.github.io/ek/ek.md): the whole documentation as one Markdown file - `.html.md`: a rendered Markdown twin of every page, advertised from each page’s `` with `` - [`objects.inv`](https://thorwhalen.github.io/ek/objects.inv): the Sphinx inventory: a symbol-to-URL index (`sphobjinv convert plain objects.inv -`)