> built 2026-10-14 03:52 UTC from 15ac259 (main) · ef 0.1.22. Details: build_info.json
# index.html.md
# ef — Embedding Flow
**A facade for boilerplate-less semantic search, corpus indexing, and
RAG-plug-in readiness.**
`ef` makes the modern embedding pipeline — corpus → segment → embed → vector
store → retrieve — usable with *progressive disclosure*: the light case (a list
of strings, search in one or two lines) and the heavy case (huge corpora, many
segmentations and embedders, varied sources and vector DBs) share **one
facade**. `ef` is *not* a RAG framework — it returns ranked context; you bring
your own LLM.
```python
import ef
index = ef.ingest([
"The cat sat on the mat",
"Dogs are loyal companions",
"Neural networks learn from data",
])
for hit in index.search("loyal dogs", limit=2):
print(hit.score, hit.segment["text"])
```
That is the whole light path — no configuration, no install beyond `ef` itself.
`ingest` returns a `SearchableCorpus` ready to `search`.
## Installation
```bash
pip install ef # core: search, indexing, refresh, eval
pip install "ef[openai]" # OpenAI embeddings
pip install "ef[sentence-transformers]" # local sentence-transformers embeddings
pip install "ef[explore]" # the L5 explore layer (UMAP, HDBSCAN)
pip install "ef[imbed]" # imbed-backed components & cluster labelling
```
The core install needs only **numpy** (plus `dol`, `i2`, `vd`). The default
embedder is dependency-free — feature hashing, lexical not semantic; for real
semantic search pass a `sentence-transformers` or provider embedder (see below).
## What `ef` is
`ef` is a **facade, not a framework**. It owns the schemas (`Segment`,
`Embedder`, `Segmenter`, `Corpus`), the indexing core, refresh, and the
RAG-plug-in surface — and it stops there: no agent loops, no prompt templating,
no answer synthesis. “Bring your own LLM, your own agent framework, your own UI.”
It is built on five layers, the same facade covering all of them:
```default
L0 Sources Corpus = MutableMapping[source_id, Source] (dol store: fs/S3/API/RAM)
L1 Parse pluggable text extraction
L2 Segment Segmenter facade (chunkers)
L3 Embed Embedder facade (provider / local adapters)
L4 Index vd.Collection (ef writes; vd owns the index)
L5 Derive project / cluster / label ("explore the corpus")
──────────────────────────────────────────────────────────────
Search search(query) -> ranked SearchHits
RAG plug retrieve(query) -> list[Segment] handed to your LLM/agent
```
## Choosing an embedder
`ingest` and `SourceManager` take an `embedder=` — a string, a callable, a URL,
or an `Embedder`. The `as_embedder` seam normalizes all of them:
```python
from ef import as_embedder, openai_embedder, sentence_transformers_embedder
index = ef.ingest(corpus, embedder=sentence_transformers_embedder("all-MiniLM-L6-v2"))
index = ef.ingest(corpus, embedder=openai_embedder("text-embedding-3-small"))
index = ef.ingest(corpus, embedder=as_embedder("cohere:embed-v4.0")) # also voyage:/gemini:
index = ef.ingest(corpus, embedder=as_embedder(my_callable, model_id="custom@768"))
```
Hosted-API adapters: `openai_embedder` (needs `ef[openai]`) plus
`cohere_embedder`, `voyage_embedder` and `gemini_embedder` — the latter three
speak their providers’ REST endpoints directly, so they need only an API key,
no SDK. Each translates `ef`’s canonical `input_type` (`query` / `document` /
`classification` / `clustering`) to the vendor’s own task name. Local options:
`sentence_transformers_embedder`, `http_embedder` (any TEI-style service), and
the dependency-free `HashingEmbedder` default.
An `Embedder` is just a batch callable `Iterable[str] -> ndarray(n, dim)` with a
little metadata. Composition wrappers — `CachedEmbedder`, `RetryingEmbedder`,
`MultiEmbedder`, `NormalizingEmbedder`, `InstructionEmbedder` — each wrap an inner embedder.
For instruction-tuned models (E5-instruct, BGE, GTE, Qwen3-Embedding), condition the query side on the task: `InstructionEmbedder(inner, query_instruction="Represent this sentence for searching relevant passages: ")` prefixes query texts and leaves documents bare, or pass `query_prompt=` / `prompt_names={"query": "query"}` to `sentence_transformers_embedder`. Either way the conditioned roles show up in `honored_input_types` and the instruction text is baked into `model_id`, so caches never mix vectors. Only trust this on an instruction-tuned model; on `all-MiniLM` it should be a near no-op.
## The heavy case — `SourceManager`
For large or changing corpora, multiple segmentations/embedders, and explicit
control, use `SourceManager`. Configs that share a pipeline step **share its
artifacts for free** — the indexing core is a content-addressed producer graph,
so a second embedder or segmenter re-uses everything upstream of it.
```python
from ef import SourceManager
manager = SourceManager(corpus, store="my_vectors")
manager.ingest(segmenter="recursive", embedder="openai:text-embedding-3-small")
index = manager.searchable()
```
## Keeping an index fresh
As sources change, an index drifts. `SourceManager` diagnoses and repairs it:
```python
report = manager.diagnose() # the four staleness conditions
manager.refresh(mode="incremental") # none | incremental | full | scoped_full
manager = SourceManager(corpus, store="my_vectors", auto_refresh=True) # live
```
## RAG plug-in & evaluation
`ef` hands a corpus to *your* RAG/agent framework and measures retrieval
quality — it does not synthesize answers.
```python
segments = index.retrieve("how do neural networks learn?", limit=5)
context = "\n\n".join(s["text"] for s in segments) # feed context to your LLM
from ef import evaluate_retrieval, evaluate_rag
retrieval = evaluate_retrieval(index.retrieve, qrels, queries) # BEIR-shaped, NDCG@10
rag = evaluate_rag(samples) # deterministic lexical metrics
```
`retrieve()` returns plain `Segment`s (provenance preserved in
`metadata["source"]`); `search()` returns scored `SearchHit`s. `with_reranker`
adds a two-stage reranking pass. `as_ragas_dataset` bridges to Ragas for
LLM-judged metrics.
## Exploring a corpus (layer L5)
The secondary “see the shape of the corpus” surface — `ef`’s visualization
heritage, the backend an `app_ef` corpus map consumes. Three functions, each
taking a corpus *or* a vector matrix:
```python
coords = ef.project(index, dims=2) # PCA -> UMAP, 2-D coordinates
labels = ef.cluster(index, n_clusters=8) # k-means (or method="hdbscan")
titles = ef.label_clusters(segments, labels) # LLM-titled clusters (via imbed)
```
`project` and `cluster` import numpy-only; their default paths (PCA, k-means)
need no extra. `method="umap"`, `method="hdbscan"` and `label_clusters` use the
`ef[explore]` / `ef[imbed]` extras, imported lazily.
## What `ef` is *not*
No agent loops, no tool-calling, no conversation memory, no prompt templating,
no **LLM answer synthesis**, no bundled UI, no global config singleton. The
RAG-plug-in surface is the boundary: `ef` returns `retrieve(query) -> list[Segment]`; the application (or `srag` / `raglab` / LangGraph) takes it from
there.
## Links
- **GitHub**: https://github.com/thorwhalen/ef
- **vd** (vector-store interface): https://github.com/i2mint/vd
- **imbed** (heavy embedding/clustering implementations): https://github.com/thorwhalen/imbed
## Skills
ef ships a **developer** agent skill — install it into your agent host with
[`gh skill`](https://cli.github.com/manual/gh_skill) (don’t have it?
[install gh](https://cli.github.com/)) before working in this repo:
```bash
gh skill install thorwhalen/ef ef-architecture --agent claude-code
```
| Skill | Use it when… |
|-------------------|----------------------------------------------------------------------------------------------------------------------------------|
| `ef-architecture` | (developer) starting any non-trivial change in the ef repo — refactoring, the Embedder/Segmenter facades, the Corpus abstraction |
## License
MIT
This documentation as a single file: ef.md (Markdown, for agents).
# _autosummary/ef.artifact_graph.html.md
# ef.artifact_graph
The `ArtifactGraph` — `ef`’s content-addressed corpus-indexing core.
The corpus-indexing engine is **not a pipeline** — a pipeline is the wrong
abstraction. It is a *declared, content-addressed producer graph* (a DAG). The
one idea this module is built on:
```default
Cascade invalidation and configuration branching are the same operation.
```
A source edit changes a leaf’s content hash; a parameter change changes a leaf’s
content hash. Both produce a new `artifact_id` whose downstream cone needs
(re-)materialization — and the graph does not distinguish them. “Incremental
refresh”, “experiment tracking”, “hot reload”, “cache invalidation” and
“reactivity” are five names for that one operation.
Every artifact is addressed by **what produced it**:
```default
artifact_id = H(op, op_version, inputs, params)
```
so two configs that share an upstream step (same `op`, same `params`, same
`inputs`) share the *same* artifact id — each segment is computed once, each
vector once, no matter how many configs branch through the graph. Bumping an
`op_version` (or any `param`, or any input) yields a *new* id; the old
artifact keeps its old id and its old cached value, still valid for the config
that produced it. Config branching is therefore free.
The module’s surface:
- [`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec) — the declarative recipe for one produced artifact:
*which op*, *which version*, *over which inputs*, *with which params*. It is
fully serializable (string `op` key, not a raw callable — see below), so an
[`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph)’s stores can be persisted.
- [`artifact_id()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.artifact_id) — the content-addressed id of a [`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec).
- [`producer_spec()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.producer_spec) — an ergonomic [`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec) constructor
(positional inputs, keyword params).
- [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph) — the graph itself: four injectable
`MutableMapping`s (a content-addressed cache `store`, the `producers`
registry, a reverse-`edges` index, an `ops` registry) and the four
contract operations [`materialize()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.materialize) (lazy backward),
[`mark_stale()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.mark_stale) / [`delete_cascade()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.delete_cascade)
> (forward) and [`freshness()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.freshness).
\*\*Why `op` is a string, not a callable.\*\* The design wants the dependency
graph to scale to ~10⁷ nodes, which means the `producers` registry must be
*persistable* (a `dol` store — SQLite-backed for the heavy case). A raw
`Callable` cannot be serialized; an op *name* can. So [`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec)
records a string op-key, and the [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph) resolves it to the
actual function through an injected `ops` registry at materialize time. Every
store (`store` / `producers` / `edges`) is a plain `MutableMapping`;
`ef` itself takes no SQLite dependency — you swap the backend in.
This module is the *engine*. Wiring it to a corpus — turning a Phase-3
[`ChangeEvent`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeEvent) into a [`delete_cascade()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.delete_cascade),
exposing the sources-and-configs facade — is the job of later phases
(`source_manager.py` / `refresh.py`); [`delete_cascade()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.delete_cascade)
is exactly the cascade a `"deleted"` / `"modified"` event will trigger.
Example — a two-step graph, a source “change”, a recompute:
```pycon
>>> graph = ArtifactGraph()
>>> _ = graph.register_op('upper', str.upper)
>>> _ = graph.register_op('repeat', lambda s, *, times: s * times)
>>> src = graph.put('src', 'ab') # a leaf source
>>> shout = graph.add(producer_spec('upper', src, op_version='1'))
>>> echo = graph.add(producer_spec('repeat', shout, op_version='1', times=3))
>>> graph.materialize(echo) # lazy backward compute
'ABABAB'
>>> graph.freshness(echo)
'materialized'
>>> _ = graph.mark_stale(src) # the source 'changed'
>>> graph.freshness(echo) # downstream invalidated
'stale'
>>> graph.materialize(echo) # rebuilt from the recipe
'ABABAB'
```
### Module Attributes
| [`ArtifactId`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactId) | A content-addressed artifact identifier — a `sha256` hex digest for a produced artifact ([`artifact_id()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.artifact_id)), or a caller-supplied content hash for a leaf (e.g. [`ef.corpus.content_hash()`](_autosummary/ef.corpus.html.md#ef.corpus.content_hash) of an L0 source). |
|---------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [`OpKey`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.OpKey) | The key an [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph) resolves to a producer callable through its `ops` registry. |
| [`Freshness`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.Freshness) | The state of an artifact, reported by [`ArtifactGraph.freshness()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.freshness): `"materialized"` (a value is cached), `"stale"` (a recipe exists but no cached value — [`materialize()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.materialize) will (re)build it) or `"unknown"` (the graph has neither a value nor a recipe for the id). |
### Functions
| [`artifact_id`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.artifact_id)(spec) | Content-addressed id of `spec` — `sha256` over its four fields. |
|------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------|
| [`producer_spec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.producer_spec)(op, \*inputs, op_version, \*\*params) | Build a [`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec) ergonomically — inputs positional, params keyword. |
### Classes
| [`ArtifactId`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactId) | A content-addressed artifact identifier — a `sha256` hex digest for a produced artifact ([`artifact_id()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.artifact_id)), or a caller-supplied content hash for a leaf (e.g. [`ef.corpus.content_hash()`](_autosummary/ef.corpus.html.md#ef.corpus.content_hash) of an L0 source). |
|----------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [`OpKey`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.OpKey) | The key an [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph) resolves to a producer callable through its `ops` registry. |
| [`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec)(op, op_version[, inputs, params]) | A declarative recipe for one produced artifact — the graph's node type. |
| [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph)(\*[, store, producers, edges, ops]) | A content-addressed producer graph — the corpus-indexing & refresh core. |
### *class* ef.artifact_graph.ArtifactGraph(, store=None, producers=None, edges=None, ops=None)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
A content-addressed producer graph — the corpus-indexing & refresh core.
The graph holds two kinds of node:
- **Leaves** — externally content-addressed inputs (an L0 source, hashed by
[`ef.corpus.content_hash()`](_autosummary/ef.corpus.html.md#ef.corpus.content_hash)). A leaf has a *value* but no recipe; it is
> supplied via [`put()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.put).
- **Produced artifacts** — a node with a [`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec) recipe,
registered via [`add()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.add). Its id *is* [`artifact_id()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.artifact_id) of its recipe.
and four small `MutableMapping` stores, every one injectable so the graph
can be backed by RAM (the default) or by a persistent `dol` store:
- `store` — the content-addressed cache: `artifact_id -> value`.
- `producers` — the recipe registry: `artifact_id -> ProducerSpec`.
- `edges` — the reverse index: `artifact_id -> (dependent ids, ...)`,
so the *downstream* cone of any node is one lookup away.
- `ops` — the op registry: `op_key -> callable`. The one store that is
*not* persisted (callables are not serializable) — repopulate it with
[`register_op()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.register_op) each process.
The four contract operations:
- [`materialize()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.materialize) — lazy *backward*: compute a node by recursively
materializing its inputs, then memoize.
- [`mark_stale()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.mark_stale) — *forward*: drop the cached values of a node’s
produced downstream cone, keeping the recipes (they will rebuild).
- [`delete_cascade()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.delete_cascade) — *forward*: remove a node and its whole downstream
cone entirely (values, recipes and edges).
- [`freshness()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.freshness) — query a node’s state ([`Freshness`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.Freshness)).
plus the reachability queries [`descendants()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.descendants) (the downstream cone) and
[`ancestors()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.ancestors) (lineage — what an artifact was produced from).
* **Parameters:**
* **store** ([`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the content-addressed value cache. Defaults to a fresh `dict`.
* **producers** ([`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the recipe registry. Defaults to a fresh `dict`. If passed
already-populated (a resumed/persisted graph) and `edges` is left
empty, the reverse index is rebuilt from it on construction.
* **edges** ([`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis)]] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the reverse-dependency index. Defaults to a fresh `dict`; if
passed already-populated it is trusted as-is (not rebuilt).
* **ops** ([`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the op registry. Defaults to a fresh `dict`; [`register_op()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.register_op)
adds to it.
```pycon
>>> graph = ArtifactGraph()
>>> _ = graph.register_op('join', lambda a, b, *, sep: f'{a}{sep}{b}')
>>> x = graph.put('x', 'hello')
>>> y = graph.put('y', 'world')
>>> both = graph.add(producer_spec('join', x, y, op_version='1', sep=', '))
>>> graph.materialize(both)
'hello, world'
>>> sorted(graph.descendants('x'))
[...]
>>> graph.delete_cascade('y') # the 'y' source is removed
frozenset(...)
>>> graph.freshness(both) # its consumer is gone too
'unknown'
```
#### add(spec)
Register a producer recipe; return its content-addressed id.
Idempotent: the id *is* the recipe, so adding an equal [`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec)
twice (or adding the shared upstream step of two branching configs)
registers one node and returns the same id — the basis of free config
branching. Inputs need not exist yet; the graph is declarative and
[`materialize()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.materialize) resolves them lazily.
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
```pycon
>>> graph = ArtifactGraph()
>>> a = graph.add(producer_spec('op', 'leaf', op_version='1'))
>>> a == graph.add(producer_spec('op', 'leaf', op_version='1')) # idempotent
True
>>> len(graph.producers)
1
```
#### ancestors(key)
The lineage of `key` — every artifact it was transitively produced from.
Excludes `key` itself. Answers “what produced this?” — provenance /
lineage queries. The leaves of the returned set are the original
sources the artifact ultimately derives from.
* **Return type:**
[`frozenset`](https://docs.python.org/3/builtins/stdtypes.html#frozenset)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]
```pycon
>>> graph = ArtifactGraph()
>>> _ = graph.register_op('id', lambda x: x)
>>> a = graph.add(producer_spec('id', 'src', op_version='1'))
>>> b = graph.add(producer_spec('id', a, op_version='1'))
>>> graph.ancestors(b) == {'src', a}
True
```
#### delete_cascade(key)
Remove `key` and its whole downstream cone — *forward*; return what was removed.
Deletes every artifact in `{key} ∪ descendants(key)` *entirely* —
cached value, recipe and edges — and detaches the cone from any upstream
inputs that survive it. Unlike [`mark_stale()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.mark_stale) this removes the
recipes too: use it when an artifact can never be rebuilt, the canonical
case being a **deleted source** (its segments, vectors and index entries
cascade away with it).
Returns the ids actually removed (those that had a value or a recipe).
* **Return type:**
[`frozenset`](https://docs.python.org/3/builtins/stdtypes.html#frozenset)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]
```pycon
>>> graph = ArtifactGraph()
>>> _ = graph.register_op('id', lambda x: x)
>>> graph.put('s', 'v')
's'
>>> a = graph.add(producer_spec('id', 's', op_version='1'))
>>> graph.delete_cascade('s') == {'s', a}
True
>>> 's' in graph, a in graph
(False, False)
```
#### descendants(key)
The downstream cone of `key` — every artifact transitively produced from it.
Excludes `key` itself. This is the set of artifacts a change to
`key` would invalidate; [`mark_stale()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.mark_stale) and [`delete_cascade()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.delete_cascade)
are built on it.
* **Return type:**
[`frozenset`](https://docs.python.org/3/builtins/stdtypes.html#frozenset)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]
```pycon
>>> graph = ArtifactGraph()
>>> _ = graph.register_op('id', lambda x: x)
>>> a = graph.add(producer_spec('id', 'src', op_version='1'))
>>> b = graph.add(producer_spec('id', a, op_version='1'))
>>> graph.descendants('src') == {a, b}
True
```
#### freshness(key)
Report the state of `key` — [`Freshness`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.Freshness).
`"materialized"` if a value is cached; `"stale"` if a recipe exists
but no value (a [`materialize()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.materialize) will build it); `"unknown"` if the
graph has neither.
Because the graph is content-addressed, a *cached* value is always
correct *for its id* — staleness here means “not yet computed”, never
“computed wrong”. The richer staleness conditions (orphan / missing /
misconfigured, computed against `vd` index metadata) belong to a later
diagnostics phase, not to the graph core.
* **Return type:**
[`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'materialized'`, `'stale'`, `'unknown'`]
```pycon
>>> graph = ArtifactGraph()
>>> _ = graph.register_op('id', lambda x: x)
>>> graph.put('s', 'v')
's'
>>> a = graph.add(producer_spec('id', 's', op_version='1'))
>>> graph.freshness('s'), graph.freshness(a), graph.freshness('?')
('materialized', 'stale', 'unknown')
```
#### mark_stale(key)
Invalidate `key`’s produced downstream cone — *forward*; return that cone.
Drops the **cached values** of every produced artifact in
`{key} ∪ descendants(key)`, *keeping the recipes* — so the next
[`materialize()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.materialize) recomputes them. Leaf nodes (inputs, no recipe) are
never touched: they cannot be recomputed, so dropping their value would
be data loss — use [`delete_cascade()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.delete_cascade) to remove an input.
This is the invalidation half of a refresh: when an op’s implementation
changes, or to force a recompute. (A *source* change is usually a
[`delete_cascade()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.delete_cascade) of the old content hash — the new content yields
a new leaf id.) Returns the produced artifacts that are now stale.
* **Return type:**
[`frozenset`](https://docs.python.org/3/builtins/stdtypes.html#frozenset)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]
```pycon
>>> graph = ArtifactGraph()
>>> _ = graph.register_op('id', lambda x: x)
>>> graph.put('s', 'v')
's'
>>> a = graph.add(producer_spec('id', 's', op_version='1'))
>>> _ = graph.materialize(a)
>>> graph.mark_stale('s') == {a} # the leaf 's' itself is not "stale"
True
>>> graph.freshness(a)
'stale'
```
#### materialize(key)
Compute (or fetch the cached value of) the artifact `key` — lazy *backward*.
If `key` is already cached in `store` its value is returned
directly. Otherwise its recipe’s inputs are materialized recursively
(depth-first), the op is called `op(*input_values, **params)`, and the
result is memoized in `store` before being returned — so a second
[`materialize()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.materialize) is a cache hit, and a shared upstream artifact is
computed exactly once however many configs depend on it.
* **Raises:**
* [**KeyError**](https://docs.python.org/3/builtins/exceptions.html#KeyError) – if `key` is neither cached nor has a producer recipe
(an unresolved leaf — supply it with [`put()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.put)).
* [**LookupError**](https://docs.python.org/3/builtins/exceptions.html#LookupError) – if a recipe’s `op` is not in the `ops` registry.
* [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if the producer graph contains a cycle (a content-
addressed graph cannot — this guards a hand-corrupted registry).
* **Return type:**
[*Any*](https://docs.python.org/3/library/typing.html#typing.Any)
```pycon
>>> graph = ArtifactGraph()
>>> _ = graph.register_op('double', lambda x: x * 2)
>>> graph.put('n', 21)
'n'
>>> out = graph.add(producer_spec('double', 'n', op_version='1'))
>>> graph.materialize(out)
42
```
* **Return type:**
[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)
#### put(key, value)
Store a **leaf** `value` (an external input) under `key`; return `key`.
A leaf has a value but no recipe — typically an L0 source keyed by its
[`ef.corpus.content_hash()`](_autosummary/ef.corpus.html.md#ef.corpus.content_hash). Returns `key` so the call composes:
```default
src = graph.put(content_hash(text), text)
```
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
```pycon
>>> graph = ArtifactGraph()
>>> graph.put('doc-1', 'some text')
'doc-1'
>>> graph.freshness('doc-1')
'materialized'
```
#### register_op(name, fn)
Register the callable `fn` under the op key `name`; return `fn`.
[`materialize()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.materialize) resolves a [`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec)’s string `op`
through this registry. The op is called `fn(*input_values, **params)`:
the materialized inputs positionally, the spec’s `params` as keywords.
* **Return type:**
[`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]
```pycon
>>> graph = ArtifactGraph()
>>> graph.register_op('upper', str.upper)
```
### ef.artifact_graph.ArtifactId
A content-addressed artifact identifier — a `sha256` hex digest for a
produced artifact ([`artifact_id()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.artifact_id)), or a caller-supplied content hash
for a leaf (e.g. [`ef.corpus.content_hash()`](_autosummary/ef.corpus.html.md#ef.corpus.content_hash) of an L0 source).
### ef.artifact_graph.Freshness
The state of an artifact, reported by [`ArtifactGraph.freshness()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.freshness):
`"materialized"` (a value is cached), `"stale"` (a recipe exists but no
cached value — [`materialize()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.materialize) will (re)build it) or
`"unknown"` (the graph has neither a value nor a recipe for the id).
alias of [`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[‘materialized’, ‘stale’, ‘unknown’]
### ef.artifact_graph.OpKey
The key an [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph) resolves to a producer callable through its
`ops` registry. A short, stable operation name, e.g. `"segment"` or
`"embed"`.
### *class* ef.artifact_graph.ProducerSpec(op, op_version, inputs=(), params=)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
A declarative recipe for one produced artifact — the graph’s node type.
A [`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec) says: *this artifact is produced by op* `op` \*at
version\* `op_version`, *applied to the artifacts* `inputs` *(passed
positionally), with the keyword parameters* `params`. It does **not**
hold the producer callable — only the string `op` key — so a spec is
fully serializable ([`as_dict()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec.as_dict) / [`from_dict()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec.from_dict)) and the graph’s
`producers` registry can be persisted.
The recipe is the artifact’s *identity*: [`artifact_id()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.artifact_id) hashes exactly
these four fields. Treat an instance as immutable — it is frozen, and
mutating `params` in place would silently desynchronize it from its id.
#### op
the op key, resolved to a callable via the graph’s `ops` registry.
#### op_version
the op’s version string. It participates in the id, so a
behavior change to an op **must** be paired with a version bump —
reusing a version for changed behavior silently serves stale cached
values (the graph trusts the version; it cannot inspect the code).
#### inputs
the input artifact ids, in the order the op receives them
positionally. Order is significant — it is part of the id.
#### params
the op’s keyword arguments. Key order is *not* significant
(the id is computed over canonical, sorted-key JSON); pass
defaults-filled, fully-named params so that two semantically equal
calls hash equal.
```pycon
>>> spec = producer_spec('embed', 'seg-1', op_version='3', model='small')
>>> spec.op, spec.inputs, spec.op_version
('embed', ('seg-1',), '3')
>>> spec.params == {'model': 'small'}
True
```
#### as_dict()
Return a plain-`dict` form of the spec — JSON-serializable.
The inverse of [`from_dict()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec.from_dict). Use it to persist a `producers`
registry to a `dol` JSON store.
* **Return type:**
[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]
```pycon
>>> producer_spec('embed', 'a', 'b', op_version='1', dim=256).as_dict()
{'op': 'embed', 'op_version': '1', 'inputs': ['a', 'b'], 'params': {'dim': 256}}
```
#### *classmethod* from_dict(d)
Rebuild a [`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec) from its [`as_dict()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec.as_dict) form.
* **Return type:**
[`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec)
```pycon
>>> spec = producer_spec('embed', 'a', 'b', op_version='1', dim=256)
>>> ProducerSpec.from_dict(spec.as_dict()) == spec
True
```
### ef.artifact_graph.artifact_id(spec)
Content-addressed id of `spec` — `sha256` over its four fields.
The id is `sha256` of canonical (sorted-key) JSON of
`{op, op_version, inputs, params}`. Two specs that describe the same
computation hash *identically*; any change to the op, its version, its
inputs (including their order) or its params yields a *different* id. That
is what makes the graph content-addressed: the id *is* the recipe.
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
```pycon
>>> a = producer_spec('embed', 'seg-1', op_version='2', model='small')
>>> artifact_id(a) == artifact_id(producer_spec(
... 'embed', 'seg-1', op_version='2', model='small'))
True
>>> len(artifact_id(a))
64
>>> artifact_id(a) == artifact_id(producer_spec(
... 'embed', 'seg-1', op_version='3', model='small')) # version bump
False
>>> artifact_id(producer_spec('op', 'x', 'y', op_version='1')) == artifact_id(
... producer_spec('op', 'y', 'x', op_version='1')) # input order matters
False
```
### ef.artifact_graph.producer_spec(op, \*inputs, op_version, \*\*params)
Build a [`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec) ergonomically — inputs positional, params keyword.
The natural way to write a recipe: the input artifact ids are positional
(their order is significant — it is the order the op receives them), and
every keyword argument is an op parameter. `op_version` is keyword-only
and required — an op without a stated version cannot be content-addressed
safely.
* **Return type:**
[`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec)
```pycon
>>> spec = producer_spec('segment', 'doc-1', op_version='1', size=512, overlap=64)
>>> spec.op, spec.inputs
('segment', ('doc-1',))
>>> spec.params == {'size': 512, 'overlap': 64}
True
```
# _autosummary/ef.config.html.md
# ef.config
The config layer — declarative, content-addressable pipeline specs.
A *pipeline* in `ef` is not an object graph you wire by hand — it is **data**.
A [`PipelineSpec`](_autosummary/ef.config.html.md#ef.config.PipelineSpec) declares the segment→embed chain; each step is a
[`TransformSpec`](_autosummary/ef.config.html.md#ef.config.TransformSpec) (an op key, an op version, and the keyword params that
feed the content hash). Because a spec is fully serializable and content-hashed
([`config_id()`](_autosummary/ef.config.html.md#ef.config.config_id)), two configs that share a step share that step’s artifacts
in the [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph) for free — config branching
costs nothing beyond the divergent cone (design notes §3.1).
This module is the *bridge* between an ordinary Python call and a content-
addressed graph node. For content addressing to be **correct**, two
semantically-equal calls must produce identical params — so `op(text)` and
`op(text, size=512)` (`512` being the default) must hash the same. The
canonical `i2` idiom does exactly that normalization:
- [`full_kwargs()`](_autosummary/ef.config.html.md#ef.config.full_kwargs) — the raw recipe (design notes §3.8): a call `(args,
kwargs)` → a fully-named, defaults-filled kwargs dict.
- [`step_params()`](_autosummary/ef.config.html.md#ef.config.step_params) — its spec-building application: the canonical *keyword*
params of an op, with the leading positional *input* slots (the upstream
artifact ids) dropped, since those become [`ProducerSpec.inputs`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec.inputs), not params.
Phase 4’s [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph) takes [`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec)s as given; this module (and
[`ef.ops`](_autosummary/ef.ops.html.md#module-ef.ops) / [`ef.source_manager`](_autosummary/ef.source_manager.html.md#module-ef.source_manager)) is where ordinary calls *become*
those specs.
Example — two pipelines, equal segment step, divergent embed step:
```pycon
>>> seg = TransformSpec(op='segment:recursive', op_version='1', params={})
>>> embed_a = TransformSpec(op='embed:m-a', op_version='1', params={})
>>> embed_b = TransformSpec(op='embed:m-b', op_version='1', params={})
>>> pa = PipelineSpec(segment=seg, embed=embed_a)
>>> pb = PipelineSpec(segment=seg, embed=embed_b)
>>> config_id(pa) == config_id(pb) # different embedders → different config
False
>>> config_id(pa) == config_id(PipelineSpec.from_dict(pa.as_dict()))
True
```
### Module Attributes
| [`ConfigId`](_autosummary/ef.config.html.md#ef.config.ConfigId) | A content-addressed pipeline identifier — the `sha256` hex digest of a [`PipelineSpec`](_autosummary/ef.config.html.md#ef.config.PipelineSpec) ([`config_id()`](_autosummary/ef.config.html.md#ef.config.config_id)). |
|-------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
### Functions
| [`full_kwargs`](_autosummary/ef.config.html.md#ef.config.full_kwargs)(op[, args, kwargs]) | Normalize a call into a canonical, fully-named, defaults-filled kwargs dict. |
|------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|
| [`step_params`](_autosummary/ef.config.html.md#ef.config.step_params)(op[, raw_params, n_inputs]) | The canonical *keyword* params of an op step — its inputs dropped. |
| [`config_id`](_autosummary/ef.config.html.md#ef.config.config_id)(spec) | Content-addressed id of a [`PipelineSpec`](_autosummary/ef.config.html.md#ef.config.PipelineSpec) — `sha256` over its steps. |
### Classes
| [`ConfigId`](_autosummary/ef.config.html.md#ef.config.ConfigId) | A content-addressed pipeline identifier — the `sha256` hex digest of a [`PipelineSpec`](_autosummary/ef.config.html.md#ef.config.PipelineSpec) ([`config_id()`](_autosummary/ef.config.html.md#ef.config.config_id)). |
|------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [`TransformSpec`](_autosummary/ef.config.html.md#ef.config.TransformSpec)(op, op_version[, params]) | One step of a pipeline — a serializable recipe for an op application. |
| [`PipelineSpec`](_autosummary/ef.config.html.md#ef.config.PipelineSpec)(segment, embed) | A whole indexing pipeline as data — the segment→embed chain. |
### ef.config.ConfigId
A content-addressed pipeline identifier — the `sha256` hex digest of a
[`PipelineSpec`](_autosummary/ef.config.html.md#ef.config.PipelineSpec) ([`config_id()`](_autosummary/ef.config.html.md#ef.config.config_id)). It is what `ef` writes into
`vd` index metadata as `config_hash` so a later phase can tell which
pipeline produced a vector.
### *class* ef.config.PipelineSpec(segment, embed)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
A whole indexing pipeline as data — the segment→embed chain.
A [`PipelineSpec`](_autosummary/ef.config.html.md#ef.config.PipelineSpec) is the declarative description of one *config*: a
[`TransformSpec`](_autosummary/ef.config.html.md#ef.config.TransformSpec) for the segmenter and one for the embedder. It is
fully serializable ([`as_dict()`](_autosummary/ef.config.html.md#ef.config.PipelineSpec.as_dict) / [`from_dict()`](_autosummary/ef.config.html.md#ef.config.PipelineSpec.from_dict)) and content-hashed
by [`config_id()`](_autosummary/ef.config.html.md#ef.config.config_id). Registering a second [`PipelineSpec`](_autosummary/ef.config.html.md#ef.config.PipelineSpec) that
differs only in its embed step shares the segment artifacts in the
[`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph) — that is config branching, and
the graph gives it for free.
The `segment``→``embed` ordering *is* the input wiring: the embed step
consumes the segment step’s output. (Phase 5 fixes this two-step shape; a
later phase may generalize it to an arbitrary step list.)
```pycon
>>> seg = TransformSpec(op='segment:r', op_version='1', params={})
>>> emb = TransformSpec(op='embed:m@8', op_version='1', params={'input_type': 'document'})
>>> spec = PipelineSpec(segment=seg, embed=emb)
>>> spec.embed.params['input_type']
'document'
>>> PipelineSpec.from_dict(spec.as_dict()) == spec
True
```
#### as_dict()
Return a plain-`dict` (JSON-serializable) form of the pipeline.
* **Return type:**
[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]
```pycon
>>> seg = TransformSpec(op='segment:r', op_version='1', params={})
>>> emb = TransformSpec(op='embed:m', op_version='1', params={})
>>> PipelineSpec(segment=seg, embed=emb).as_dict()['segment']['op']
'segment:r'
```
#### *classmethod* from_dict(d)
Rebuild a [`PipelineSpec`](_autosummary/ef.config.html.md#ef.config.PipelineSpec) from its [`as_dict()`](_autosummary/ef.config.html.md#ef.config.PipelineSpec.as_dict) form.
* **Return type:**
[`PipelineSpec`](_autosummary/ef.config.html.md#ef.config.PipelineSpec)
### *class* ef.config.TransformSpec(op, op_version, params=)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
One step of a pipeline — a serializable recipe for an op application.
A [`TransformSpec`](_autosummary/ef.config.html.md#ef.config.TransformSpec) names *which op* (`op`, a key resolved through
the [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph)’s `ops` registry), \*at which
version\* (`op_version` — it participates in every downstream hash), and
*with which keyword params* (`params`). It does **not** record the step’s
inputs: those are wired structurally by the [`PipelineSpec`](_autosummary/ef.config.html.md#ef.config.PipelineSpec) (the embed
step consumes the segment step’s output). It is the per-step counterpart of
a [`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec).
The op key carries the *component identity* — e.g. `"embed:openai:text-
embedding-3-large@1024"` or `"segment:"` (see
[`ef.ops`](_autosummary/ef.ops.html.md#module-ef.ops)) — so two configs with different components get disjoint
artifact cones automatically.
```pycon
>>> ts = TransformSpec(op='embed:m@8', op_version='2', params={'input_type': 'document'})
>>> ts.op, ts.op_version
('embed:m@8', '2')
>>> TransformSpec.from_dict(ts.as_dict()) == ts
True
```
#### as_dict()
Return a plain-`dict` (JSON-serializable) form of the spec.
* **Return type:**
[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]
```pycon
>>> TransformSpec(op='segment:r', op_version='1', params={}).as_dict()
{'op': 'segment:r', 'op_version': '1', 'params': {}}
```
#### *classmethod* from_dict(d)
Rebuild a [`TransformSpec`](_autosummary/ef.config.html.md#ef.config.TransformSpec) from its [`as_dict()`](_autosummary/ef.config.html.md#ef.config.TransformSpec.as_dict) form.
* **Return type:**
[`TransformSpec`](_autosummary/ef.config.html.md#ef.config.TransformSpec)
### ef.config.config_id(spec)
Content-addressed id of a [`PipelineSpec`](_autosummary/ef.config.html.md#ef.config.PipelineSpec) — `sha256` over its steps.
The id is `sha256` of the canonical (sorted-key) JSON of
[`PipelineSpec.as_dict()`](_autosummary/ef.config.html.md#ef.config.PipelineSpec.as_dict). Two pipelines that describe the same
computation hash identically; any change to a step’s op, version or params
yields a different id. `ef` writes this digest into `vd` index metadata
as `config_hash`.
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
```pycon
>>> seg = TransformSpec(op='segment:r', op_version='1', params={})
>>> emb = TransformSpec(op='embed:m@8', op_version='1', params={})
>>> cid = config_id(PipelineSpec(segment=seg, embed=emb))
>>> len(cid)
64
>>> cid == config_id(PipelineSpec(segment=seg, embed=emb))
True
```
### ef.config.full_kwargs(op, args=(), kwargs=None)
Normalize a call into a canonical, fully-named, defaults-filled kwargs dict.
The `i2.Sig` recipe of design notes §3.8. A call expressed as positional
`args` + keyword `kwargs` is mapped onto `op`’s signature: every
parameter is named, and any parameter the call left implicit is filled with
its default. The result is therefore *stable* — `op(x)` and
`op(x, size=512)` (`512` the default) yield the **same** dict, which is
what makes a content hash built from it correct.
* **Parameters:**
* **op** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – the callable whose signature defines the parameter names/defaults.
* **args** ([`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any), [`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis)]) – the call’s positional arguments — for an [`ef.ops`](_autosummary/ef.ops.html.md#module-ef.ops) op these
are the upstream artifact ids; [`step_params()`](_autosummary/ef.config.html.md#ef.config.step_params) drops them again.
* **kwargs** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the call’s keyword arguments.
* **Return type:**
[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]
```pycon
>>> def embed(segments, *, input_type='document', dim=None): ...
>>> full_kwargs(embed, ('seg-1',), {'input_type': 'query'})
{'segments': 'seg-1', 'input_type': 'query', 'dim': None}
>>> full_kwargs(embed, ('seg-1',)) # defaults filled in
{'segments': 'seg-1', 'input_type': 'document', 'dim': None}
```
### ef.config.step_params(op, raw_params=None, , n_inputs=1)
The canonical *keyword* params of an op step — its inputs dropped.
An [`ef.ops`](_autosummary/ef.ops.html.md#module-ef.ops) op is called `op(*input_values, **params)`: the first
`n_inputs` positional parameters are upstream artifacts (they belong in
[`ProducerSpec.inputs`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec.inputs), not in
the params), and every remaining parameter is a hashable step param. This
runs [`full_kwargs()`](_autosummary/ef.config.html.md#ef.config.full_kwargs) with placeholder inputs, then drops those input
slots — leaving exactly the params, defaults filled, ready for a
[`TransformSpec`](_autosummary/ef.config.html.md#ef.config.TransformSpec).
* **Parameters:**
* **op** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – the op callable whose signature is the param contract.
* **raw_params** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the keyword params the caller actually supplied; anything
omitted is defaulted.
* **n_inputs** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – how many leading positional parameters are upstream inputs.
* **Return type:**
[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]
```pycon
>>> def embed(segments, *, input_type='document'): ...
>>> step_params(embed, {'input_type': 'query'})
{'input_type': 'query'}
>>> step_params(embed) # default filled in
{'input_type': 'document'}
>>> def segment(source): ...
>>> step_params(segment) # no params at all
{}
```
# _autosummary/ef.corpus.html.md
# ef.corpus
The `Corpus` abstraction — `ef`’s source layer (L0) — and change detection.
A *corpus* is the entry point of the pipeline: the set of source documents to
be segmented, embedded and indexed. In `ef` a corpus is deliberately *not* a
new class — it is just a mapping:
```default
Corpus = MutableMapping[source_id, Source]
```
Any `dol`-style mapping is a corpus: a plain `dict` (RAM), a filesystem
folder, an S3 bucket, an API-backed store. The backing store is swappable
because the rest of `ef` only ever sees a `MutableMapping`. This module
provides:
- [`Source`](_autosummary/ef.corpus.html.md#ef.corpus.Source) / [`Corpus`](_autosummary/ef.corpus.html.md#ef.corpus.Corpus) — the type aliases that name the contract.
- [`as_corpus()`](_autosummary/ef.corpus.html.md#ef.corpus.as_corpus) — the dependency-injection seam: `None` / a mapping / a
directory path / an iterable of strings → a corpus. Mirrors
[`as_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.as_embedder) and
[`as_segmenter()`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.as_segmenter).
- [`content_hash()`](_autosummary/ef.corpus.html.md#ef.corpus.content_hash) — the content hash of a source, the basis of change
detection. Hashes *normalized* content (via [`ef.hashing`](_autosummary/ef.hashing.html.md#module-ef.hashing)); for a
structured source it hashes the text only, so metadata-only edits never
trigger a spurious re-embed.
- [`ChangeDetectingCorpus`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus) — a corpus *wrapper* that content-hashes
values and reports what changed, both through-the-wrapper (via an
`on_change` callback) and out-of-band (via [`scan()`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus.scan)).
[`ChangeDetectingCorpus`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus) is the *“something changed”* half of incremental
refresh; the downstream cascade is the `ArtifactGraph` (a later phase). The
`on_change` callback is the seam those two halves meet at.
Example — detect changes as they are written:
```pycon
>>> events = []
>>> corpus = ChangeDetectingCorpus(on_change=events.append)
>>> corpus['doc1'] = 'hello world'
>>> corpus['doc1'] = 'hello world' # identical content — idempotent, no event
>>> corpus['doc1'] = 'hello there' # a real change
>>> del corpus['doc1']
>>> [(e.source_id, e.kind) for e in events]
[('doc1', 'added'), ('doc1', 'modified'), ('doc1', 'deleted')]
```
### Module Attributes
| [`Source`](_autosummary/ef.corpus.html.md#ef.corpus.Source) | A source document. |
|-------------------------------------------------------------|-----------------------------------------------------------------------------------------------------|
| [`Corpus`](_autosummary/ef.corpus.html.md#ef.corpus.Corpus) | a mapping from `source_id` to [`Source`](_autosummary/ef.corpus.html.md#ef.corpus.Source). |
| [`ChangeKind`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeKind) | The kinds of change [`ChangeDetectingCorpus`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus) reports. |
### Functions
| [`content_hash`](_autosummary/ef.corpus.html.md#ef.corpus.content_hash)(source, \*[, content_keys]) | Content hash of a source document — `sha256` of its normalized content. |
|---------------------------------------------------------------------------------------------|---------------------------------------------------------------------------|
| [`as_corpus`](_autosummary/ef.corpus.html.md#ef.corpus.as_corpus)([x]) | Normalize `x` into a corpus (`MutableMapping[source_id, Source]`). |
### Classes
| [`ChangeEvent`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeEvent)(source_id, kind[, old_hash, ...]) | A single detected change to a source in a corpus. |
|----------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------|
| [`CorpusDiff`](_autosummary/ef.corpus.html.md#ef.corpus.CorpusDiff)([added, modified, deleted, unchanged]) | The result of comparing a corpus's content against known content hashes. |
| [`ChangeDetectingCorpus`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus)([corpus, on_change, ...]) | A corpus wrapper that content-hashes values and reports what changed. |
### *class* ef.corpus.ChangeDetectingCorpus(corpus=None, , on_change=None, hashes=None, content_keys=None, fingerprint=None)
Bases: [`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)
A corpus wrapper that content-hashes values and reports what changed.
Wraps any inner corpus (a `MutableMapping[source_id, Source]`) and adds
change detection *without* altering its mapping behavior — reads, writes and
deletes pass straight through to the inner store. It is to a corpus what
[`CachedEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.CachedEmbedder) is to an embedder: a
transparent, composable layer.
Changes are noticed two ways:
- **Through the wrapper.** A `corpus[id] = source` or `del corpus[id]`
that actually changes content fires the injected `on_change` callback
with a [`ChangeEvent`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeEvent). Writing *identical* content is a no-op (no
event) — the basis of idempotent ingestion.
- **Out of band.** When the backing store is edited directly (a file
changed on disk, an S3 object replaced), [`scan()`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus.scan) re-reads every
source, compares hashes, fires events for what differs and returns a
[`CorpusDiff`](_autosummary/ef.corpus.html.md#ef.corpus.CorpusDiff). [`diff()`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus.diff) runs the same comparison with no side
> effects.
The `on_change` callback is the seam the downstream `ArtifactGraph`
plugs into: a `"modified"` / `"deleted"` event is exactly the trigger
for a cascade invalidation.
On construction a *baseline* is established: the current content hashes of
the inner corpus are recorded silently (no events). Pass an already-populated
`hashes` registry to instead resume from a persisted baseline — then call
[`scan()`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus.scan) to reconcile any drift.
* **Parameters:**
* **corpus** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the inner corpus to wrap. Coerced via [`as_corpus()`](_autosummary/ef.corpus.html.md#ef.corpus.as_corpus), so a
directory path / `dict` / iterable of sources / `None` all work.
* **on_change** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`ChangeEvent`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeEvent)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – called with each [`ChangeEvent`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeEvent). Defaults to a no-op.
It should be cheap and not raise; an exception propagates to the
caller *after* the corpus and registry are already consistent.
* **hashes** ([`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the hash registry — a `MutableMapping[source_id, str]` of the
last-known content hash per source. Injectable so it can be
persisted (e.g. a `dol` store); defaults to an in-RAM `dict`.
If passed already-populated it is trusted as the baseline.
* **content_keys** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – forwarded to [`content_hash()`](_autosummary/ef.corpus.html.md#ef.corpus.content_hash) — metadata keys that
participate in the hash.
* **fingerprint** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`Hashable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Hashable)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – an optional cheap `source_id -> Hashable` callback (an
mtime / etag prefilter). When given, [`scan()`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus.scan) / [`diff()`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus.diff)
skip re-hashing a source whose fingerprint is unchanged. The
content hash always remains the source of truth — the fingerprint
is only a speed optimization, never trusted to *declare* a change.
```pycon
>>> backing = {'a': 'one', 'b': 'two'}
>>> corpus = ChangeDetectingCorpus(backing) # baseline adopts 'a' and 'b'
>>> backing['a'] = 'ONE' # edited behind the wrapper's back
>>> backing['c'] = 'three'
>>> del backing['b']
>>> d = corpus.scan()
>>> (d.added, d.modified, d.deleted)
(('c',), ('a',), ('b',))
```
#### diff()
Compare current corpus content against the registry — **no side effects**.
Detects out-of-band edits (changes made to the backing store directly)
without firing events or updating the registry. [`scan()`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus.scan) is the
effectful counterpart.
* **Return type:**
[`CorpusDiff`](_autosummary/ef.corpus.html.md#ef.corpus.CorpusDiff)
#### scan()
Reconcile the registry with the corpus, fire events, return the diff.
Use this to pick up **out-of-band** edits — changes made to the backing
store directly rather than through this wrapper. Every detected change
fires a [`ChangeEvent`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeEvent) and the registry (and fingerprints) are
advanced to the new state, so a second [`scan()`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus.scan) with nothing
further changed is a no-op. [`diff()`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus.diff) is the side-effect-free version.
* **Return type:**
[`CorpusDiff`](_autosummary/ef.corpus.html.md#ef.corpus.CorpusDiff)
### *class* ef.corpus.ChangeEvent(source_id, kind, old_hash=None, new_hash=None)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
A single detected change to a source in a corpus.
Emitted by [`ChangeDetectingCorpus`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus) to its `on_change` callback.
`old_hash` is `None` for an `"added"` event, `new_hash` is `None`
for a `"deleted"` one.
#### source_id
the key of the source that changed.
#### kind
`"added"`, `"modified"` or `"deleted"`.
#### old_hash
the source’s previous content hash, if any.
#### new_hash
the source’s new content hash, if any.
### ef.corpus.ChangeKind
The kinds of change [`ChangeDetectingCorpus`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus) reports.
alias of [`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[‘added’, ‘modified’, ‘deleted’]
### ef.corpus.Corpus
a mapping from `source_id` to [`Source`](_autosummary/ef.corpus.html.md#ef.corpus.Source). `ef` never
requires a dedicated class — any `dol`-style `MutableMapping` (RAM,
filesystem, S3, API) *is* a corpus. [`as_corpus()`](_autosummary/ef.corpus.html.md#ef.corpus.as_corpus) builds one.
* **Type:**
A corpus
alias of [`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]]
### *class* ef.corpus.CorpusDiff(added=(), modified=(), deleted=(), unchanged=())
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
The result of comparing a corpus’s content against known content hashes.
The four fields partition the corpus’s source ids; each is a sorted tuple,
so a [`CorpusDiff`](_autosummary/ef.corpus.html.md#ef.corpus.CorpusDiff) is deterministic and JSON-friendly. Truthiness
reports whether *anything* changed.
```pycon
>>> d = CorpusDiff(added=('b',), modified=('a',))
>>> bool(d), d.changed
(True, ('b', 'a'))
>>> bool(CorpusDiff(unchanged=('a', 'b')))
False
```
#### *property* changed *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), ...]*
The source ids that are new or modified — the set to (re-)materialize.
### ef.corpus.Source *= str | collections.abc.Mapping[str, typing.Any]*
A source document. Either plain `text`, or a `Mapping` carrying a
`text` key alongside metadata — the same shape a
[`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter) accepts as input.
### ef.corpus.as_corpus(x=None, , \*\*kwargs)
Normalize `x` into a corpus (`MutableMapping[source_id, Source]`).
The single dependency-injection seam through which every `ef` entry point
accepts a user-supplied corpus — the mirror of
[`as_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.as_embedder) and
[`as_segmenter()`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.as_segmenter). Accepts, in order:
1. `None` — a fresh empty in-RAM `dict`;
2. a `Mapping` — returned **unchanged** (a `dict`, a `dol` store,
another corpus: it already *is* a corpus);
3. a directory path (`str` / `os.PathLike` naming an existing
directory) — a `dol` filesystem store of the text files under it
(`dol` is imported lazily; only this branch needs it);
4. an iterable of sources (strings or mappings) — an in-RAM `dict` keyed
by each source’s [`content_hash()`](_autosummary/ef.corpus.html.md#ef.corpus.content_hash) (content-addressed, so duplicate
sources collapse to one entry).
Extra `**kwargs` are forwarded to the `dol` filesystem store (the
directory branch only) and ignored otherwise.
* **Raises:**
[**TypeError**](https://docs.python.org/3/builtins/exceptions.html#TypeError) – if `x` is a non-directory string, or none of the above.
```pycon
>>> as_corpus()
{}
>>> as_corpus({'a': 'x'}) # a mapping passes straight through
{'a': 'x'}
>>> sorted(as_corpus(['alpha', 'beta']).values())
['alpha', 'beta']
>>> len(as_corpus(['dup', 'dup'])) # content-addressed: duplicates collapse
1
```
### ef.corpus.content_hash(source, , content_keys=None)
Content hash of a source document — `sha256` of its normalized content.
This is the basis of change detection: re-reading an unchanged source
yields an identical hash, so a refresh that finds the same hash does
nothing. Hashing *normalized* content (NFC, no BOM, `\n` line endings —
see [`ef.hashing`](_autosummary/ef.hashing.html.md#module-ef.hashing)) means cosmetic encoding differences never look like
a change.
The treatment of a `Mapping` source is deliberate:
- a plain `str` source — the hash covers the text;
- a `Mapping` with a `text` key — the hash covers `source['text']`
**only**. A change to title, tags, ACL or any other metadata does *not*
change the hash, so it never triggers a spurious re-segment/re-embed
(embedding is the pipeline’s most expensive step). Name the metadata keys
that *do* feed downstream in `content_keys` and changes to those keys
will be detected;
- a `Mapping` *without* a `text` key — treated as a structured document
and hashed via canonical (sorted-key) JSON of the whole mapping;
- `bytes` — hashed raw (no text normalization is possible pre-parse).
* **Parameters:**
* **source** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – a `str`, `bytes`, or a `Mapping` (ideally with a `text`
key).
* **content_keys** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – metadata keys whose values join the text in the hash.
Each key is looked up at the top level of `source` or inside a
`source['metadata']` sub-mapping. Use this *only* for metadata
that genuinely affects the L1–L4 pipeline (e.g. a title prepended
to the text before embedding); key order does not matter.
* **Raises:**
[**TypeError**](https://docs.python.org/3/builtins/exceptions.html#TypeError) – if `source` is not a `str`, `bytes` or `Mapping`.
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
```pycon
>>> content_hash('hello') == content_hash('hello')
True
>>> content_hash('hello') == content_hash('hello world')
False
>>> content_hash('a\r\nb') == content_hash('a\nb') # line endings normalized
True
>>> a = {'text': 'body', 'metadata': {'tags': ['x']}}
>>> b = {'text': 'body', 'metadata': {'tags': ['y']}}
>>> content_hash(a) == content_hash(b) # metadata ignored by default
True
>>> content_hash(a, content_keys=['tags']) == content_hash(b, content_keys=['tags'])
False
>>> content_hash({'text': 'body'}) == content_hash('body') # wrapper is transparent
True
```
# _autosummary/ef.diagnostics.html.md
# ef.diagnostics
Staleness diagnostics — has an indexed corpus drifted from its source?
Phase 5 wired `corpus → segment → embed → vd` and recorded provenance on
every `vd` document: `source_id`, `source_hash` and `config_hash` ride
in the document metadata (see `ef.source_manager._RESERVED_METADATA_KEYS`).
This module reads that provenance back and answers a single question — \*is the
index still a faithful picture of the corpus?\* — by computing the \*\*four
staleness conditions\*\* (`ef_use_cases.md` §C):
The detection is a set comparison, exactly as the design intends — the
`vd` collection is the single source of truth for *what is indexed*, the
corpus for *what should be indexed*, and the current
[`ConfigId`](_autosummary/ef.config.html.md#ef.config.ConfigId) for *how*:
- `orphan` / `missing` partition the symmetric difference of the two id
sets;
- `stale` compares each indexed source’s recorded `source_hash` against a
freshly recomputed [`content_hash()`](_autosummary/ef.corpus.html.md#ef.corpus.content_hash);
- `misconfigured` compares each document’s recorded `config_hash` against
the config being diagnosed.
This module is **read-only** — it never mutates the corpus, the collection or
the graph. Acting on a [`StalenessReport`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.StalenessReport) is [`ef.refresh`](_autosummary/ef.refresh.html.md#module-ef.refresh)’s job.
Example — diagnose a config whose corpus has since drifted:
```pycon
>>> from ef import SourceManager
>>> sm = SourceManager({'a': 'alpha', 'b': 'beta'}, embedder='hashing')
>>> _ = sm.materialize()
>>> sm.corpus['b'] = 'beta changed' # an edit, behind the index's back
>>> sm.corpus['c'] = 'gamma' # a brand-new source
>>> del sm.corpus['a'] # and a deletion
>>> report = sm.diagnose()
>>> report.orphan, report.missing, report.stale
(('a',), ('c',), ('b',))
>>> bool(report) # True — the index has drifted
True
```
### Functions
| [`indexed_state`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.indexed_state)(collection) | Group a `vd` collection's documents by source — its recorded state. |
|---------------------------------------------------------------------------------------|-------------------------------------------------------------------------|
| [`corpus_state`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.corpus_state)(corpus) | The current content hash of every source in `corpus`. |
| [`diagnose`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.diagnose)(corpus, collection, config) | Compute the four staleness conditions of `collection` against `corpus`. |
### Classes
| [`IndexedSource`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.IndexedSource)(source_id, doc_ids, ...) | The provenance a `vd` collection records for one source document. |
|--------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------|
| [`StalenessReport`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.StalenessReport)(config[, orphan, missing, ...]) | The four staleness conditions of one config — what [`diagnose()`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.diagnose) returns. |
### *class* ef.diagnostics.IndexedSource(source_id, doc_ids, source_hashes, config_hashes)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
The provenance a `vd` collection records for one source document.
A source is segmented into several documents; they all carry the same
`source_id` and *should* carry one `source_hash` / `config_hash` each.
[`indexed_state()`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.indexed_state) groups a collection’s documents into one of these per
source — collecting the hashes as **sets** so a partially re-indexed source
(documents disagreeing on a hash) is detectable rather than papered over.
#### source_id
the corpus key of the source these documents were cut from.
#### doc_ids
the ids of every `vd` document belonging to the source.
#### source_hashes
the distinct `source_hash` values its documents record
— a single-element set for a cleanly indexed source.
#### config_hashes
likewise the distinct `config_hash` values.
### *class* ef.diagnostics.StalenessReport(config, orphan=(), missing=(), stale=(), misconfigured=(), fresh=())
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
The four staleness conditions of one config — what [`diagnose()`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.diagnose) returns.
Each field is a sorted tuple of `source_id`s, so a report is
deterministic and JSON-friendly. `orphan` and `missing` are disjoint
from the rest (a source absent from one side cannot also be compared); a
source can be **both** `stale` and `misconfigured` (its content changed
*and* the config did) — the conditions are reported independently.
`fresh` is the complement: indexed, current and built by this config.
Truthiness reports whether *anything is wrong* — `bool(report)` is
`False` only for a perfectly in-sync index.
```pycon
>>> r = StalenessReport(config='c0', stale=('a',), fresh=('b',))
>>> bool(r), r.needs_reindexing
(True, ('a',))
>>> bool(StalenessReport(config='c0', fresh=('a', 'b')))
False
```
#### *property* needs_reindexing *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), ...]*
Sources whose vectors must be recomputed — `stale` ∪ `misconfigured`.
### ef.diagnostics.corpus_state(corpus)
The current content hash of every source in `corpus`.
The *what should be indexed* half of [`diagnose()`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.diagnose): a fresh
[`content_hash()`](_autosummary/ef.corpus.html.md#ef.corpus.content_hash) per source — recomputed, not read from any
registry, so it reflects out-of-band edits to the backing store. It matches
the `source_hash` [`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager) writes into
`vd` (both are `content_hash` of the source with no `content_keys`).
* **Return type:**
[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]
```pycon
>>> sorted(corpus_state({'a': 'one', 'b': 'two'}))
['a', 'b']
>>> corpus_state({'a': 'x'})['a'] == corpus_state({'a': 'x'})['a']
True
```
### ef.diagnostics.diagnose(corpus, collection, config)
Compute the four staleness conditions of `collection` against `corpus`.
The pure core of `ef`’s diagnostics — no side effects. It reads what the
`vd` `collection` records ([`indexed_state()`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.indexed_state)), what the `corpus`
currently holds ([`corpus_state()`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.corpus_state)), and partitions every source id into
exactly the buckets it belongs in:
- **orphan** — in the collection, not in the corpus;
- **missing** — in the corpus, not in the collection;
- **stale** — in both, but the recorded `source_hash` ≠ the current one;
- **misconfigured** — in both, but a recorded `config_hash` ≠ `config`;
- **fresh** — in both, content current and config matching.
* **Parameters:**
* **corpus** ([`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]] | [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – the source corpus.
* **collection** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the `vd` collection backing the config being diagnosed.
* **config** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the [`ConfigId`](_autosummary/ef.config.html.md#ef.config.ConfigId) the collection *should* hold
— documents recording any other `config_hash` are misconfigured.
* **Return type:**
[`StalenessReport`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.StalenessReport)
* **Returns:**
a [`StalenessReport`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.StalenessReport).
### ef.diagnostics.indexed_state(collection)
Group a `vd` collection’s documents by source — its recorded state.
Iterates the collection (a `MutableMapping` of `vd` documents) and
folds every document `ef` wrote into an [`IndexedSource`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.IndexedSource) keyed by
its `source_id`. Documents with no `source_id` metadata (not written by
`ef`) are skipped. This is the *what is indexed* half of [`diagnose()`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.diagnose).
* **Parameters:**
**collection** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the `vd` collection backing one config.
* **Return type:**
[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`IndexedSource`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.IndexedSource)]
* **Returns:**
a `{source_id: IndexedSource}` mapping.
# _autosummary/ef.embedder_adapters.html.md
# ef.embedder_adapters
Concrete [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) adapters and the DI seam.
Adapters are **lazily-imported factory functions** — heavy dependencies
(`openai`, `sentence-transformers`) are imported inside the factory, never
at module load, and live in optional extras. Importing this module costs only
numpy + the stdlib.
- [`openai_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.openai_embedder) — OpenAI embeddings (sync + the async Batch API).
Needs `ef[openai]`.
- [`sentence_transformers_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.sentence_transformers_embedder) — local PyTorch models. Needs
`ef[sentence-transformers]`.
- [`http_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.http_embedder) — any remote HTTP embedding service (TEI, infinity,
…). Stdlib-only.
- [`cohere_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.cohere_embedder), [`voyage_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.voyage_embedder), [`gemini_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.gemini_embedder) —
the Cohere / Voyage / Gemini hosted embedding APIs, spoken over their REST
endpoints directly. Stdlib-only (no SDK dependency); each needs only an API
key. The whole point of these adapters is translating the canonical
[`InputType`](_autosummary/ef.embedders.html.md#ef.embedders.InputType) vocabulary to the vendor’s own task name.
- [`as_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.as_embedder) — the single dependency-injection seam: turns a string
(model name / `openai:` / `cohere:` / `voyage:` / `gemini:` prefix /
URL), a bare callable, or an existing embedder into an
[`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder).
The **canonical task vocabulary** ([`InputType`](_autosummary/ef.embedders.html.md#ef.embedders.InputType)) is
translated to each vendor’s own name *at the adapter boundary* — that
translation is the whole point of having adapters.
Example — the DI seam in action (no network / no heavy deps needed):
```pycon
>>> import numpy as np
>>> from ef.embedders import Embedder
>>> e = as_embedder(lambda ts: np.ones((len(ts), 5)), model_id='ones@5')
>>> isinstance(e, Embedder)
True
>>> e(['x', 'y']).shape
(2, 5)
>>> as_embedder(e) is e # an Embedder passes straight through
True
```
### Functions
| [`openai_embedder`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.openai_embedder)([model, dim, api_key, ...]) | Build an OpenAI-embeddings [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder). |
|----------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------|
| [`sentence_transformers_embedder`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.sentence_transformers_embedder)([model_name, ...]) | Build a local sentence-transformers [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder). |
| [`http_embedder`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.http_embedder)(url, \*[, model_id, dim, ...]) | Build an [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) over a remote HTTP service. |
| [`cohere_embedder`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.cohere_embedder)([model, dim, api_key, ...]) | Build a Cohere-embeddings [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder). |
| [`voyage_embedder`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.voyage_embedder)([model, dim, api_key, ...]) | Build a Voyage AI [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder). |
| [`gemini_embedder`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.gemini_embedder)([model, dim, api_key, ...]) | Build a Google Gemini [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder). |
| [`as_embedder`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.as_embedder)(x, \*\*kwargs) | Normalize `x` into an [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) — the DI seam. |
### ef.embedder_adapters.as_embedder(x, \*\*kwargs)
Normalize `x` into an [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) — the DI seam.
The single place every `ef` entry point coerces a user-supplied embedder
argument. Accepts, in order:
1. an existing [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) — returned unchanged;
2. a URL string (`http://` / `https://`) — [`http_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.http_embedder);
3. a provider-prefixed string — `"openai:"`, `"cohere:"`,
`"voyage:"` or `"gemini:"` — the matching hosted-API
adapter;
4. the bare string `"hashing"` — the dependency-free
[`HashingEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.HashingEmbedder) (`ef`’s zero-install default);
5. an `"st:"` string, or any other bare model name —
[`sentence_transformers_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.sentence_transformers_embedder) (the local neural default);
6. a bare callable — wrapped in
[`FunctionEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.FunctionEmbedder) (the `imbed`-embedder bridge).
Extra `**kwargs` are forwarded to the chosen factory.
* **Raises:**
[**TypeError**](https://docs.python.org/3/builtins/exceptions.html#TypeError) – if `x` is none of the above.
* **Return type:**
[*Embedder*](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)
```pycon
>>> import numpy as np
>>> e = as_embedder(lambda ts: np.zeros((len(ts), 2)), model_id='zero@2')
>>> type(e).__name__
'FunctionEmbedder'
```
* **Return type:**
[`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)
### ef.embedder_adapters.cohere_embedder(model='embed-v4.0', , dim=None, api_key=None, batch_size=96, timeout=60.0)
Build a Cohere-embeddings [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder).
Talks to Cohere’s `/v2/embed` REST endpoint directly — no `cohere` SDK,
no extra dependency. Cohere’s v3+ models *require* a task hint; this adapter
supplies one for every canonical [`InputType`](_autosummary/ef.embedders.html.md#ef.embedders.InputType) and
defaults to the document role when none is given.
* **Parameters:**
* **model** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – A Cohere embedding model name.
* **dim** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Matryoshka-truncated width (`embed-v4.0` only). Omit for the
model’s native width. The dim is part of the embedder’s identity —
set it once here, not per call.
* **api_key** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Cohere API key. Falls back to `CO_API_KEY` / `COHERE_API_KEY`.
* **batch_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Texts per request (capped at Cohere’s 96 limit).
* **timeout** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Per-request timeout in seconds.
* **Return type:**
[`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)
* **Returns:**
An [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) over the Cohere API.
### ef.embedder_adapters.gemini_embedder(model='gemini-embedding-001', , dim=None, api_key=None, batch_size=100, timeout=60.0)
Build a Google Gemini [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder).
Talks to the Generative Language `batchEmbedContents` REST endpoint
directly — no `google-genai` SDK, no extra dependency.
* **Parameters:**
* **model** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – A Gemini embedding model name (a leading `models/` is
tolerated and stripped).
* **dim** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Matryoshka-truncated width (`gemini-embedding-001` only). Omit
for the model’s native width. Note that Gemini, unlike OpenAI, does
*not* re-normalize MRL-truncated output — a truncated embedder
therefore reports `normalized=False`.
* **api_key** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Gemini API key. Falls back to `GEMINI_API_KEY` /
`GOOGLE_API_KEY`.
* **batch_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Texts per request (capped at Gemini’s 100 limit).
* **timeout** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Per-request timeout in seconds.
* **Return type:**
[`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)
* **Returns:**
An [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) over the Gemini API.
### ef.embedder_adapters.http_embedder(url, \*, model_id=None, dim=None, normalized=False, batch_size=64, headers=None, timeout=30.0, payload_builder=, response_parser=)
Build an [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) over a remote HTTP service.
Defaults target the Text-Embeddings-Inference / infinity shape (a POST of
`{"inputs": [...]}` returning a JSON list of vectors). For a different
service, override `payload_builder` and/or `response_parser`.
* **Parameters:**
* **url** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The embedding endpoint.
* **model_id** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Vector-identity string (defaults to `"http:"`).
* **dim** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Output width; inferred from the first response if omitted.
* **normalized** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether the service returns unit vectors.
* **batch_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Texts per request (used with length-sorted batching).
* **headers** ([`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Extra HTTP headers (e.g. an auth token).
* **timeout** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Per-request timeout in seconds.
* **payload_builder** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Sequence`](https://docs.python.org/3/library/typing.html#typing.Sequence)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]], [`bytes`](https://docs.python.org/3/builtins/stdtypes.html#bytes)]) – `texts -> request body bytes`.
* **response_parser** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – `decoded JSON -> array-like (n, dim)`.
* **Return type:**
[`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)
Uses only the stdlib — no extra dependency.
### ef.embedder_adapters.openai_embedder(model='text-embedding-3-small', , dim=None, api_key=None, client=None, batch_size=2048, poll_interval=30.0)
Build an OpenAI-embeddings [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder).
* **Parameters:**
* **model** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – An OpenAI embedding model name.
* **dim** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Matryoshka-truncated output width (`text-embedding-3-*` only).
Omit for the model’s native dimension. The dim is part of the
embedder’s identity — set it once here, not per call.
* **api_key** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – OpenAI API key (else the SDK’s usual env-var resolution).
* **client** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – A pre-built `openai.OpenAI` client (overrides `api_key`).
* **batch_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Texts per request (capped at OpenAI’s 2048 limit).
* **poll_interval** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Seconds between polls when blocking on a Batch-API job.
* **Return type:**
[`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)
* **Returns:**
An embedder whose `embed_batch` uses the async OpenAI Batch API.
Requires the `openai` package (`pip install 'ef[openai]'`).
### ef.embedder_adapters.sentence_transformers_embedder(model_name='all-MiniLM-L6-v2', , device=None, normalize=False, batch_size=128, model_path=None, query_prompt=None, document_prompt=None, prompt_names=None, \*\*st_kwargs)
Build a local sentence-transformers [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder).
Instruction-tuned models (bge, gte, e5-instruct, Qwen3-Embedding…) are
conditioned per `input_type`: pass a literal `query_prompt` (and, rarely,
`document_prompt`), or map input types to prompts the model defines itself
with `prompt_names` (e.g. `{"query": "query"}` for Qwen3-Embedding). Those
roles are then advertised in `honored_input_types`, and the prompt text is
baked into `model_id`. Only trust prompts on genuinely instruction-tuned
models; on `all-MiniLM` they are expected to be a near no-op. Prefer these
over wrapping the adapter in [`InstructionEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.InstructionEmbedder):
sentence-transformers knows the prompt’s length, so models that pool without
the prompt tokens (INSTRUCTOR, GRIT) stay correct. Needs
sentence-transformers >= 2.3.
* **Parameters:**
* **model_name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – A sentence-transformers model name (also used in
`model_id`).
* **device** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Torch device (`"cuda"` / `"cpu"` / `"mps"`); `None`
auto-selects.
* **normalize** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to L2-normalize outputs. sentence-transformers does
**not** normalize by default — set this truthfully.
* **batch_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Encoding batch size (smaller on CPU).
* **model_path** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – A local filesystem path to load from instead of the hub —
for air-gapped use. Loading still records `model_name` in
`model_id`.
* **query_prompt** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Prompt prepended to `input_type="query"` texts (passed
as `encode(prompt=...)`).
* **document_prompt** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Prompt for `input_type="document"` texts (usually
none: the asymmetric pattern leaves documents bare). `""` means an
explicit empty prompt, which stops a model’s `default_prompt_name`
from being applied to documents.
* **prompt_names** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Mapping`](https://docs.python.org/3/library/typing.html#typing.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]]) – `{input_type: name}` of prompts defined in the model’s
own `prompts` config (passed as `encode(prompt_name=...)`).
An explicit `prompt=` / `prompt_name=` in a call overrides both.
* **\*\*st_kwargs** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Forwarded to the `SentenceTransformer` constructor.
* **Return type:**
[`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)
Requires `sentence-transformers` (`pip install 'ef[sentence-transformers]'`).
### ef.embedder_adapters.voyage_embedder(model='voyage-3.5', , dim=None, api_key=None, batch_size=1000, timeout=60.0)
Build a Voyage AI [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder).
Talks to Voyage’s `/v1/embeddings` REST endpoint directly — no
`voyageai` SDK, no extra dependency. Voyage embeddings are L2-normalized.
Voyage distinguishes only `query` from `document`; a `classification`
or `clustering` hint is simply not sent.
* **Parameters:**
* **model** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – A Voyage embedding model name.
* **dim** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Matryoshka-truncated width (the `voyage-3.5` family and
`voyage-code-3` only). Omit for the model’s native width.
* **api_key** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Voyage API key. Falls back to `VOYAGE_API_KEY`.
* **batch_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Texts per request (capped at Voyage’s 1000 limit).
* **timeout** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Per-request timeout in seconds.
* **Return type:**
[`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)
* **Returns:**
An [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) over the Voyage API.
# _autosummary/ef.embedder_wrappers.html.md
# ef.embedder_wrappers
Composition wrappers for [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder).
Each wrapper *takes an inner embedder and is itself an embedder* — they compose
freely (`CachedEmbedder(RetryingEmbedder(NormalizingEmbedder(inner)), store)`).
This keeps the core protocol tiny: caching, retries, routing and normalization
are opt-in layers, not protocol obligations.
- [`CachedEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.CachedEmbedder) — memoize vectors in any `MutableMapping` store.
- [`RetryingEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.RetryingEmbedder) — exponential backoff + jitter, retryable-aware.
- [`MultiEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.MultiEmbedder) — route texts to different embedders by a predicate.
- [`NormalizingEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.NormalizingEmbedder) — L2-normalize an inner embedder’s output.
- [`InstructionEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.InstructionEmbedder) — prepend a per-`input_type` instruction
(task-conditioned, asymmetric query/document embedding).
They delegate their metadata (`model_id` / `dim` / `normalized` /
`honored_input_types`) to the inner embedder, except where the wrapper changes
the vector ([`NormalizingEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.NormalizingEmbedder) reports `normalized=True`;
[`InstructionEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.InstructionEmbedder) bakes its instructions into `model_id` and adds the
conditioned roles to `honored_input_types`).
The inner argument must already be an [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder); use
[`ef.embedder_adapters.as_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.as_embedder) to coerce a string / callable first.
Example — cache an embedder so repeats are free:
```pycon
>>> import numpy as np
>>> from ef.embedders import FunctionEmbedder
>>> calls = []
>>> def toy(texts):
... calls.extend(texts)
... return np.ones((len(texts), 3))
>>> cached = CachedEmbedder(FunctionEmbedder(toy, model_id='toy@3'), store={})
>>> _ = cached(['a', 'b'])
>>> _ = cached(['b', 'c']) # 'b' served from cache
>>> calls # 'b' embedded once
['a', 'b', 'c']
```
### Classes
| [`CachedEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.CachedEmbedder)(inner, store, \*[, cache_queries]) | Memoize embeddings in a `MutableMapping` store. |
|------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------|
| [`RetryPolicy`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.RetryPolicy)([max_attempts, base_delay, ...]) | Exponential-backoff retry policy for [`RetryingEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.RetryingEmbedder). |
| [`RetryingEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.RetryingEmbedder)(inner[, policy, sleep]) | Retry an inner embedder's calls on transient failures. |
| [`MultiEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.MultiEmbedder)(routes, predicate, \*[, default]) | Route each text to one of several embedders by a predicate. |
| [`NormalizingEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.NormalizingEmbedder)(inner) | L2-normalize an inner embedder's output. |
| [`InstructionEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.InstructionEmbedder)(inner, \*[, ...]) | Prepend a per-`input_type` instruction before delegating to `inner`. |
### *class* ef.embedder_wrappers.CachedEmbedder(inner, store, , cache_queries=False)
Bases: [`BaseEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.BaseEmbedder)
Memoize embeddings in a `MutableMapping` store.
On each call, texts already in the store are served from it; only the misses
are forwarded to the inner embedder (in a single batch) and then written
back. The store can be any `dol`-style mapping — an in-RAM `dict`, a
directory of `.npy` files, a key-value DB.
Document embeddings are cached unconditionally; *query* embeddings are
skipped by default (they are rarely repeated) — pass `cache_queries=True`
to cache them too.
* **Parameters:**
* **inner** ([`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)) – The embedder to memoize.
* **store** ([`MutableMapping`](https://docs.python.org/3/library/typing.html#typing.MutableMapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), `ndarray`]) – A `MutableMapping[str, ndarray]` keyed by `cache_key()`.
* **cache_queries** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to also cache `input_type="query"` calls.
#### *property* honored_input_types *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['query', 'document', 'classification', 'clustering'], ...]*
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple.
If iterable is specified the tuple is initialized from iterable’s items.
If the argument is a tuple, the return value is the same object.
#### *property* normalized *: [bool](https://docs.python.org/3/builtins/functions.html#bool)*
bool(x) -> bool
Returns True when the argument x is true, False otherwise.
The builtins True and False are the only two instances of the class bool.
The class bool is a subclass of the class int, and cannot be subclassed.
### *class* ef.embedder_wrappers.InstructionEmbedder(inner, , query_instruction=None, document_instruction=None, classification_instruction=None, clustering_instruction=None)
Bases: [`BaseEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.BaseEmbedder)
Prepend a per-`input_type` instruction before delegating to `inner`.
Instruction-tuned embedding models (E5-instruct, BGE, GTE, Qwen3-Embedding…)
are trained to read a task description in front of the text, typically on the
*query* side only, with documents embedded bare. That asymmetry is where the
gain comes from: the query says *what kind of relevance* is wanted (topic vs.
keyword), which a plain model cannot be told. This wrapper adds that to any
embedder: a text embedded with `input_type="query"` becomes
`query_instruction + text`, and so on per role. Roles without an instruction
(and calls with `input_type=None`) pass through unchanged.
**Only trust the prefix on a genuinely instruction-tuned model.** On a plain
model (e.g. `all-MiniLM-L6-v2`) the prefix is expected to be a near no-op or
noise: use that as the control arm of an experiment, not as an improvement.
The instruction strings are used verbatim, so include the model’s expected
separator (e.g. `"Instruct: ...\nQuery: "` for Qwen3/E5-instruct,
`"Represent this sentence for searching relevant passages: "` for BGE).
Metadata:
- `model_id` bakes in a digest of the instructions (they change the vector),
so [`CachedEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.CachedEmbedder) and content-addressed artifacts never mix vectors
made with different instructions.
- `honored_input_types` is the inner embedder’s plus every instructed role.
- `input_type` is still forwarded to `inner`; a backend that also acts on
it natively (Cohere, Voyage, Gemini, or a sentence-transformers adapter
built with `query_prompt=`) would then be conditioned twice, so use this
wrapper for backends that ignore the hint. For sentence-transformers models
prefer the adapter’s own `query_prompt=`.
* **Parameters:**
* **inner** ([`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)) – The embedder to condition.
* **query_instruction** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Prefix for `input_type="query"`.
* **document_instruction** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Prefix for `input_type="document"` (usually none).
* **classification_instruction** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Prefix for `input_type="classification"`.
* **clustering_instruction** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Prefix for `input_type="clustering"`.
```pycon
>>> import numpy as np
>>> from ef.embedders import FunctionEmbedder
>>> seen = []
>>> def toy(texts):
... seen.extend(texts)
... return np.ones((len(texts), 2))
>>> base = FunctionEmbedder(toy, model_id='toy@2')
>>> emb = InstructionEmbedder(base, query_instruction='Find packages about: ')
>>> _ = emb(['graphs'], input_type='query')
>>> _ = emb(['a DAG library'], input_type='document')
>>> seen
['Find packages about: graphs', 'a DAG library']
>>> emb.honored_input_types
('query',)
>>> emb.model_id.startswith('toy@2+instruct:')
True
```
#### embed_batch(texts, , input_type=None, \*\*backend)
Prefix, then use the inner embedder’s own (possibly async) batch API.
#### *property* honored_input_types *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['query', 'document', 'classification', 'clustering'], ...]*
The inner embedder’s honored roles plus every instructed role.
#### *property* model_id *: [str](https://docs.python.org/3/builtins/stdtypes.html#str)*
The inner `model_id` plus a digest of the instructions.
#### *property* normalized *: [bool](https://docs.python.org/3/builtins/functions.html#bool)*
bool(x) -> bool
Returns True when the argument x is true, False otherwise.
The builtins True and False are the only two instances of the class bool.
The class bool is a subclass of the class int, and cannot be subclassed.
### *class* ef.embedder_wrappers.MultiEmbedder(routes, predicate, , default=None)
Bases: [`BaseEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.BaseEmbedder)
Route each text to one of several embedders by a predicate.
Useful for domain-specific models (e.g. a code embedder for code, a prose
embedder for prose). All routes must share the same `dim` — the output is
a single `(n, dim)` array, with each row produced by its text’s route.
* **Parameters:**
* **routes** ([`Mapping`](https://docs.python.org/3/library/typing.html#typing.Mapping)[[`Hashable`](https://docs.python.org/3/library/typing.html#typing.Hashable), [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)]) – `{route_key: Embedder}`.
* **predicate** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`Hashable`](https://docs.python.org/3/library/typing.html#typing.Hashable)]) – `text -> route_key` — picks a route per text.
* **default** ([`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Embedder for texts whose `route_key` is not in `routes`
(otherwise an unknown key raises `EmbedderError`).
```pycon
>>> import numpy as np
>>> from ef.embedders import FunctionEmbedder
>>> short = FunctionEmbedder(lambda ts: np.zeros((len(ts), 2)), model_id='s@2')
>>> long = FunctionEmbedder(lambda ts: np.ones((len(ts), 2)), model_id='l@2')
>>> m = MultiEmbedder({'s': short, 'l': long},
... predicate=lambda t: 's' if len(t) < 4 else 'l')
>>> m(['hi', 'hello']).tolist()
[[0.0, 0.0], [1.0, 1.0]]
```
### *class* ef.embedder_wrappers.NormalizingEmbedder(inner)
Bases: [`BaseEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.BaseEmbedder)
L2-normalize an inner embedder’s output.
Use this for MRL-truncated vectors (which lose unit norm when sliced to a
smaller dimension) or any backend that does not normalize by construction.
Reports `normalized=True`; `cache_key()` keys on `normalized`, so a
normalized and an un-normalized view of the same model never collide in a
shared cache.
* **Parameters:**
**inner** ([`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)) – The embedder whose output to renormalize.
#### *property* honored_input_types *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['query', 'document', 'classification', 'clustering'], ...]*
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple.
If iterable is specified the tuple is initialized from iterable’s items.
If the argument is a tuple, the return value is the same object.
### *class* ef.embedder_wrappers.RetryPolicy(max_attempts=4, base_delay=1.0, max_delay=30.0, jitter=0.3, retry_on=)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
Exponential-backoff retry policy for [`RetryingEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.RetryingEmbedder).
#### max_attempts
Total tries (the first call counts as attempt 1).
#### base_delay
Seconds before the first retry; doubles each attempt.
#### max_delay
Upper bound on any single backoff sleep.
#### jitter
Random fraction (`0..1`) of the delay added on top, to
de-correlate concurrent clients.
#### retry_on
Predicate deciding whether an exception is worth retrying.
* **Return type:**
[*bool*](https://docs.python.org/3/builtins/functions.html#bool)
#### retry_on()
Heuristic: retry transient failures (429, 5xx, connection/timeout).
Per-request 400-class errors are *not* retried — the same request would fail
identically. Recognizes the `status_code`/`status` attribute carried by
most HTTP-client SDK exceptions (incl. the OpenAI SDK).
* **Return type:**
[`bool`](https://docs.python.org/3/builtins/functions.html#bool)
### *class* ef.embedder_wrappers.RetryingEmbedder(inner, policy=None, \*, sleep=)
Bases: [`BaseEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.BaseEmbedder)
Retry an inner embedder’s calls on transient failures.
* **Parameters:**
* **inner** ([`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)) – The embedder to guard.
* **policy** ([`RetryPolicy`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.RetryPolicy) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – A [`RetryPolicy`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.RetryPolicy) (sensible defaults if omitted).
* **sleep** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`float`](https://docs.python.org/3/builtins/functions.html#float)], [`None`](https://docs.python.org/3/builtins/constants.html#None)]) – The sleep function — injectable so tests run instantly.
#### *property* honored_input_types *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['query', 'document', 'classification', 'clustering'], ...]*
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple.
If iterable is specified the tuple is initialized from iterable’s items.
If the argument is a tuple, the return value is the same object.
#### *property* normalized *: [bool](https://docs.python.org/3/builtins/functions.html#bool)*
bool(x) -> bool
Returns True when the argument x is true, False otherwise.
The builtins True and False are the only two instances of the class bool.
The class bool is a subclass of the class int, and cannot be subclassed.
# _autosummary/ef.embedders.html.md
# ef.embedders
The `Embedder` facade — `ef`’s embed layer (L3).
An *embedder* is, fundamentally, a batch callable:
```default
Iterable[str] -> ndarray of shape (n, dim)
```
Everything else — batching, async/batch-APIs, normalization, task hints,
caching, retries — is *layered scaffolding* over that one contract.
This module defines the small, structural core of the facade:
- [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) — the `@runtime_checkable` protocol (a callable plus four
introspectable metadata attributes). It is a protocol, **not** an ABC: any
object with the right shape *is* an embedder.
- [`BatchHandle`](_autosummary/ef.embedders.html.md#ef.embedders.BatchHandle) / [`ready_handle()`](_autosummary/ef.embedders.html.md#ef.embedders.ready_handle) — a uniform poll/result shape that
collapses synchronous embedding and provider Batch APIs into one thing.
- [`BaseEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.BaseEmbedder) — an implementation convenience base (supplies a default
synchronous [`embed_batch()`](_autosummary/ef.embedders.html.md#ef.embedders.BaseEmbedder.embed_batch) and a `repr`). Subclassing it
is optional; satisfying [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) structurally is what matters.
- [`FunctionEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.FunctionEmbedder) — wraps a bare `texts -> vectors` callable into a
full [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) (the bridge for e.g. `imbed`’s embedder functions).
- [`HashingEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.HashingEmbedder) — a dependency-free embedder (the feature-hashing
trick, numpy only); `ef`’s zero-install default when no embedder is given.
- [`cache_key()`](_autosummary/ef.embedders.html.md#ef.embedders.cache_key) — a deterministic key pinning everything that changes a
vector, for [`CachedEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.CachedEmbedder).
- [`embed_length_sorted()`](_autosummary/ef.embedders.html.md#ef.embedders.embed_length_sorted) — length-sorted batching with order
reconstruction; a 5–10× throughput win that `ef` owns so users never have
to think about padding or sort order.
Composition wrappers live in [`ef.embedder_wrappers`](_autosummary/ef.embedder_wrappers.html.md#module-ef.embedder_wrappers); concrete provider /
local adapters and the [`as_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.as_embedder) dependency
-injection seam live in [`ef.embedder_adapters`](_autosummary/ef.embedder_adapters.html.md#module-ef.embedder_adapters).
Example — wrap a plain function and embed two strings:
```pycon
>>> import numpy as np
>>> def toy(texts): # a bare callable: list[str] -> array(n, 3)
... return np.array([[len(t), t.count(' '), 1.0] for t in texts])
>>> embedder = FunctionEmbedder(toy, model_id='toy@3')
>>> vectors = embedder(['hello', 'a b c'])
>>> vectors.shape
(2, 3)
>>> embedder.dim # inferred on first call
3
>>> isinstance(embedder, Embedder)
True
```
### Module Attributes
| [`InputType`](_autosummary/ef.embedders.html.md#ef.embedders.InputType) | Canonical embedding-task vocabulary. |
|--------------------------------------------------------------|-----------------------------------------------------------------------------------------|
| [`BatchStatus`](_autosummary/ef.embedders.html.md#ef.embedders.BatchStatus) | The lifecycle states of a [`BatchHandle`](_autosummary/ef.embedders.html.md#ef.embedders.BatchHandle). |
### Functions
| [`ready_handle`](_autosummary/ef.embedders.html.md#ef.embedders.ready_handle)(array) | Wrap an already-computed array in a finished [`BatchHandle`](_autosummary/ef.embedders.html.md#ef.embedders.BatchHandle). |
|------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------|
| [`cache_key`](_autosummary/ef.embedders.html.md#ef.embedders.cache_key)(embedder, text[, input_type]) | Deterministic cache key for one `(embedder, text, input_type)` triple. |
| [`embed_length_sorted`](_autosummary/ef.embedders.html.md#ef.embedders.embed_length_sorted)(texts, encode, \*[, ...]) | Encode `texts` in length-sorted batches; return vectors in caller order. |
### Classes
| [`BatchHandle`](_autosummary/ef.embedders.html.md#ef.embedders.BatchHandle)(poll, result[, cancel]) | A poll/result handle over an in-flight (or already-finished) embed job. |
|---------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------|
| [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)(\*args, \*\*kwargs) | Structural type for an embedder: a batch callable plus metadata. |
| [`BaseEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.BaseEmbedder)() | Implementation base for embedders (optional — [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) is structural). |
| [`FunctionEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.FunctionEmbedder)(func, \*[, model_id, dim, ...]) | Promote a bare `texts -> vectors` callable to a full [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder). |
| [`HashingEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.HashingEmbedder)(\*[, dim, ngram_range, ...]) | A dependency-free embedder — the feature-hashing trick, numpy only. |
### Exceptions
| [`EmbedderError`](_autosummary/ef.embedders.html.md#ef.embedders.EmbedderError) | Raised for embedder-specific failures (shape mismatch, batch job failure). |
|------------------------------------------------------------------|------------------------------------------------------------------------------|
### *class* ef.embedders.BaseEmbedder
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
Implementation base for embedders (optional — [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) is structural).
Subclasses set `model_id` / `dim` / `normalized` /
`honored_input_types` and implement `__call__`; they then satisfy
[`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) structurally. In return they get:
- a default *synchronous* [`embed_batch()`](_autosummary/ef.embedders.html.md#ef.embedders.BaseEmbedder.embed_batch) (`ready_handle(self(...))`) —
override it only for a genuinely async provider Batch API;
- a readable `repr`.
#### embed_batch(texts, , input_type=None, \*\*backend)
Default: embed synchronously and return a finished handle.
* **Return type:**
[`BatchHandle`](_autosummary/ef.embedders.html.md#ef.embedders.BatchHandle)
### *class* ef.embedders.BatchHandle(poll, result, cancel=>)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
A poll/result handle over an in-flight (or already-finished) embed job.
Batch-and-poll is just a special case of async. A synchronous backend
returns a *ready* handle (via [`ready_handle()`](_autosummary/ef.embedders.html.md#ef.embedders.ready_handle)); a provider Batch API
returns a genuinely pending one. Either way the caller sees one shape.
#### poll
`() -> BatchStatus` — non-blocking status check.
#### result
`() -> np.ndarray` — blocks until done, returns `(n, dim)`.
#### cancel
`() -> None` — best-effort cancellation.
### ef.embedders.BatchStatus
The lifecycle states of a [`BatchHandle`](_autosummary/ef.embedders.html.md#ef.embedders.BatchHandle).
alias of [`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[‘pending’, ‘done’, ‘failed’]
### *class* ef.embedders.Embedder(\*args, \*\*kwargs)
Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol)
Structural type for an embedder: a batch callable plus metadata.
An embedder maps `Iterable[str]` to an `ndarray` of shape `(n, dim)`.
The four metadata attributes are the single source of truth \*on the object
itself\* — no global registry, no config singleton.
#### model_id
Identity that bakes in everything affecting the vector —
`"openai:text-embedding-3-large@1024"` (`provider:model@dim`).
Works directly as a cache namespace.
#### dim
Output dimensionality. May be `None` until first call for
lazily-probed embedders (e.g. a bare callable of unknown width).
#### normalized
`True` iff `||v|| == 1` by construction.
#### honored_input_types
The [`InputType`](_autosummary/ef.embedders.html.md#ef.embedders.InputType) values this embedder acts on
(empty if it ignores the task hint).
#### embed_batch(texts, , input_type=None, \*\*backend)
Submit an embed job, returning a [`BatchHandle`](_autosummary/ef.embedders.html.md#ef.embedders.BatchHandle).
* **Return type:**
[`BatchHandle`](_autosummary/ef.embedders.html.md#ef.embedders.BatchHandle)
### *exception* ef.embedders.EmbedderError
Bases: [`Exception`](https://docs.python.org/3/builtins/exceptions.html#Exception)
Raised for embedder-specific failures (shape mismatch, batch job failure).
### *class* ef.embedders.FunctionEmbedder(func, , model_id=None, dim=None, normalized=False, honored_input_types=(), pass_backend=True)
Bases: [`BaseEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.BaseEmbedder)
Promote a bare `texts -> vectors` callable to a full [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder).
This is the bridge for embedder *functions* that carry no metadata — most
importantly `imbed`’s registered embedders. The wrapped callable is asked
only for `func(texts)` (it knows nothing of `input_type`); the wrapper
advertises `honored_input_types=()` accordingly.
* **Parameters:**
* **func** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – A callable mapping a list of strings to an array-like of shape
`(n, dim)` (a numpy array or a nested sequence — both accepted).
* **model_id** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Vector-identity string. Defaults to `"function:"`.
* **dim** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Output width. If `None` it is inferred (and frozen) on the first
call.
* **normalized** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether `func` already returns unit vectors.
* **honored_input_types** ([`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'query'`, `'document'`, `'classification'`, `'clustering'`]]) – Task hints `func` honors (usually none).
* **pass_backend** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – If `True` (default), forward any `**backend` kwargs to
`func`; set `False` for callables with a strict `(texts,)`
signature.
```pycon
>>> import numpy as np
>>> e = FunctionEmbedder(lambda ts: np.ones((len(ts), 4)), model_id='ones@4')
>>> e(['a', 'b', 'c']).shape
(3, 4)
```
### *class* ef.embedders.HashingEmbedder(, dim=512, ngram_range=(1, 2), sublinear_tf=True)
Bases: [`BaseEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.BaseEmbedder)
A dependency-free embedder — the feature-hashing trick, numpy only.
[`HashingEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.HashingEmbedder) is `ef`’s **zero-install default**: the embedder
[`ingest()`](_autosummary/ef.source_manager.html.md#ef.source_manager.ingest) resolves to when none is given, so the
headline `ingest([...]).search(query)` works on a bare `pip install ef`
— no torch, no model download, no network.
It is a *lexical* embedder, not a neural one. Each text is tokenized into
lowercased word n-grams; every token is hashed into one of `dim` buckets
with a sign (the hashing trick — no vocabulary, no fitting, fixed memory);
the signed, sublinear-tf-weighted bucket counts are L2-normalized. The
cosine similarity of two vectors then ranks by shared vocabulary — enough
for demos, doctests and offline tests, and a sane fallback in production.
For genuine semantic quality install a real embedder (`ef[sentence-transformers]`,
`ef[openai]`) and pass it explicitly.
Every parameter that changes a vector is baked into `model_id`, so a
[`HashingEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.HashingEmbedder)’s artifacts stay correctly content-addressed in the
[`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph).
* **Parameters:**
* **dim** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Output width. Wider → fewer hash collisions, larger vectors.
* **ngram_range** ([`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`int`](https://docs.python.org/3/builtins/functions.html#int), [`int`](https://docs.python.org/3/builtins/functions.html#int)]) – Inclusive `(min, max)` word-n-gram sizes. The default
`(1, 2)` mixes single words with adjacent pairs — a little phrase
sensitivity at no real cost.
* **sublinear_tf** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Weight a token by `1 + log(count)` rather than its raw
`count`, damping a word repeated many times in one text.
```pycon
>>> e = HashingEmbedder()
>>> e.model_id
'hashing:v1@512'
>>> import numpy as np
>>> v = e(['the quick brown fox'])
>>> v.shape
(1, 512)
>>> bool(np.isclose(np.linalg.norm(v[0]), 1.0)) # rows are L2-normalized
True
>>> a, b = e(['hello world']), e(['hello world'])
>>> round(float(a[0] @ b[0]), 5) # identical text → cosine similarity 1.0
1.0
>>> isinstance(e, Embedder)
True
```
### ef.embedders.InputType
Canonical embedding-task vocabulary. Every provider names the same
distinction differently (`input_type` / `task_type` / `task` /
`prompt_name`); `ef` defines *one* canonical set and each adapter
translates it at its boundary. This is the facade’s single most valuable
normalization. Backends that ignore the hint advertise an empty
`honored_input_types`.
alias of [`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[‘query’, ‘document’, ‘classification’, ‘clustering’]
### ef.embedders.cache_key(embedder, text, input_type=None)
Deterministic cache key for one `(embedder, text, input_type)` triple.
The key pins everything that changes the resulting vector. `model_id` is
the embedder-identity SSOT — it already bakes in provider, model and `dim`
— so `dim` is *not* keyed separately (keying it would also break
[`FunctionEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.FunctionEmbedder)’s lazy `dim` inference). `normalized` is keyed
because a [`NormalizingEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.NormalizingEmbedder) shares its
inner’s `model_id` yet yields different vectors.
Returns `"/"` — a namespaced key safe for any
`MutableMapping` store.
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
```pycon
>>> class _E:
... model_id, normalized = 'm@8', False
>>> k1 = cache_key(_E(), 'hello', 'document')
>>> k2 = cache_key(_E(), 'hello', 'query')
>>> k1 == k2 # input_type participates
False
>>> k1.startswith('m@8/')
True
```
### ef.embedders.embed_length_sorted(texts, encode, , batch_size=128)
Encode `texts` in length-sorted batches; return vectors in caller order.
Transformer self-attention is O(n²) in sequence length and a batch is padded
to its longest member, so grouping similar-length texts together avoids
wasted compute — a 5–10× speedup on skewed corpora. `ef` owns this so
callers never sort, pad, or un-sort themselves.
* **Parameters:**
* **texts** ([`Sequence`](https://docs.python.org/3/library/typing.html#typing.Sequence)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – The strings to embed.
* **encode** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Sequence`](https://docs.python.org/3/library/typing.html#typing.Sequence)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – Maps a list of strings to an array-like of shape `(k, dim)`.
* **batch_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Texts per call to `encode`.
* **Return type:**
`ndarray`
* **Returns:**
A `float32` array of shape `(len(texts), dim)` in the *original*
order of `texts`.
```pycon
>>> import numpy as np
>>> def enc(batch): # toy encoder: vector = [length]
... return np.array([[len(t)] for t in batch], dtype=np.float32)
>>> out = embed_length_sorted(['aaa', 'a', 'aaaaa', 'aa'], enc, batch_size=2)
>>> out.ravel().tolist() # order preserved despite internal sorting
[3.0, 1.0, 5.0, 2.0]
```
### ef.embedders.ready_handle(array)
Wrap an already-computed array in a finished [`BatchHandle`](_autosummary/ef.embedders.html.md#ef.embedders.BatchHandle).
* **Return type:**
[`BatchHandle`](_autosummary/ef.embedders.html.md#ef.embedders.BatchHandle)
```pycon
>>> import numpy as np
>>> h = ready_handle(np.zeros((2, 3)))
>>> h.poll()
'done'
>>> h.result().shape
(2, 3)
```
# _autosummary/ef.evaluation.html.md
# ef.evaluation
Evaluation hookpoints — measuring retrieval and RAG quality.
Phase 7 makes an indexed `ef` corpus **measurable**. It stays a *facade*: it
computes the standard, deterministic metrics and provides the standard data
shapes, but it never synthesizes an answer and never requires an LLM (the §6
boundary). LLM-judged metrics are reached through a thin Ragas bridge.
Two journeys (`ef_use_cases.md` §F7–F8), two evaluation conventions
(`ef_design_notes.md` §6):
- **Retrieval evaluation** — [`evaluate_retrieval()`](_autosummary/ef.evaluation.html.md#ef.evaluation.evaluate_retrieval), BEIR/MTEB-shaped. The
inputs are the BEIR triple — a corpus, a set of `queries` and `qrels`
(graded relevance judgements) — and the primary metric is **NDCG@10**. The
metric primitives ([`ndcg_at_k()`](_autosummary/ef.evaluation.html.md#ef.evaluation.ndcg_at_k), [`recall_at_k()`](_autosummary/ef.evaluation.html.md#ef.evaluation.recall_at_k), …) are pure
functions, usable on their own. [`read_beir()`](_autosummary/ef.evaluation.html.md#ef.evaluation.read_beir) / [`write_beir()`](_autosummary/ef.evaluation.html.md#ef.evaluation.write_beir) move
the triple to and from the on-disk `corpus.jsonl` / `queries.jsonl` /
`qrels.tsv` format.
- **RAG evaluation** — [`evaluate_rag()`](_autosummary/ef.evaluation.html.md#ef.evaluation.evaluate_rag), Ragas `SingleTurnSample`-shaped.
A sample is `(user_input, response, retrieved_contexts, reference)`;
[`RagSample`](_autosummary/ef.evaluation.html.md#ef.evaluation.RagSample) names that shape. [`evaluate_rag()`](_autosummary/ef.evaluation.html.md#ef.evaluation.evaluate_rag) computes only
> **deterministic, reference-based lexical metrics** ([`exact_match()`](_autosummary/ef.evaluation.html.md#ef.evaluation.exact_match),
[`token_f1()`](_autosummary/ef.evaluation.html.md#ef.evaluation.token_f1), [`context_recall()`](_autosummary/ef.evaluation.html.md#ef.evaluation.context_recall), [`context_precision()`](_autosummary/ef.evaluation.html.md#ef.evaluation.context_precision)) — no LLM.
For the LLM-judged metrics (faithfulness, answer relevancy, …) hand the
samples to [`as_ragas_dataset()`](_autosummary/ef.evaluation.html.md#ef.evaluation.as_ragas_dataset) and run Ragas with your own LLM.
Example — score a retriever against a tiny BEIR-shaped triple:
```pycon
>>> qrels = {'q1': {'d1': 1.0, 'd2': 1.0}, 'q2': {'d3': 1.0}}
>>> queries = {'q1': 'first query', 'q2': 'second query'}
>>> rankings = {'first query': ['d1', 'd9', 'd2'], 'second query': ['d9', 'd3']}
>>> def retriever(query, *, limit=10):
... return rankings[query][:limit]
>>> report = evaluate_retrieval(retriever, qrels, queries, k_values=(1, 3))
>>> report.metrics['recall@3']
1.0
>>> round(report.primary_at(1), 3) # NDCG@1, averaged over the two queries
0.5
```
### Module Attributes
| [`Qrels`](_autosummary/ef.evaluation.html.md#ef.evaluation.Qrels) | Graded relevance judgements — `{query_id: {doc_id: relevance}}`. |
|--------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [`RETRIEVAL_METRICS`](_autosummary/ef.evaluation.html.md#ef.evaluation.RETRIEVAL_METRICS) | The retrieval metrics [`evaluate_retrieval()`](_autosummary/ef.evaluation.html.md#ef.evaluation.evaluate_retrieval) can compute, each keyed by the short name used in the `metrics=` argument and in report keys. |
### Functions
| [`dcg_at_k`](_autosummary/ef.evaluation.html.md#ef.evaluation.dcg_at_k)(ranked, relevant, k) | Discounted cumulative gain of `ranked` at cutoff `k`. |
|----------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------|
| [`ndcg_at_k`](_autosummary/ef.evaluation.html.md#ef.evaluation.ndcg_at_k)(ranked, relevant, k) | Normalized DCG at cutoff `k` — `ef`'s **primary** retrieval metric. |
| [`recall_at_k`](_autosummary/ef.evaluation.html.md#ef.evaluation.recall_at_k)(ranked, relevant, k) | Fraction of the relevant docs that appear in the top `k` results. |
| [`precision_at_k`](_autosummary/ef.evaluation.html.md#ef.evaluation.precision_at_k)(ranked, relevant, k) | Fraction of the top `k` results that are relevant. |
| [`reciprocal_rank`](_autosummary/ef.evaluation.html.md#ef.evaluation.reciprocal_rank)(ranked, relevant[, k]) | Reciprocal of the rank of the first relevant result (`0` if none). |
| [`average_precision`](_autosummary/ef.evaluation.html.md#ef.evaluation.average_precision)(ranked, relevant[, k]) | Average precision of `ranked` — the per-query term of MAP. |
| [`evaluate_retrieval`](_autosummary/ef.evaluation.html.md#ef.evaluation.evaluate_retrieval)(retriever, qrels, queries, \*) | Score a retriever against a BEIR-shaped evaluation set. |
| [`read_beir`](_autosummary/ef.evaluation.html.md#ef.evaluation.read_beir)(directory, \*[, qrels]) | Load a BEIR-format dataset from disk into the in-memory evaluation triple. |
| [`write_beir`](_autosummary/ef.evaluation.html.md#ef.evaluation.write_beir)(directory, corpus, queries, qrels) | Write an evaluation triple to disk in BEIR format — the inverse of [`read_beir()`](_autosummary/ef.evaluation.html.md#ef.evaluation.read_beir). |
| [`exact_match`](_autosummary/ef.evaluation.html.md#ef.evaluation.exact_match)(response, reference) | `1.0` if `response` equals `reference` after normalization, else `0.0`. |
| [`token_f1`](_autosummary/ef.evaluation.html.md#ef.evaluation.token_f1)(response, reference) | Token-overlap F1 of `response` against `reference` (the SQuAD F1). |
| [`context_recall`](_autosummary/ef.evaluation.html.md#ef.evaluation.context_recall)(reference, retrieved_contexts) | Lexical recall of the `reference` tokens by the retrieved contexts. |
| [`context_precision`](_autosummary/ef.evaluation.html.md#ef.evaluation.context_precision)(reference, retrieved_contexts) | Fraction of the retrieved contexts that lexically overlap the `reference`. |
| [`evaluate_rag`](_autosummary/ef.evaluation.html.md#ef.evaluation.evaluate_rag)(samples, \*[, metrics]) | Score RAG samples with deterministic, reference-based lexical metrics. |
| [`as_ragas_dataset`](_autosummary/ef.evaluation.html.md#ef.evaluation.as_ragas_dataset)(samples) | Convert [`RagSample`](_autosummary/ef.evaluation.html.md#ef.evaluation.RagSample)s into a Ragas `EvaluationDataset`. |
### Classes
| [`RagSample`](_autosummary/ef.evaluation.html.md#ef.evaluation.RagSample) | One end-to-end RAG interaction — the unit [`evaluate_rag()`](_autosummary/ef.evaluation.html.md#ef.evaluation.evaluate_rag) scores. |
|-----------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------|
| [`RetrievalEvalReport`](_autosummary/ef.evaluation.html.md#ef.evaluation.RetrievalEvalReport)(metrics, per_query, ...) | The outcome of [`evaluate_retrieval()`](_autosummary/ef.evaluation.html.md#ef.evaluation.evaluate_retrieval) — metrics over a query set. |
| [`RagEvalReport`](_autosummary/ef.evaluation.html.md#ef.evaluation.RagEvalReport)(metrics, coverage, n_samples) | The outcome of [`evaluate_rag()`](_autosummary/ef.evaluation.html.md#ef.evaluation.evaluate_rag) — lexical RAG metrics over a sample set. |
### ef.evaluation.Qrels
Graded relevance judgements — `{query_id: {doc_id: relevance}}`. A
`relevance` of `0` means *judged non-relevant*; `> 0` means relevant
(graded judgements feed [`ndcg_at_k()`](_autosummary/ef.evaluation.html.md#ef.evaluation.ndcg_at_k), binary relevance the rest). This
is the in-memory form of a BEIR `qrels.tsv`.
alias of [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`float`](https://docs.python.org/3/builtins/functions.html#float)]]
### ef.evaluation.RETRIEVAL_METRICS *: [dict](https://docs.python.org/3/builtins/stdtypes.html#dict)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), [Callable](https://docs.python.org/3/library/typing.html#typing.Callable)[[[Sequence](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[str](https://docs.python.org/3/builtins/stdtypes.html#str)], [Mapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), [float](https://docs.python.org/3/builtins/functions.html#float)], [int](https://docs.python.org/3/builtins/functions.html#int)], [float](https://docs.python.org/3/builtins/functions.html#float)]]* *= {'map': , 'mrr': , 'ndcg': , 'precision': , 'recall': }*
The retrieval metrics [`evaluate_retrieval()`](_autosummary/ef.evaluation.html.md#ef.evaluation.evaluate_retrieval) can compute, each keyed by
the short name used in the `metrics=` argument and in report keys. Public
so downstream packages (e.g. `ir.eval`) can consume this single source of
truth — the name→function registry *and* the set of valid metric names —
instead of hand-mirroring it.
### *class* ef.evaluation.RagEvalReport(metrics, coverage, n_samples)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
The outcome of [`evaluate_rag()`](_autosummary/ef.evaluation.html.md#ef.evaluation.evaluate_rag) — lexical RAG metrics over a sample set.
`metrics` maps each metric to its mean over the samples it applied to;
`coverage` records how many samples that was (a metric needing a
`reference` skips samples without one). `n_samples` is the total scored.
```pycon
>>> report = RagEvalReport(
... metrics={'exact_match': 0.5}, coverage={'exact_match': 4}, n_samples=4)
>>> report.metrics['exact_match']
0.5
```
### *class* ef.evaluation.RagSample
Bases: [`TypedDict`](https://docs.python.org/3/library/typing.html#typing.TypedDict)
One end-to-end RAG interaction — the unit [`evaluate_rag()`](_autosummary/ef.evaluation.html.md#ef.evaluation.evaluate_rag) scores.
The field names match Ragas’ `SingleTurnSample` exactly, so a
[`RagSample`](_autosummary/ef.evaluation.html.md#ef.evaluation.RagSample) is the interchange shape `app_ef` / `srag` / `raglab`
produce and [`as_ragas_dataset()`](_autosummary/ef.evaluation.html.md#ef.evaluation.as_ragas_dataset) forwards unchanged. `ef` itself fills
only `retrieved_contexts` (from [`retrieve()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.retrieve));
`response` is the *caller’s* LLM output — `ef` does not synthesize it.
Keys:
: user_input: the user’s question.
response: the generated answer (the caller’s LLM produced it).
retrieved_contexts: the context passages handed to the LLM — what
> [`retrieve()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.retrieve) returned, as text.
reference: the ground-truth answer, if known.
reference_contexts: the ground-truth context passages, if known.
### *class* ef.evaluation.RetrievalEvalReport(metrics, per_query, n_queries, k_values)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
The outcome of [`evaluate_retrieval()`](_autosummary/ef.evaluation.html.md#ef.evaluation.evaluate_retrieval) — metrics over a query set.
`metrics` maps a `"@"` key (e.g. `"ndcg@10"`) to the mean of
that metric over every scored query; `per_query` keeps the same keys per
query for drill-down. `n_queries` is how many queries were actually
scored (a query with no positive judgement is skipped).
```pycon
>>> report = RetrievalEvalReport(
... metrics={'ndcg@10': 0.83}, per_query={}, n_queries=12, k_values=(10,))
>>> report.primary
0.83
```
#### *property* primary *: [float](https://docs.python.org/3/builtins/functions.html#float) | [None](https://docs.python.org/3/builtins/constants.html#None)*
The headline number — mean [NDCG@10](mailto:NDCG@10), or `None` if it was not computed.
#### primary_at(k)
Mean NDCG at cutoff `k` — `None` if NDCG@\`\`k\`\` was not computed.
* **Return type:**
[`float`](https://docs.python.org/3/builtins/functions.html#float) | [`None`](https://docs.python.org/3/builtins/constants.html#None)
### ef.evaluation.as_ragas_dataset(samples)
Convert [`RagSample`](_autosummary/ef.evaluation.html.md#ef.evaluation.RagSample)s into a Ragas `EvaluationDataset`.
The hookpoint for the *LLM-judged* RAG metrics `ef` deliberately does not
compute itself (faithfulness, answer relevancy, …). [`RagSample`](_autosummary/ef.evaluation.html.md#ef.evaluation.RagSample)
already mirrors Ragas’ `SingleTurnSample` field-for-field, so this is a
thin, lossless adapter. Run the evaluation with your own LLM:
```default
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy
dataset = as_ragas_dataset(samples)
result = evaluate(dataset, metrics=[faithfulness, answer_relevancy])
```
* **Parameters:**
**samples** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`RagSample`](_autosummary/ef.evaluation.html.md#ef.evaluation.RagSample) | [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]]) – an iterable of [`RagSample`](_autosummary/ef.evaluation.html.md#ef.evaluation.RagSample)s (or plain mappings).
* **Return type:**
[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)
* **Returns:**
a `ragas.EvaluationDataset`.
* **Raises:**
[**ImportError**](https://docs.python.org/3/builtins/exceptions.html#ImportError) – if the optional `ragas` package is not installed.
### ef.evaluation.average_precision(ranked, relevant, k=None)
Average precision of `ranked` — the per-query term of MAP.
The mean of the precision measured at each relevant result’s rank. `k`
optionally caps the ranking.
* **Return type:**
[`float`](https://docs.python.org/3/builtins/functions.html#float)
```pycon
>>> average_precision(['d1', 'x', 'd2'], {'d1': 1.0, 'd2': 1.0})
0.8333333333333333
```
### ef.evaluation.context_precision(reference, retrieved_contexts)
Fraction of the retrieved contexts that lexically overlap the `reference`.
A deterministic *proxy* for retrieval precision — a context counts as useful
if it shares at least one (normalized) token with the reference answer.
* **Return type:**
[`float`](https://docs.python.org/3/builtins/functions.html#float)
```pycon
>>> context_precision('paris', ['Paris is in France.', 'Unrelated text.'])
0.5
```
### ef.evaluation.context_recall(reference, retrieved_contexts)
Lexical recall of the `reference` tokens by the retrieved contexts.
The fraction of the (normalized) reference’s distinct tokens that appear
anywhere in `retrieved_contexts` — a deterministic *proxy* for “did
retrieval surface the information the answer needs”. For the LLM-judged
context recall, use [`as_ragas_dataset()`](_autosummary/ef.evaluation.html.md#ef.evaluation.as_ragas_dataset) and Ragas.
* **Return type:**
[`float`](https://docs.python.org/3/builtins/functions.html#float)
```pycon
>>> context_recall('paris france', ['Paris is the capital of France.'])
1.0
>>> context_recall('paris france', ['London is in England.'])
0.0
```
### ef.evaluation.dcg_at_k(ranked, relevant, k)
Discounted cumulative gain of `ranked` at cutoff `k`.
`sum(grade_i / log2(i + 2))` over the top `k` results (`i` 0-indexed),
with `grade_i` the relevance of the `i`-th doc (`0` if unjudged) — the
linear-gain DCG that BEIR / `pytrec_eval` use.
* **Return type:**
[`float`](https://docs.python.org/3/builtins/functions.html#float)
```pycon
>>> round(dcg_at_k(['d1', 'x', 'd2'], {'d1': 1.0, 'd2': 1.0}, 3), 4)
1.5
```
### ef.evaluation.evaluate_rag(samples, , metrics=('exact_match', 'token_f1', 'context_recall', 'context_precision', 'non_empty_rate'))
Score RAG samples with deterministic, reference-based lexical metrics.
Each sample is a [`RagSample`](_autosummary/ef.evaluation.html.md#ef.evaluation.RagSample) — `(user_input, response,
retrieved_contexts, reference)`. `evaluate_rag` computes only metrics
that need **no LLM**: [`exact_match()`](_autosummary/ef.evaluation.html.md#ef.evaluation.exact_match), [`token_f1()`](_autosummary/ef.evaluation.html.md#ef.evaluation.token_f1),
[`context_recall()`](_autosummary/ef.evaluation.html.md#ef.evaluation.context_recall), [`context_precision()`](_autosummary/ef.evaluation.html.md#ef.evaluation.context_precision) and `non_empty_rate`
(the share of samples that retrieved any context). This keeps `ef` a
facade — it never synthesizes an answer and never calls a model (§6).
For the LLM-judged metrics — faithfulness, answer relevancy, … — pass the
same samples to [`as_ragas_dataset()`](_autosummary/ef.evaluation.html.md#ef.evaluation.as_ragas_dataset) and run Ragas with your own LLM.
* **Parameters:**
* **samples** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`RagSample`](_autosummary/ef.evaluation.html.md#ef.evaluation.RagSample) | [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]]) – an iterable of [`RagSample`](_autosummary/ef.evaluation.html.md#ef.evaluation.RagSample)s (or plain mappings of the
same shape).
* **metrics** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – which metrics to compute — any of `exact_match` /
`token_f1` / `context_recall` / `context_precision` /
`non_empty_rate`. The reference-based ones skip samples that carry
no `reference`.
* **Return type:**
[`RagEvalReport`](_autosummary/ef.evaluation.html.md#ef.evaluation.RagEvalReport)
* **Returns:**
a [`RagEvalReport`](_autosummary/ef.evaluation.html.md#ef.evaluation.RagEvalReport).
* **Raises:**
[**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – for an unknown metric name, or if `samples` is empty.
```pycon
>>> samples = [
... {'user_input': 'capital of France?', 'response': 'Paris',
... 'reference': 'Paris',
... 'retrieved_contexts': ['Paris is the capital of France.']},
... {'user_input': '2 + 2?', 'response': 'five', 'reference': 'four',
... 'retrieved_contexts': []},
... ]
>>> report = evaluate_rag(samples)
>>> report.metrics['exact_match']
0.5
>>> report.metrics['non_empty_rate']
0.5
```
### ef.evaluation.evaluate_retrieval(retriever, qrels, queries, , k_values=(1, 5, 10, 100), metrics=('ndcg', 'recall', 'precision', 'mrr', 'map'), limit=None)
Score a retriever against a BEIR-shaped evaluation set.
For every query judged in `qrels`, the `retriever` is run, its results
are mapped back to *source document ids* (`_result_doc_id()`,
de-duplicated to the best rank per source), and each requested metric is
computed at each cutoff in `k_values`. The report holds the mean of each
`"@"` over all scored queries — primary metric **NDCG@10**.
The natural pairing with [`ingest()`](_autosummary/ef.source_manager.html.md#ef.source_manager.ingest):
```default
corpus, queries, qrels = read_beir("path/to/beir/dataset")
index = ingest(corpus, embedder="st:all-MiniLM-L6-v2")
report = evaluate_retrieval(index, qrels, queries)
report.primary # mean NDCG@10
```
* **Parameters:**
* **retriever** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – what answers a query — a [`SearchableCorpus`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus),
a [`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager), or any callable
`query -> ranked results`. Results may be
[`SearchHit`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchHit)s, [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)s
or bare doc-id strings.
* **qrels** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`float`](https://docs.python.org/3/builtins/functions.html#float)]]) – the relevance judgements — `{query_id: {doc_id: relevance}}`.
Queries with no positive judgement are skipped.
* **queries** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – the query texts — `{query_id: query_text}`. A judged query
absent here cannot be run and is skipped.
* **k_values** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`int`](https://docs.python.org/3/builtins/functions.html#int)]) – the rank cutoffs to evaluate at.
* **metrics** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – which metrics to compute — any of `ndcg` / `recall` /
`precision` / `mrr` / `map`.
* **limit** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – how many results to retrieve per query. `None` uses
`max(k_values)`; raise it when sources segment into many pieces
(de-duplication to source level can otherwise shorten the ranking).
* **Return type:**
[`RetrievalEvalReport`](_autosummary/ef.evaluation.html.md#ef.evaluation.RetrievalEvalReport)
* **Returns:**
a [`RetrievalEvalReport`](_autosummary/ef.evaluation.html.md#ef.evaluation.RetrievalEvalReport).
* **Raises:**
[**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – for an unknown metric name, or if no query can be scored.
```pycon
>>> qrels = {'q1': {'d1': 1.0}, 'q2': {'d2': 1.0}}
>>> queries = {'q1': 'alpha', 'q2': 'beta'}
>>> def retriever(query, *, limit=10):
... return {'alpha': ['d1', 'd2'], 'beta': ['d1', 'd2']}[query][:limit]
>>> report = evaluate_retrieval(retriever, qrels, queries, k_values=(1, 2))
>>> report.metrics['recall@2'], report.n_queries
(1.0, 2)
```
### ef.evaluation.exact_match(response, reference)
`1.0` if `response` equals `reference` after normalization, else `0.0`.
* **Return type:**
[`float`](https://docs.python.org/3/builtins/functions.html#float)
```pycon
>>> exact_match('The Eiffel Tower', 'eiffel tower')
1.0
>>> exact_match('Paris', 'London')
0.0
```
### ef.evaluation.ndcg_at_k(ranked, relevant, k)
Normalized DCG at cutoff `k` — `ef`’s **primary** retrieval metric.
[`dcg_at_k()`](_autosummary/ef.evaluation.html.md#ef.evaluation.dcg_at_k) divided by the ideal DCG (the DCG of the best possible
ranking of the judged docs). `1.0` is a perfect ranking; `0.0` when no
relevant doc was judged.
* **Return type:**
[`float`](https://docs.python.org/3/builtins/functions.html#float)
```pycon
>>> ndcg_at_k(['d1', 'd2'], {'d1': 1.0, 'd2': 1.0}, 2) # perfect order
1.0
>>> round(ndcg_at_k(['x', 'd1'], {'d1': 1.0}, 2), 4) # relevant doc at rank 2
0.6309
```
### ef.evaluation.precision_at_k(ranked, relevant, k)
Fraction of the top `k` results that are relevant.
* **Return type:**
[`float`](https://docs.python.org/3/builtins/functions.html#float)
```pycon
>>> precision_at_k(['d1', 'x', 'd2'], {'d1': 1.0, 'd2': 1.0}, 2)
0.5
```
### ef.evaluation.read_beir(directory, , qrels=None)
Load a BEIR-format dataset from disk into the in-memory evaluation triple.
Reads `corpus.jsonl` (records `{"_id", "title", "text"}`),
`queries.jsonl` (records `{"_id", "text"}`) and a `qrels` TSV
(`query-id`, `corpus-id`, `score` columns). The corpus text is the
record’s `title` and `text` joined — the BEIR convention.
* **Parameters:**
* **directory** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the dataset directory.
* **qrels** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the qrels file, relative to `directory`. `None` tries
`_DEFAULT_QRELS_PATHS` in order.
* **Return type:**
[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`float`](https://docs.python.org/3/builtins/functions.html#float)]]]
* **Returns:**
a `(corpus, queries, qrels)` triple — `corpus` and `queries` are
`{id: text}` dicts, `qrels` is `{query_id: {doc_id: relevance}}`.
Feed it straight to [`evaluate_retrieval()`](_autosummary/ef.evaluation.html.md#ef.evaluation.evaluate_retrieval) (after indexing `corpus`
with [`ingest()`](_autosummary/ef.source_manager.html.md#ef.source_manager.ingest)).
* **Raises:**
[**FileNotFoundError**](https://docs.python.org/3/builtins/exceptions.html#FileNotFoundError) – if `corpus.jsonl`, `queries.jsonl` or the qrels
file is missing.
### ef.evaluation.recall_at_k(ranked, relevant, k)
Fraction of the relevant docs that appear in the top `k` results.
* **Return type:**
[`float`](https://docs.python.org/3/builtins/functions.html#float)
```pycon
>>> recall_at_k(['d1', 'x', 'd2'], {'d1': 1.0, 'd2': 1.0}, 2)
0.5
>>> recall_at_k(['d1', 'x', 'd2'], {'d1': 1.0, 'd2': 1.0}, 3)
1.0
```
### ef.evaluation.reciprocal_rank(ranked, relevant, k=None)
Reciprocal of the rank of the first relevant result (`0` if none).
Averaged over queries this is MRR. `k` optionally caps how far down the
ranking to look.
* **Return type:**
[`float`](https://docs.python.org/3/builtins/functions.html#float)
```pycon
>>> reciprocal_rank(['x', 'd1', 'd2'], {'d1': 1.0})
0.5
>>> reciprocal_rank(['x', 'd1'], {'d1': 1.0}, k=1) # relevant doc past the cutoff
0.0
```
### ef.evaluation.token_f1(response, reference)
Token-overlap F1 of `response` against `reference` (the SQuAD F1).
The harmonic mean of token precision and recall over the normalized token
bags — a graded credit where [`exact_match()`](_autosummary/ef.evaluation.html.md#ef.evaluation.exact_match) is all-or-nothing.
* **Return type:**
[`float`](https://docs.python.org/3/builtins/functions.html#float)
```pycon
>>> token_f1('the quick brown fox', 'a quick brown fox')
1.0
>>> round(token_f1('quick brown fox', 'quick brown dog'), 4)
0.6667
```
### ef.evaluation.write_beir(directory, corpus, queries, qrels)
Write an evaluation triple to disk in BEIR format — the inverse of [`read_beir()`](_autosummary/ef.evaluation.html.md#ef.evaluation.read_beir).
Creates `directory` (and parents) and writes `corpus.jsonl`,
`queries.jsonl` and `qrels.tsv`. Use it to snapshot an `ef` corpus as
a reusable benchmark (`ef_use_cases.md` §H4).
* **Parameters:**
* **directory** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the output directory — created if absent.
* **corpus** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – the documents — `{doc_id: text}`.
* **queries** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – the queries — `{query_id: query_text}`.
* **qrels** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`float`](https://docs.python.org/3/builtins/functions.html#float)]]) – the relevance judgements — `{query_id: {doc_id: relevance}}`.
* **Return type:**
[`None`](https://docs.python.org/3/builtins/constants.html#None)
# _autosummary/ef.exploration.html.md
# ef.exploration
Layer L5 — *explore* a corpus: project to 2-D/3-D, cluster, label clusters.
This is `ef`’s visualization heritage, inherited from `imbed` and
re-presented as a **secondary** surface. The *primary* surface of `ef` is
search / RAG / corpus-indexing (see [`ef.ingest()`](_autosummary/ef.html.md#ef.ingest) and
[`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager)); `explore` is the “see the shape of
the corpus” tool — the backend an `app_ef` corpus map consumes.
Four free functions, each accepting a corpus *or* an already-computed vector
matrix (whatever `_resolve_vectors()` understands):
- [`project()`](_autosummary/ef.exploration.html.md#ef.exploration.project) — reduce embeddings to 2-D/3-D coordinates (PCA → UMAP).
- [`cluster()`](_autosummary/ef.exploration.html.md#ef.exploration.cluster) — group embeddings into clusters (k-means / HDBSCAN).
- [`label_clusters()`](_autosummary/ef.exploration.html.md#ef.exploration.label_clusters) — name clusters with an LLM (via `imbed`).
- [`explore()`](_autosummary/ef.exploration.html.md#ef.exploration.explore) — the orchestrator: project + cluster (+ optionally label) in
one call, returning a structured, JSON-friendly [`ExploreResult`](_autosummary/ef.exploration.html.md#ef.exploration.ExploreResult)
(`ids` / `coords` / `labels` / `cluster_titles`) — the surface an
`app_ef` corpus map (or [`ef.service.EfService.explore_corpus()`](_autosummary/ef.service.html.md#ef.service.EfService.explore_corpus))
consumes directly.
The module imports with **numpy only**. UMAP, HDBSCAN and `imbed` are
optional: each is imported lazily, inside the function that needs it, and only
that heavy path requires the corresponding extra (`ef[explore]` /
`ef[imbed]`). The numpy-only paths — PCA projection and k-means clustering —
work out of the box with no install.
Example — project a small set of vectors to the plane (numpy only):
```pycon
>>> import numpy as np
>>> vectors = np.random.RandomState(0).rand(12, 16)
>>> coords = project(vectors, dims=2, method='pca')
>>> coords.shape
(12, 2)
>>> labels = cluster(vectors, method='kmeans', n_clusters=3, random_state=0)
>>> labels.shape
(12,)
```
Each returned array’s rows are in the iteration order of the input — for a
[`SearchableCorpus`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus) that is `list(searchable.collection)`,
which the caller zips back against to recover `{id: coords}`. [`explore()`](_autosummary/ef.exploration.html.md#ef.exploration.explore)
does that zip-back for you — its [`ExploreResult`](_autosummary/ef.exploration.html.md#ef.exploration.ExploreResult) keeps `ids` aligned
with `coords` and `labels`.
### Functions
| [`project`](_autosummary/ef.exploration.html.md#ef.exploration.project)(data, \*[, dims, method, embedder, ...]) | Project a corpus's embeddings to `dims` coordinates for visualization. |
|-----------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------|
| [`cluster`](_autosummary/ef.exploration.html.md#ef.exploration.cluster)(data, \*[, method, n_clusters, ...]) | Cluster a corpus's embeddings (use cases §G2). |
| [`label_clusters`](_autosummary/ef.exploration.html.md#ef.exploration.label_clusters)(segments, labels, \*[, ...]) | Name each cluster with a short LLM-generated title (use cases §G3). |
| [`explore`](_autosummary/ef.exploration.html.md#ef.exploration.explore)(data, \*[, dims, projection_method, ...]) | Project, cluster and (optionally) label a corpus in one call. |
### Classes
| [`ExploreResult`](_autosummary/ef.exploration.html.md#ef.exploration.ExploreResult) | A structured, JSON-friendly corpus-exploration result. |
|------------------------------------------------------------------|----------------------------------------------------------|
### *class* ef.exploration.ExploreResult
Bases: [`TypedDict`](https://docs.python.org/3/library/typing.html#typing.TypedDict)
A structured, JSON-friendly corpus-exploration result.
What [`explore()`](_autosummary/ef.exploration.html.md#ef.exploration.explore) returns and [`ef.service.EfService.explore_corpus()`](_autosummary/ef.service.html.md#ef.service.EfService.explore_corpus)
serves. Every list is **row-aligned**: `ids[i]`, `coords[i]` and
`labels[i]` all describe the same item.
Keys:
: ids: the per-item identifiers — a corpus’s keys, or positional `"0"`,
: `"1"`, … for an id-less vector matrix.
coords: the projected coordinates — one `[x, y]` (or `[x, y, z]`)
: row per item.
labels: the cluster id of each item (HDBSCAN marks noise `-1`).
cluster_titles: `{cluster_id: title}` — empty unless [`explore()`](_autosummary/ef.exploration.html.md#ef.exploration.explore)
> was called with `label=True`.
### ef.exploration.cluster(data, , method='kmeans', n_clusters=8, min_cluster_size=5, embedder=None, normalize=True, random_state=42)
Cluster a corpus’s embeddings (use cases §G2).
`method='kmeans'` is the default and is **dependency-free** — a numpy
Lloyd’s algorithm with k-means++ seeding. `method='hdbscan'` finds a
density-based clustering (and a noise label `-1`) and needs the
`ef[explore]` extra.
* **Parameters:**
* **data** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – what to cluster — see [`project()`](_autosummary/ef.exploration.html.md#ef.exploration.project) / `_resolve_vectors()`.
* **method** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'kmeans'`, `'hdbscan'`]) – `'kmeans'` (numpy-only) or `'hdbscan'` (optional extra).
* **n_clusters** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – number of clusters for k-means (clamped to `n_samples`).
* **min_cluster_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – smallest cluster HDBSCAN will form.
* **embedder** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – embedder used when `data` is raw text.
* **normalize** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – L2-normalize the vectors first, so Euclidean distance
behaves like cosine similarity — the right default for semantic
embeddings. Set `False` to cluster raw vectors.
* **random_state** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – seed for k-means initialization.
* **Return type:**
`ndarray`
* **Returns:**
An `ndarray` of integer cluster labels, shape `(n_samples,)`, in
input order. HDBSCAN labels unclustered points `-1`.
```pycon
>>> import numpy as np
>>> vectors = np.random.RandomState(2).rand(15, 8)
>>> labels = cluster(vectors, method='kmeans', n_clusters=3, random_state=0)
>>> labels.shape
(15,)
>>> set(labels.tolist()) <= {0, 1, 2}
True
```
### ef.exploration.explore(data, , dims=2, projection_method='auto', cluster_method='kmeans', n_clusters=8, min_cluster_size=5, label=False, context=' ', n_words=4, embedder=None, random_state=42)
Project, cluster and (optionally) label a corpus in one call.
The layer-L5 orchestrator — [`project()`](_autosummary/ef.exploration.html.md#ef.exploration.project) + [`cluster()`](_autosummary/ef.exploration.html.md#ef.exploration.cluster) (+ optionally
[`label_clusters()`](_autosummary/ef.exploration.html.md#ef.exploration.label_clusters)) wired into one structured result. Where
[`project()`](_autosummary/ef.exploration.html.md#ef.exploration.project) / [`cluster()`](_autosummary/ef.exploration.html.md#ef.exploration.cluster) return *bare arrays in input order*,
`explore` keeps every row tied to its id: it returns an
[`ExploreResult`](_autosummary/ef.exploration.html.md#ef.exploration.ExploreResult) with `ids` / `coords` / `labels` row-aligned
(plus `cluster_titles`) — the JSON-friendly shape an `app_ef` corpus map
(or [`ef.service.EfService.explore_corpus()`](_autosummary/ef.service.html.md#ef.service.EfService.explore_corpus)) consumes directly.
* **Parameters:**
* **data** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the corpus to explore — anything `_resolve_explorable()`
understands (a [`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager) /
[`SearchableCorpus`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus), a
[`Corpus`](_autosummary/ef.corpus.html.md#ef.corpus.Corpus) mapping, an iterable of texts or
[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)s, or a vector matrix).
* **dims** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – projection target dimensionality — `2` or `3`.
* **projection_method** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'auto'`, `'umap'`, `'pca'`]) – forwarded to [`project()`](_autosummary/ef.exploration.html.md#ef.exploration.project) — `'auto'` /
`'umap'` / `'pca'`.
* **cluster_method** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'kmeans'`, `'hdbscan'`]) – forwarded to [`cluster()`](_autosummary/ef.exploration.html.md#ef.exploration.cluster) — `'kmeans'` /
`'hdbscan'`.
* **n_clusters** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – number of k-means clusters.
* **min_cluster_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – smallest cluster HDBSCAN will form.
* **label** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – when `True`, name each cluster with [`label_clusters()`](_autosummary/ef.exploration.html.md#ef.exploration.label_clusters)
(needs the `ef[imbed]` extra, an LLM key, and text-bearing
`data`); when `False` (the default) `cluster_titles` is empty.
* **context** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the corpus topic passed to [`label_clusters()`](_autosummary/ef.exploration.html.md#ef.exploration.label_clusters).
* **n_words** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – maximum cluster-title length, in words.
* **embedder** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – embedder used when `data` is raw text.
* **random_state** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – seed — projection and clustering are reproducible.
* **Return type:**
[`ExploreResult`](_autosummary/ef.exploration.html.md#ef.exploration.ExploreResult)
* **Returns:**
an [`ExploreResult`](_autosummary/ef.exploration.html.md#ef.exploration.ExploreResult) — `ids`, `coords` and `labels`
row-aligned, and `cluster_titles` (empty unless `label=True`).
* **Raises:**
[**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if the corpus is empty, has fewer than 2 samples, or
`label=True` is asked of a text-less vector matrix.
```pycon
>>> import numpy as np
>>> vectors = np.random.RandomState(0).rand(12, 16)
>>> result = explore(vectors, projection_method='pca', n_clusters=3, random_state=0)
>>> len(result['ids']), len(result['coords']), len(result['labels'])
(12, 12, 12)
>>> result['ids'][:3]
['0', '1', '2']
>>> len(result['coords'][0]) # each row has `dims` coordinates
2
>>> set(result['labels']) <= {0, 1, 2}
True
>>> result['cluster_titles'] # empty unless label=True
{}
```
### ef.exploration.label_clusters(segments, labels, , context=' ', n_words=4, n_samples=None, \*\*labeler_kwargs)
Name each cluster with a short LLM-generated title (use cases §G3).
A thin wrapper over `imbed`’s `imbed.tools.ClusterLabeler`: it
samples each cluster’s segments, asks an LLM for a few-word title, and
returns `{cluster_id: title}`. `ef` does not reimplement this — it
reuses `imbed` (design notes §7).
* **Parameters:**
* **segments** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – the segment texts — plain strings or [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)
mappings (the `"text"` field is used). Aligned with `labels`.
* **labels** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – the cluster id of each segment — typically the output of
[`cluster()`](_autosummary/ef.exploration.html.md#ef.exploration.cluster).
* **context** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the corpus’s overall topic, given to the LLM so titles
describe how a cluster *differs* from the rest, not the shared topic.
* **n_words** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – maximum title length, in words.
* **n_samples** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – how many segments to sample per cluster when prompting;
`None` keeps `ClusterLabeler`’s default.
* **\*\*labeler_kwargs** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – forwarded to `imbed.tools.ClusterLabeler`
(e.g. `max_unique_clusters`, `prompt`).
* **Return type:**
[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`int`](https://docs.python.org/3/builtins/functions.html#int), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]
* **Returns:**
`{cluster_id: title}` for every distinct label.
* **Raises:**
* [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if `segments` and `labels` differ in length.
* [**ImportError**](https://docs.python.org/3/builtins/exceptions.html#ImportError) – if `imbed` (and its `pandas`/`oa` deps) is missing.
```pycon
>>> # needs ef[imbed] + an LLM key — illustrative only:
>>> # label_clusters(['neural net training ...', ...], [0, 1, 0, ...])
>>> # -> {0: 'Neural network training', 1: 'Dataset preprocessing'}
```
### ef.exploration.project(data, , dims=2, method='auto', embedder=None, n_neighbors=15, min_dist=0.1, metric='cosine', pca_components=50, random_state=42)
Project a corpus’s embeddings to `dims` coordinates for visualization.
The canonical “show me the shape of the corpus” operation (use cases §G1).
The default `method='auto'` runs the design-notes §8.4 recipe — \*\*PCA →
UMAP\*\*, `cosine` metric, seeded: PCA first compresses high-dimensional
embeddings to `pca_components` (cheap denoising), then UMAP lays them out
in `dims` dimensions.
* **Parameters:**
* **data** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – what to project — anything `_resolve_vectors()` understands:
an `ndarray` (or nested sequence) of vectors, an iterable of
texts (embedded with `embedder`), a [`Corpus`](_autosummary/ef.corpus.html.md#ef.corpus.Corpus)
mapping, or a [`SearchableCorpus`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus) /
[`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager) (vectors pulled from the
index).
* **dims** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – target dimensionality — `2` or `3` for plotting.
* **method** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'auto'`, `'umap'`, `'pca'`]) – `'auto'` (UMAP when available and the corpus is large enough,
else PCA), `'umap'` (force UMAP — needs the `ef[explore]`
extra), or `'pca'` (numpy-only, always available).
* **embedder** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – embedder used when `data` is raw text; defaults to the
dependency-free [`DEFAULT_EMBEDDER`](_autosummary/ef.source_manager.html.md#ef.source_manager.DEFAULT_EMBEDDER).
* **n_neighbors** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – UMAP neighborhood size (clamped to `n_samples - 1`).
* **min_dist** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – UMAP minimum point separation in the embedding.
* **metric** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – UMAP distance metric — `'cosine'` for semantic embeddings.
* **pca_components** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – PCA width for the pre-UMAP compression step; PCA is
skipped when the source dimensionality is already smaller.
* **random_state** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – seed — projection is reproducible.
* **Return type:**
`ndarray`
* **Returns:**
An `ndarray` of shape `(n_samples, dims)`, rows in input order.
* **Raises:**
* [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if there are fewer than 2 samples.
* [**ImportError**](https://docs.python.org/3/builtins/exceptions.html#ImportError) – if `method='umap'` and `umap-learn` is not installed.
```pycon
>>> import numpy as np
>>> vectors = np.random.RandomState(1).rand(20, 32)
>>> project(vectors, dims=3, method='pca').shape
(20, 3)
```
# _autosummary/ef.hashing.html.md
# ef.hashing
Content-hashing primitives shared across `ef`’s layers.
Content addressing is the backbone of `ef`’s identity and change-detection
model: a segment’s id (L2, [`ef.segments.segment_id()`](_autosummary/ef.segments.html.md#ef.segments.segment_id)), a source’s content
hash (L0, [`ef.corpus.content_hash()`](_autosummary/ef.corpus.html.md#ef.corpus.content_hash)) and — from the `ArtifactGraph` on —
every artifact id are all `SHA-256` over *normalized* content.
Normalizing *before* hashing is the point. Cosmetically-different encodings of
the same content — NFD vs NFC, a stray BOM, CRLF vs LF line endings, JSON with
keys in a different order — must hash *identically*, so that re-ingesting
unchanged data is a no-op rather than a spurious “change” that re-runs the
expensive downstream pipeline. This module is the **single source of truth**
for that normalization; every other `ef` module hashes through it.
```pycon
>>> sha256_hex(normalize_text('a\r\nb')) == sha256_hex(normalize_text('a\nb'))
True
```
### Functions
| [`normalize_text`](_autosummary/ef.hashing.html.md#ef.hashing.normalize_text)(text) | Normalize `text` for hashing: Unicode NFC, no BOM, `\n` line endings. |
|-------------------------------------------------------------------------|-------------------------------------------------------------------------|
| [`canonical_json`](_autosummary/ef.hashing.html.md#ef.hashing.canonical_json)(obj) | Serialize `obj` to deterministic JSON: sorted keys, compact separators. |
| [`sha256_hex`](_autosummary/ef.hashing.html.md#ef.hashing.sha256_hex)(payload) | Return the `SHA-256` hex digest of `payload` (`str` is UTF-8 encoded). |
### ef.hashing.canonical_json(obj)
Serialize `obj` to deterministic JSON: sorted keys, compact separators.
Two mappings that are equal but were built in a different key order produce
the *same* string — so they hash identically. Non-JSON values fall back to
their `str` form, so the function never raises on exotic input (at the
cost of those values not round-tripping).
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
```pycon
>>> canonical_json({'b': 1, 'a': 2})
'{"a":2,"b":1}'
>>> canonical_json({'a': 2, 'b': 1}) == canonical_json({'b': 1, 'a': 2})
True
```
### ef.hashing.normalize_text(text)
Normalize `text` for hashing: Unicode NFC, no BOM, `\n` line endings.
The normalization makes hashing robust to differences that carry no
meaning: a composed vs decomposed accent, a leading byte-order mark, or
Windows vs Unix line endings. Only the *hash input* is normalized — callers
keep the original `text` verbatim.
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
```pycon
>>> normalize_text('a\r\nb\rc') == 'a\nb\nc'
True
>>> normalize_text('\ufeffhi') == 'hi'
True
>>> normalize_text('café') == normalize_text('cafe\u0301') # NFC vs NFD
True
```
### ef.hashing.sha256_hex(payload)
Return the `SHA-256` hex digest of `payload` (`str` is UTF-8 encoded).
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
```pycon
>>> sha256_hex('')
'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855'
>>> sha256_hex('abc') == sha256_hex(b'abc')
True
```
# _autosummary/ef.html.md
# ef
ef (Embedding Flow) — a facade for semantic-embeddings user journeys.
`ef` makes the modern semantic-search pipeline — corpus, segmenter, embedder,
vector store, retriever — usable with progressive disclosure: the light case
(a list of strings, search in one or two lines) and the heavy case (huge
corpora, many segmentations/embedders, varied sources and vector DBs) share one
facade. `ef` is *not* RAG — it returns ranked segments; bring your own LLM.
The **embedder facade** is the core surface and is always importable (it costs
only numpy):
- [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) — the structural protocol for a batch
`Iterable[str] -> ndarray(n, dim)` callable.
- [`HashingEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.HashingEmbedder) — the dependency-free default embedder
(the feature-hashing trick, numpy only); what [`ingest()`](_autosummary/ef.source_manager.html.md#ef.source_manager.ingest)
resolves to when given no embedder, so the light path needs nothing to install.
- [`as_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.as_embedder) — the dependency-injection seam
(string / callable / URL / existing embedder → `Embedder`).
- Adapters: [`openai_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.openai_embedder),
[`sentence_transformers_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.sentence_transformers_embedder),
[`http_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.http_embedder),
[`cohere_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.cohere_embedder),
[`voyage_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.voyage_embedder),
[`gemini_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.gemini_embedder).
- Composition wrappers: [`CachedEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.CachedEmbedder),
[`RetryingEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.RetryingEmbedder),
[`MultiEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.MultiEmbedder),
[`NormalizingEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.NormalizingEmbedder),
[`InstructionEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.InstructionEmbedder).
The **segmenter facade** is the other always-importable core surface (it needs
no numpy at all):
- [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) — the canonical segment data model (the
`TypedDict` interchange type; [`SegmentRecord`](_autosummary/ef.segments.html.md#ef.segments.SegmentRecord) is its
dataclass convenience surface).
- [`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter) — the structural protocol for a
`str | Mapping -> Iterable[Segment]` callable.
- [`RecursiveCharacterSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.RecursiveCharacterSegmenter) — the default splitter;
[`line_segmenter()`](_autosummary/ef.segmenters.html.md#ef.segmenters.line_segmenter) a builtin line splitter.
- [`as_segmenter()`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.as_segmenter) — the dependency-injection seam.
- Composition helpers: [`with_overlap()`](_autosummary/ef.segmenters.html.md#ef.segmenters.with_overlap),
[`hierarchical()`](_autosummary/ef.segmenters.html.md#ef.segmenters.hierarchical), [`materialise()`](_autosummary/ef.segmenters.html.md#ef.segmenters.materialise).
The **corpus facade** is `ef`’s source layer (L0) — also always importable:
- [`Corpus`](_autosummary/ef.corpus.html.md#ef.corpus.Corpus) / [`Source`](_autosummary/ef.corpus.html.md#ef.corpus.Source) — the type aliases: a
corpus is just a `MutableMapping[source_id, Source]` (any `dol` store).
- [`as_corpus()`](_autosummary/ef.corpus.html.md#ef.corpus.as_corpus) — the dependency-injection seam (`None` /
mapping / directory path / iterable of sources → a corpus).
- [`content_hash()`](_autosummary/ef.corpus.html.md#ef.corpus.content_hash) — the content hash of a source.
- [`ChangeDetectingCorpus`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus) — a corpus wrapper that detects and
reports changes ([`ChangeEvent`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeEvent) / [`CorpusDiff`](_autosummary/ef.corpus.html.md#ef.corpus.CorpusDiff)).
The **artifact graph** is `ef`’s corpus-indexing core (also always importable)
— a content-addressed producer graph in which cascade invalidation and config
branching are the same operation:
- [`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec) — the declarative recipe for one
produced artifact; [`artifact_id()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.artifact_id) is its content hash;
[`producer_spec()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.producer_spec) builds one ergonomically.
- [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph) — the graph: `materialize` (lazy
backward), `mark_stale` / `delete_cascade` (forward) and `freshness`.
The **search facade** is where the layers come together — corpus → segment →
embed → `vd` → ranked search, with progressive disclosure:
- [`ingest()`](_autosummary/ef.source_manager.html.md#ef.source_manager.ingest) — the one-shot light path: a corpus in, a
[`SearchableCorpus`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus) out, `search(query)` ready.
- [`SearchableCorpus`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus) — the thin ready-search object;
[`search()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus.search) returns scored
[`SearchHit`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchHit)s, [`retrieve()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus.retrieve)
> returns plain ranked [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)s — the RAG-plug-in shape.
- [`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager) — the heavy facade: multi-config
corpus indexing, where configs sharing a step share its artifacts for free.
- [`PipelineSpec`](_autosummary/ef.config.html.md#ef.config.PipelineSpec) / [`TransformSpec`](_autosummary/ef.config.html.md#ef.config.TransformSpec) /
[`config_id()`](_autosummary/ef.config.html.md#ef.config.config_id) — a pipeline as serializable, content-hashed data.
The **refresh layer** keeps an indexed corpus in sync as its sources change:
- [`SourceManager.diagnose`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.diagnose) —
the four staleness conditions, as a [`StalenessReport`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.StalenessReport).
- [`SourceManager.refresh`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.refresh) —
re-sync the index, returning a [`RefreshReport`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshReport); one of four
[`RefreshMode`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshMode)s. `SourceManager(auto_refresh=True)` keeps
> the index live as the corpus is edited.
The **RAG-plug-in & evaluation layer** hands a corpus to an external RAG/agent
framework and measures its quality — `ef` returns ranked context, it does not
synthesize answers:
- [`SourceManager.retrieve`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.retrieve)
returns plain ranked [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)s — the RAG-plug-in shape;
[`hits_to_segments()`](_autosummary/ef.source_manager.html.md#ef.source_manager.hits_to_segments) is the projection from
[`SearchHit`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchHit)s.
- [`evaluate_retrieval()`](_autosummary/ef.evaluation.html.md#ef.evaluation.evaluate_retrieval) — BEIR/MTEB-shaped retrieval scoring
(primary metric [NDCG@10](mailto:NDCG@10)); [`read_beir()`](_autosummary/ef.evaluation.html.md#ef.evaluation.read_beir) /
[`write_beir()`](_autosummary/ef.evaluation.html.md#ef.evaluation.write_beir) move the corpus/queries/qrels triple to disk.
- [`evaluate_rag()`](_autosummary/ef.evaluation.html.md#ef.evaluation.evaluate_rag) — deterministic lexical RAG metrics over
[`RagSample`](_autosummary/ef.evaluation.html.md#ef.evaluation.RagSample)s; [`as_ragas_dataset()`](_autosummary/ef.evaluation.html.md#ef.evaluation.as_ragas_dataset)
> bridges to Ragas for the LLM-judged metrics.
- [`with_reranker()`](_autosummary/ef.reranking.html.md#ef.reranking.with_reranker) — a two-stage reranking decorator over any
retriever; [`cross_encoder_reranker()`](_autosummary/ef.reranking.html.md#ef.reranking.cross_encoder_reranker) is a ready reranker.
The **explore layer** (L5) is `ef`’s visualization heritage, kept as a
*secondary* “see the shape of the corpus” surface — search/RAG/indexing is the
primary one. It imports numpy-only; UMAP, HDBSCAN and `imbed` are lazy extras:
- [`project()`](_autosummary/ef.exploration.html.md#ef.exploration.project) — reduce embeddings to 2-D/3-D coordinates
(PCA → UMAP); [`cluster()`](_autosummary/ef.exploration.html.md#ef.exploration.cluster) — group them (k-means / HDBSCAN);
[`label_clusters()`](_autosummary/ef.exploration.html.md#ef.exploration.label_clusters) — name clusters with an LLM via `imbed`.
- [`explore()`](_autosummary/ef.exploration.html.md#ef.exploration.explore) — the orchestrator: project + cluster (+ label)
in one call, returning a structured, JSON-friendly
[`ExploreResult`](_autosummary/ef.exploration.html.md#ef.exploration.ExploreResult) (`ids` / `coords` / `labels`).
The **service layer** bridges `ef`’s stateful objects to a stateless
transport — hand its methods to `qh.mk_app()` to serve `ef` over HTTP:
- [`EfService`](_autosummary/ef.service.html.md#ef.service.EfService) — a facade holding a `{corpus_id: SourceManager}`
handle registry: [`create_corpus()`](_autosummary/ef.service.html.md#ef.service.EfService.create_corpus) indexes a corpus,
[`search()`](_autosummary/ef.service.html.md#ef.service.EfService.search) / [`retrieve()`](_autosummary/ef.service.html.md#ef.service.EfService.retrieve)
> query it by id. [`CorpusInfo`](_autosummary/ef.service.html.md#ef.service.CorpusInfo) is its JSON-friendly summary.
Example — wrap a plain function and embed two strings:
```pycon
>>> import numpy as np
>>> from ef import as_embedder
>>> embedder = as_embedder(
... lambda texts: np.array([[len(t)] for t in texts]), model_id='len@1'
... )
>>> embedder(['hello', 'hi']).ravel().tolist()
[5.0, 2.0]
```
### Functions
| [`ready_handle`](_autosummary/ef.html.md#ef.ready_handle)(array) | Wrap an already-computed array in a finished [`BatchHandle`](_autosummary/ef.html.md#ef.BatchHandle). |
|------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [`cache_key`](_autosummary/ef.html.md#ef.cache_key)(embedder, text[, input_type]) | Deterministic cache key for one `(embedder, text, input_type)` triple. |
| [`embed_length_sorted`](_autosummary/ef.html.md#ef.embed_length_sorted)(texts, encode, \*[, ...]) | Encode `texts` in length-sorted batches; return vectors in caller order. |
| [`as_embedder`](_autosummary/ef.html.md#ef.as_embedder)(x, \*\*kwargs) | Normalize `x` into an [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) — the DI seam. |
| [`openai_embedder`](_autosummary/ef.html.md#ef.openai_embedder)([model, dim, api_key, ...]) | Build an OpenAI-embeddings [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder). |
| [`sentence_transformers_embedder`](_autosummary/ef.html.md#ef.sentence_transformers_embedder)([model_name, ...]) | Build a local sentence-transformers [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder). |
| [`http_embedder`](_autosummary/ef.html.md#ef.http_embedder)(url, \*[, model_id, dim, ...]) | Build an [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) over a remote HTTP service. |
| [`cohere_embedder`](_autosummary/ef.html.md#ef.cohere_embedder)([model, dim, api_key, ...]) | Build a Cohere-embeddings [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder). |
| [`voyage_embedder`](_autosummary/ef.html.md#ef.voyage_embedder)([model, dim, api_key, ...]) | Build a Voyage AI [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder). |
| [`gemini_embedder`](_autosummary/ef.html.md#ef.gemini_embedder)([model, dim, api_key, ...]) | Build a Google Gemini [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder). |
| [`segment_id`](_autosummary/ef.html.md#ef.segment_id)(text[, metadata]) | Content-derived id for a segment — `sha256` of its normalized content. |
| [`make_segment`](_autosummary/ef.html.md#ef.make_segment)(text, \*[, id, parent_id, ...]) | Build a [`Segment`](_autosummary/ef.html.md#ef.Segment), deriving a content-based `id` if none given. |
| [`as_segment`](_autosummary/ef.html.md#ef.as_segment)(x, \*\*defaults) | Coerce `x` into a [`Segment`](_autosummary/ef.html.md#ef.Segment) — accepts a `str`, `Mapping` or record. |
| [`segment_record`](_autosummary/ef.html.md#ef.segment_record)(x) | Coerce `x` (`str` / `Mapping` / [`SegmentRecord`](_autosummary/ef.html.md#ef.SegmentRecord)) into a record. |
| [`approx_token_count`](_autosummary/ef.html.md#ef.approx_token_count)(text) | Rough token count — the common `~4 characters per token` heuristic. |
| [`line_segmenter`](_autosummary/ef.html.md#ef.line_segmenter)(doc, /) | Segment a document into its non-empty lines. |
| [`with_overlap`](_autosummary/ef.html.md#ef.with_overlap)(segmenter, n_chars, /, \*[, ...]) | Wrap `segmenter` so consecutive segments overlap by `n_chars` characters. |
| [`hierarchical`](_autosummary/ef.html.md#ef.hierarchical)(segmenters, /) | Compose segmenters into levels — each segments the previous level's output. |
| [`materialise`](_autosummary/ef.html.md#ef.materialise)(x) | Force a streaming segmenter (or segment stream) to a concrete list. |
| [`as_segmenter`](_autosummary/ef.html.md#ef.as_segmenter)([x]) | Normalize `x` into a [`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter) — the DI seam. |
| [`imbed_segmenter`](_autosummary/ef.html.md#ef.imbed_segmenter)([name]) | Build a [`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter) from `imbed`'s segmenter registry. |
| [`content_hash`](_autosummary/ef.html.md#ef.content_hash)(source, \*[, content_keys]) | Content hash of a source document — `sha256` of its normalized content. |
| [`as_corpus`](_autosummary/ef.html.md#ef.as_corpus)([x]) | Normalize `x` into a corpus (`MutableMapping[source_id, Source]`). |
| [`artifact_id`](_autosummary/ef.html.md#ef.artifact_id)(spec) | Content-addressed id of `spec` — `sha256` over its four fields. |
| [`producer_spec`](_autosummary/ef.html.md#ef.producer_spec)(op, \*inputs, op_version, \*\*params) | Build a [`ProducerSpec`](_autosummary/ef.html.md#ef.ProducerSpec) ergonomically — inputs positional, params keyword. |
| [`config_id`](_autosummary/ef.html.md#ef.config_id)(spec) | Content-addressed id of a [`PipelineSpec`](_autosummary/ef.html.md#ef.PipelineSpec) — `sha256` over its steps. |
| [`full_kwargs`](_autosummary/ef.html.md#ef.full_kwargs)(op[, args, kwargs]) | Normalize a call into a canonical, fully-named, defaults-filled kwargs dict. |
| [`step_params`](_autosummary/ef.html.md#ef.step_params)(op[, raw_params, n_inputs]) | The canonical *keyword* params of an op step — its inputs dropped. |
| [`ingest`](_autosummary/ef.html.md#ef.ingest)(sources, \*[, segmenter, embedder, ...]) | Index `sources` and return a ready-to-search [`SearchableCorpus`](_autosummary/ef.html.md#ef.SearchableCorpus) — one call. |
| [`hits_to_segments`](_autosummary/ef.html.md#ef.hits_to_segments)(hits) | Project [`SearchHit`](_autosummary/ef.html.md#ef.SearchHit)s to plain [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)s. |
| [`diagnose`](_autosummary/ef.html.md#ef.diagnose)(corpus, collection, config) | Compute the four staleness conditions of `collection` against `corpus`. |
| [`plan_refresh`](_autosummary/ef.html.md#ef.plan_refresh)(report, \*[, mode, scope]) | Turn a [`StalenessReport`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.StalenessReport) into a [`RefreshPlan`](_autosummary/ef.html.md#ef.RefreshPlan). |
| [`refresh_on_change`](_autosummary/ef.html.md#ef.refresh_on_change)(manager) | Build a [`ChangeEvent`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeEvent) handler that auto-refreshes `manager`. |
| [`evaluate_retrieval`](_autosummary/ef.html.md#ef.evaluate_retrieval)(retriever, qrels, queries, \*) | Score a retriever against a BEIR-shaped evaluation set. |
| [`evaluate_rag`](_autosummary/ef.html.md#ef.evaluate_rag)(samples, \*[, metrics]) | Score RAG samples with deterministic, reference-based lexical metrics. |
| [`read_beir`](_autosummary/ef.html.md#ef.read_beir)(directory, \*[, qrels]) | Load a BEIR-format dataset from disk into the in-memory evaluation triple. |
| [`write_beir`](_autosummary/ef.html.md#ef.write_beir)(directory, corpus, queries, qrels) | Write an evaluation triple to disk in BEIR format — the inverse of [`read_beir()`](_autosummary/ef.html.md#ef.read_beir). |
| [`as_ragas_dataset`](_autosummary/ef.html.md#ef.as_ragas_dataset)(samples) | Convert [`RagSample`](_autosummary/ef.html.md#ef.RagSample)s into a Ragas `EvaluationDataset`. |
| [`ndcg_at_k`](_autosummary/ef.html.md#ef.ndcg_at_k)(ranked, relevant, k) | Normalized DCG at cutoff `k` — `ef`'s **primary** retrieval metric. |
| [`dcg_at_k`](_autosummary/ef.html.md#ef.dcg_at_k)(ranked, relevant, k) | Discounted cumulative gain of `ranked` at cutoff `k`. |
| [`recall_at_k`](_autosummary/ef.html.md#ef.recall_at_k)(ranked, relevant, k) | Fraction of the relevant docs that appear in the top `k` results. |
| [`precision_at_k`](_autosummary/ef.html.md#ef.precision_at_k)(ranked, relevant, k) | Fraction of the top `k` results that are relevant. |
| [`reciprocal_rank`](_autosummary/ef.html.md#ef.reciprocal_rank)(ranked, relevant[, k]) | Reciprocal of the rank of the first relevant result (`0` if none). |
| [`average_precision`](_autosummary/ef.html.md#ef.average_precision)(ranked, relevant[, k]) | Average precision of `ranked` — the per-query term of MAP. |
| [`exact_match`](_autosummary/ef.html.md#ef.exact_match)(response, reference) | `1.0` if `response` equals `reference` after normalization, else `0.0`. |
| [`token_f1`](_autosummary/ef.html.md#ef.token_f1)(response, reference) | Token-overlap F1 of `response` against `reference` (the SQuAD F1). |
| [`context_recall`](_autosummary/ef.html.md#ef.context_recall)(reference, retrieved_contexts) | Lexical recall of the `reference` tokens by the retrieved contexts. |
| [`context_precision`](_autosummary/ef.html.md#ef.context_precision)(reference, retrieved_contexts) | Fraction of the retrieved contexts that lexically overlap the `reference`. |
| [`rerank`](_autosummary/ef.html.md#ef.rerank)(query, segments, reranker, \*[, limit]) | Re-score `segments` against `query` with `reranker` and re-order them. |
| [`with_reranker`](_autosummary/ef.html.md#ef.with_reranker)(retriever, reranker, \*[, fetch_k]) | Decorate a retrieve-style callable with a two-stage reranking pass. |
| [`cross_encoder_reranker`](_autosummary/ef.html.md#ef.cross_encoder_reranker)([model_name, device, ...]) | Build a [`Reranker`](_autosummary/ef.html.md#ef.Reranker) backed by a sentence-transformers `CrossEncoder`. |
| [`project`](_autosummary/ef.html.md#ef.project)(data, \*[, dims, method, embedder, ...]) | Project a corpus's embeddings to `dims` coordinates for visualization. |
| [`cluster`](_autosummary/ef.html.md#ef.cluster)(data, \*[, method, n_clusters, ...]) | Cluster a corpus's embeddings (use cases §G2). |
| [`label_clusters`](_autosummary/ef.html.md#ef.label_clusters)(segments, labels, \*[, ...]) | Name each cluster with a short LLM-generated title (use cases §G3). |
| [`explore`](_autosummary/ef.html.md#ef.explore)(data, \*[, dims, projection_method, ...]) | Project, cluster and (optionally) label a corpus in one call. |
### Classes
| [`Embedder`](_autosummary/ef.html.md#ef.Embedder)(\*args, \*\*kwargs) | Structural type for an embedder: a batch callable plus metadata. |
|-----------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------|
| [`BaseEmbedder`](_autosummary/ef.html.md#ef.BaseEmbedder)() | Implementation base for embedders (optional — [`Embedder`](_autosummary/ef.html.md#ef.Embedder) is structural). |
| [`BatchHandle`](_autosummary/ef.html.md#ef.BatchHandle)(poll, result[, cancel]) | A poll/result handle over an in-flight (or already-finished) embed job. |
| [`FunctionEmbedder`](_autosummary/ef.html.md#ef.FunctionEmbedder)(func, \*[, model_id, dim, ...]) | Promote a bare `texts -> vectors` callable to a full [`Embedder`](_autosummary/ef.html.md#ef.Embedder). |
| [`HashingEmbedder`](_autosummary/ef.html.md#ef.HashingEmbedder)(\*[, dim, ngram_range, ...]) | A dependency-free embedder — the feature-hashing trick, numpy only. |
| [`CachedEmbedder`](_autosummary/ef.html.md#ef.CachedEmbedder)(inner, store, \*[, cache_queries]) | Memoize embeddings in a `MutableMapping` store. |
| [`RetryingEmbedder`](_autosummary/ef.html.md#ef.RetryingEmbedder)(inner[, policy, sleep]) | Retry an inner embedder's calls on transient failures. |
| [`RetryPolicy`](_autosummary/ef.html.md#ef.RetryPolicy)([max_attempts, base_delay, ...]) | Exponential-backoff retry policy for [`RetryingEmbedder`](_autosummary/ef.html.md#ef.RetryingEmbedder). |
| [`MultiEmbedder`](_autosummary/ef.html.md#ef.MultiEmbedder)(routes, predicate, \*[, default]) | Route each text to one of several embedders by a predicate. |
| [`NormalizingEmbedder`](_autosummary/ef.html.md#ef.NormalizingEmbedder)(inner) | L2-normalize an inner embedder's output. |
| [`InstructionEmbedder`](_autosummary/ef.html.md#ef.InstructionEmbedder)(inner, \*[, ...]) | Prepend a per-`input_type` instruction before delegating to `inner`. |
| [`Segment`](_autosummary/ef.html.md#ef.Segment) | A piece of text carved from a source document — the interchange type. |
| [`SegmentRecord`](_autosummary/ef.html.md#ef.SegmentRecord)(text[, id, parent_id, start, ...]) | Frozen-dataclass view over a [`Segment`](_autosummary/ef.html.md#ef.Segment) — the convenience surface. |
| [`Segmenter`](_autosummary/ef.html.md#ef.Segmenter)(\*args, \*\*kwargs) | Structural type for a segmenter: `str | Mapping -> Iterable[Segment]`. |
| [`BatchedSegmenter`](_autosummary/ef.html.md#ef.BatchedSegmenter)(\*args, \*\*kwargs) | Optional trait: a segmenter that can process many documents at once. |
| [`BaseSegmenter`](_autosummary/ef.html.md#ef.BaseSegmenter)() | Implementation base for segmenters (optional — [`Segmenter`](_autosummary/ef.html.md#ef.Segmenter) is structural). |
| [`RecursiveCharacterSegmenter`](_autosummary/ef.html.md#ef.RecursiveCharacterSegmenter)(\*[, chunk_size, ...]) | The default segmenter — recursive character splitting. |
| [`FunctionSegmenter`](_autosummary/ef.html.md#ef.FunctionSegmenter)(func, \*[, pass_doc, ...]) | Promote a bare `text -> pieces` callable to a full [`Segmenter`](_autosummary/ef.html.md#ef.Segmenter). |
| [`ChangeDetectingCorpus`](_autosummary/ef.html.md#ef.ChangeDetectingCorpus)([corpus, on_change, ...]) | A corpus wrapper that content-hashes values and reports what changed. |
| [`ChangeEvent`](_autosummary/ef.html.md#ef.ChangeEvent)(source_id, kind[, old_hash, ...]) | A single detected change to a source in a corpus. |
| [`CorpusDiff`](_autosummary/ef.html.md#ef.CorpusDiff)([added, modified, deleted, unchanged]) | The result of comparing a corpus's content against known content hashes. |
| [`ArtifactGraph`](_autosummary/ef.html.md#ef.ArtifactGraph)(\*[, store, producers, edges, ops]) | A content-addressed producer graph — the corpus-indexing & refresh core. |
| [`ArtifactId`](_autosummary/ef.html.md#ef.ArtifactId) | |
| [`OpKey`](_autosummary/ef.html.md#ef.OpKey) | |
| [`ProducerSpec`](_autosummary/ef.html.md#ef.ProducerSpec)(op, op_version[, inputs, params]) | A declarative recipe for one produced artifact — the graph's node type. |
| [`TransformSpec`](_autosummary/ef.html.md#ef.TransformSpec)(op, op_version[, params]) | One step of a pipeline — a serializable recipe for an op application. |
| [`PipelineSpec`](_autosummary/ef.html.md#ef.PipelineSpec)(segment, embed) | A whole indexing pipeline as data — the segment→embed chain. |
| [`ConfigId`](_autosummary/ef.html.md#ef.ConfigId) | |
| [`SearchableCorpus`](_autosummary/ef.html.md#ef.SearchableCorpus)(collection, embedder, \*[, ...]) | A ready-to-search view over one indexed config — the thin facade. |
| [`SourceManager`](_autosummary/ef.html.md#ef.SourceManager)([corpus, segmenter, embedder, ...]) | The heavy search facade — multi-config corpus indexing over one corpus. |
| [`SearchHit`](_autosummary/ef.html.md#ef.SearchHit)(segment, score[, source_id]) | One ranked search result — a [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) and its score. |
| [`StalenessReport`](_autosummary/ef.html.md#ef.StalenessReport)(config[, orphan, missing, ...]) | The four staleness conditions of one config — what [`diagnose()`](_autosummary/ef.html.md#ef.diagnose) returns. |
| [`IndexedSource`](_autosummary/ef.html.md#ef.IndexedSource)(source_id, doc_ids, ...) | The provenance a `vd` collection records for one source document. |
| [`RefreshReport`](_autosummary/ef.html.md#ef.RefreshReport)(config, mode[, added, ...]) | The outcome of a refresh — what [`SourceManager.refresh()`](_autosummary/ef.html.md#ef.SourceManager.refresh) returns. |
| [`RefreshPlan`](_autosummary/ef.html.md#ef.RefreshPlan)([to_materialize, to_delete]) | What a refresh will do — [`plan_refresh()`](_autosummary/ef.html.md#ef.plan_refresh)'s output. |
| [`RetrievalEvalReport`](_autosummary/ef.html.md#ef.RetrievalEvalReport)(metrics, per_query, ...) | The outcome of [`evaluate_retrieval()`](_autosummary/ef.html.md#ef.evaluate_retrieval) — metrics over a query set. |
| [`RagEvalReport`](_autosummary/ef.html.md#ef.RagEvalReport)(metrics, coverage, n_samples) | The outcome of [`evaluate_rag()`](_autosummary/ef.html.md#ef.evaluate_rag) — lexical RAG metrics over a sample set. |
| [`RagSample`](_autosummary/ef.html.md#ef.RagSample) | One end-to-end RAG interaction — the unit [`evaluate_rag()`](_autosummary/ef.html.md#ef.evaluate_rag) scores. |
| [`Reranker`](_autosummary/ef.html.md#ef.Reranker)(\*args, \*\*kwargs) | A callable that re-scores `(query, segments)` — higher means more relevant. |
| [`ExploreResult`](_autosummary/ef.html.md#ef.ExploreResult) | A structured, JSON-friendly corpus-exploration result. |
| [`EfService`](_autosummary/ef.html.md#ef.EfService)(\*[, default_embedder]) | A registry-backed facade bridging stateful `ef` to a stateless transport. |
| [`CorpusInfo`](_autosummary/ef.html.md#ef.CorpusInfo) | A JSON-friendly summary of one registered corpus. |
### Exceptions
| [`EmbedderError`](_autosummary/ef.html.md#ef.EmbedderError) | Raised for embedder-specific failures (shape mismatch, batch job failure). |
|------------------------------------------------------------------|------------------------------------------------------------------------------|
### *class* ef.ArtifactGraph(, store=None, producers=None, edges=None, ops=None)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
A content-addressed producer graph — the corpus-indexing & refresh core.
The graph holds two kinds of node:
- **Leaves** — externally content-addressed inputs (an L0 source, hashed by
[`ef.corpus.content_hash()`](_autosummary/ef.corpus.html.md#ef.corpus.content_hash)). A leaf has a *value* but no recipe; it is
> supplied via [`put()`](_autosummary/ef.html.md#ef.ArtifactGraph.put).
- **Produced artifacts** — a node with a [`ProducerSpec`](_autosummary/ef.html.md#ef.ProducerSpec) recipe,
registered via [`add()`](_autosummary/ef.html.md#ef.ArtifactGraph.add). Its id *is* [`artifact_id()`](_autosummary/ef.html.md#ef.artifact_id) of its recipe.
and four small `MutableMapping` stores, every one injectable so the graph
can be backed by RAM (the default) or by a persistent `dol` store:
- `store` — the content-addressed cache: `artifact_id -> value`.
- `producers` — the recipe registry: `artifact_id -> ProducerSpec`.
- `edges` — the reverse index: `artifact_id -> (dependent ids, ...)`,
so the *downstream* cone of any node is one lookup away.
- `ops` — the op registry: `op_key -> callable`. The one store that is
*not* persisted (callables are not serializable) — repopulate it with
[`register_op()`](_autosummary/ef.html.md#ef.ArtifactGraph.register_op) each process.
The four contract operations:
- [`materialize()`](_autosummary/ef.html.md#ef.ArtifactGraph.materialize) — lazy *backward*: compute a node by recursively
materializing its inputs, then memoize.
- [`mark_stale()`](_autosummary/ef.html.md#ef.ArtifactGraph.mark_stale) — *forward*: drop the cached values of a node’s
produced downstream cone, keeping the recipes (they will rebuild).
- [`delete_cascade()`](_autosummary/ef.html.md#ef.ArtifactGraph.delete_cascade) — *forward*: remove a node and its whole downstream
cone entirely (values, recipes and edges).
- [`freshness()`](_autosummary/ef.html.md#ef.ArtifactGraph.freshness) — query a node’s state (`Freshness`).
plus the reachability queries [`descendants()`](_autosummary/ef.html.md#ef.ArtifactGraph.descendants) (the downstream cone) and
[`ancestors()`](_autosummary/ef.html.md#ef.ArtifactGraph.ancestors) (lineage — what an artifact was produced from).
* **Parameters:**
* **store** ([`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the content-addressed value cache. Defaults to a fresh `dict`.
* **producers** ([`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the recipe registry. Defaults to a fresh `dict`. If passed
already-populated (a resumed/persisted graph) and `edges` is left
empty, the reverse index is rebuilt from it on construction.
* **edges** ([`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis)]] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the reverse-dependency index. Defaults to a fresh `dict`; if
passed already-populated it is trusted as-is (not rebuilt).
* **ops** ([`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the op registry. Defaults to a fresh `dict`; [`register_op()`](_autosummary/ef.html.md#ef.ArtifactGraph.register_op)
adds to it.
```pycon
>>> graph = ArtifactGraph()
>>> _ = graph.register_op('join', lambda a, b, *, sep: f'{a}{sep}{b}')
>>> x = graph.put('x', 'hello')
>>> y = graph.put('y', 'world')
>>> both = graph.add(producer_spec('join', x, y, op_version='1', sep=', '))
>>> graph.materialize(both)
'hello, world'
>>> sorted(graph.descendants('x'))
[...]
>>> graph.delete_cascade('y') # the 'y' source is removed
frozenset(...)
>>> graph.freshness(both) # its consumer is gone too
'unknown'
```
#### add(spec)
Register a producer recipe; return its content-addressed id.
Idempotent: the id *is* the recipe, so adding an equal [`ProducerSpec`](_autosummary/ef.html.md#ef.ProducerSpec)
twice (or adding the shared upstream step of two branching configs)
registers one node and returns the same id — the basis of free config
branching. Inputs need not exist yet; the graph is declarative and
[`materialize()`](_autosummary/ef.html.md#ef.ArtifactGraph.materialize) resolves them lazily.
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
```pycon
>>> graph = ArtifactGraph()
>>> a = graph.add(producer_spec('op', 'leaf', op_version='1'))
>>> a == graph.add(producer_spec('op', 'leaf', op_version='1')) # idempotent
True
>>> len(graph.producers)
1
```
#### ancestors(key)
The lineage of `key` — every artifact it was transitively produced from.
Excludes `key` itself. Answers “what produced this?” — provenance /
lineage queries. The leaves of the returned set are the original
sources the artifact ultimately derives from.
* **Return type:**
[`frozenset`](https://docs.python.org/3/builtins/stdtypes.html#frozenset)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]
```pycon
>>> graph = ArtifactGraph()
>>> _ = graph.register_op('id', lambda x: x)
>>> a = graph.add(producer_spec('id', 'src', op_version='1'))
>>> b = graph.add(producer_spec('id', a, op_version='1'))
>>> graph.ancestors(b) == {'src', a}
True
```
#### delete_cascade(key)
Remove `key` and its whole downstream cone — *forward*; return what was removed.
Deletes every artifact in `{key} ∪ descendants(key)` *entirely* —
cached value, recipe and edges — and detaches the cone from any upstream
inputs that survive it. Unlike [`mark_stale()`](_autosummary/ef.html.md#ef.ArtifactGraph.mark_stale) this removes the
recipes too: use it when an artifact can never be rebuilt, the canonical
case being a **deleted source** (its segments, vectors and index entries
cascade away with it).
Returns the ids actually removed (those that had a value or a recipe).
* **Return type:**
[`frozenset`](https://docs.python.org/3/builtins/stdtypes.html#frozenset)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]
```pycon
>>> graph = ArtifactGraph()
>>> _ = graph.register_op('id', lambda x: x)
>>> graph.put('s', 'v')
's'
>>> a = graph.add(producer_spec('id', 's', op_version='1'))
>>> graph.delete_cascade('s') == {'s', a}
True
>>> 's' in graph, a in graph
(False, False)
```
#### descendants(key)
The downstream cone of `key` — every artifact transitively produced from it.
Excludes `key` itself. This is the set of artifacts a change to
`key` would invalidate; [`mark_stale()`](_autosummary/ef.html.md#ef.ArtifactGraph.mark_stale) and [`delete_cascade()`](_autosummary/ef.html.md#ef.ArtifactGraph.delete_cascade)
are built on it.
* **Return type:**
[`frozenset`](https://docs.python.org/3/builtins/stdtypes.html#frozenset)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]
```pycon
>>> graph = ArtifactGraph()
>>> _ = graph.register_op('id', lambda x: x)
>>> a = graph.add(producer_spec('id', 'src', op_version='1'))
>>> b = graph.add(producer_spec('id', a, op_version='1'))
>>> graph.descendants('src') == {a, b}
True
```
#### freshness(key)
Report the state of `key` — `Freshness`.
`"materialized"` if a value is cached; `"stale"` if a recipe exists
but no value (a [`materialize()`](_autosummary/ef.html.md#ef.ArtifactGraph.materialize) will build it); `"unknown"` if the
graph has neither.
Because the graph is content-addressed, a *cached* value is always
correct *for its id* — staleness here means “not yet computed”, never
“computed wrong”. The richer staleness conditions (orphan / missing /
misconfigured, computed against `vd` index metadata) belong to a later
diagnostics phase, not to the graph core.
* **Return type:**
[`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'materialized'`, `'stale'`, `'unknown'`]
```pycon
>>> graph = ArtifactGraph()
>>> _ = graph.register_op('id', lambda x: x)
>>> graph.put('s', 'v')
's'
>>> a = graph.add(producer_spec('id', 's', op_version='1'))
>>> graph.freshness('s'), graph.freshness(a), graph.freshness('?')
('materialized', 'stale', 'unknown')
```
#### mark_stale(key)
Invalidate `key`’s produced downstream cone — *forward*; return that cone.
Drops the **cached values** of every produced artifact in
`{key} ∪ descendants(key)`, *keeping the recipes* — so the next
[`materialize()`](_autosummary/ef.html.md#ef.ArtifactGraph.materialize) recomputes them. Leaf nodes (inputs, no recipe) are
never touched: they cannot be recomputed, so dropping their value would
be data loss — use [`delete_cascade()`](_autosummary/ef.html.md#ef.ArtifactGraph.delete_cascade) to remove an input.
This is the invalidation half of a refresh: when an op’s implementation
changes, or to force a recompute. (A *source* change is usually a
[`delete_cascade()`](_autosummary/ef.html.md#ef.ArtifactGraph.delete_cascade) of the old content hash — the new content yields
a new leaf id.) Returns the produced artifacts that are now stale.
* **Return type:**
[`frozenset`](https://docs.python.org/3/builtins/stdtypes.html#frozenset)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]
```pycon
>>> graph = ArtifactGraph()
>>> _ = graph.register_op('id', lambda x: x)
>>> graph.put('s', 'v')
's'
>>> a = graph.add(producer_spec('id', 's', op_version='1'))
>>> _ = graph.materialize(a)
>>> graph.mark_stale('s') == {a} # the leaf 's' itself is not "stale"
True
>>> graph.freshness(a)
'stale'
```
#### materialize(key)
Compute (or fetch the cached value of) the artifact `key` — lazy *backward*.
If `key` is already cached in `store` its value is returned
directly. Otherwise its recipe’s inputs are materialized recursively
(depth-first), the op is called `op(*input_values, **params)`, and the
result is memoized in `store` before being returned — so a second
[`materialize()`](_autosummary/ef.html.md#ef.ArtifactGraph.materialize) is a cache hit, and a shared upstream artifact is
computed exactly once however many configs depend on it.
* **Raises:**
* [**KeyError**](https://docs.python.org/3/builtins/exceptions.html#KeyError) – if `key` is neither cached nor has a producer recipe
(an unresolved leaf — supply it with [`put()`](_autosummary/ef.html.md#ef.ArtifactGraph.put)).
* [**LookupError**](https://docs.python.org/3/builtins/exceptions.html#LookupError) – if a recipe’s `op` is not in the `ops` registry.
* [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if the producer graph contains a cycle (a content-
addressed graph cannot — this guards a hand-corrupted registry).
* **Return type:**
[*Any*](https://docs.python.org/3/library/typing.html#typing.Any)
```pycon
>>> graph = ArtifactGraph()
>>> _ = graph.register_op('double', lambda x: x * 2)
>>> graph.put('n', 21)
'n'
>>> out = graph.add(producer_spec('double', 'n', op_version='1'))
>>> graph.materialize(out)
42
```
* **Return type:**
[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)
#### put(key, value)
Store a **leaf** `value` (an external input) under `key`; return `key`.
A leaf has a value but no recipe — typically an L0 source keyed by its
[`ef.corpus.content_hash()`](_autosummary/ef.corpus.html.md#ef.corpus.content_hash). Returns `key` so the call composes:
```default
src = graph.put(content_hash(text), text)
```
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
```pycon
>>> graph = ArtifactGraph()
>>> graph.put('doc-1', 'some text')
'doc-1'
>>> graph.freshness('doc-1')
'materialized'
```
#### register_op(name, fn)
Register the callable `fn` under the op key `name`; return `fn`.
[`materialize()`](_autosummary/ef.html.md#ef.ArtifactGraph.materialize) resolves a [`ProducerSpec`](_autosummary/ef.html.md#ef.ProducerSpec)’s string `op`
through this registry. The op is called `fn(*input_values, **params)`:
the materialized inputs positionally, the spec’s `params` as keywords.
* **Return type:**
[`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]
```pycon
>>> graph = ArtifactGraph()
>>> graph.register_op('upper', str.upper)
```
### ef.ArtifactId
alias of [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
### *class* ef.BaseEmbedder
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
Implementation base for embedders (optional — [`Embedder`](_autosummary/ef.html.md#ef.Embedder) is structural).
Subclasses set `model_id` / `dim` / `normalized` /
`honored_input_types` and implement `__call__`; they then satisfy
[`Embedder`](_autosummary/ef.html.md#ef.Embedder) structurally. In return they get:
- a default *synchronous* [`embed_batch()`](_autosummary/ef.html.md#ef.BaseEmbedder.embed_batch) (`ready_handle(self(...))`) —
override it only for a genuinely async provider Batch API;
- a readable `repr`.
#### embed_batch(texts, , input_type=None, \*\*backend)
Default: embed synchronously and return a finished handle.
* **Return type:**
[`BatchHandle`](_autosummary/ef.embedders.html.md#ef.embedders.BatchHandle)
### *class* ef.BaseSegmenter
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
Implementation base for segmenters (optional — [`Segmenter`](_autosummary/ef.html.md#ef.Segmenter) is structural).
A subclass implements `__call__`; in return it gets a default
[`batch()`](_autosummary/ef.html.md#ef.BaseSegmenter.batch) (one-document-at-a-time — override it for a genuinely batched
backend) and a readable `repr`. Subclassing is purely a convenience;
satisfying [`Segmenter`](_autosummary/ef.html.md#ef.Segmenter) structurally is what matters.
#### batch(docs,)
Default: segment each document independently.
* **Return type:**
[`Iterator`](https://docs.python.org/3/library/typing.html#typing.Iterator)[[`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]]
### *class* ef.BatchHandle(poll, result, cancel=>)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
A poll/result handle over an in-flight (or already-finished) embed job.
Batch-and-poll is just a special case of async. A synchronous backend
returns a *ready* handle (via [`ready_handle()`](_autosummary/ef.html.md#ef.ready_handle)); a provider Batch API
returns a genuinely pending one. Either way the caller sees one shape.
#### poll
`() -> BatchStatus` — non-blocking status check.
#### result
`() -> np.ndarray` — blocks until done, returns `(n, dim)`.
#### cancel
`() -> None` — best-effort cancellation.
### *class* ef.BatchedSegmenter(\*args, \*\*kwargs)
Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol)
Optional trait: a segmenter that can process many documents at once.
Plain [`Segmenter`](_autosummary/ef.html.md#ef.Segmenter) handles one document per call. A segmenter that can
amortize per-call cost across documents (a shared model load, a vectorized
pass) also offers `batch`. [`BaseSegmenter`](_autosummary/ef.html.md#ef.BaseSegmenter) supplies a correct — if
un-optimized — default, so every [`BaseSegmenter`](_autosummary/ef.html.md#ef.BaseSegmenter) subclass is already
a [`BatchedSegmenter`](_autosummary/ef.html.md#ef.BatchedSegmenter).
#### batch(docs,)
Segment each document in `docs`, yielding one segment-stream each.
* **Return type:**
[`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]]
### *class* ef.CachedEmbedder(inner, store, , cache_queries=False)
Bases: [`BaseEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.BaseEmbedder)
Memoize embeddings in a `MutableMapping` store.
On each call, texts already in the store are served from it; only the misses
are forwarded to the inner embedder (in a single batch) and then written
back. The store can be any `dol`-style mapping — an in-RAM `dict`, a
directory of `.npy` files, a key-value DB.
Document embeddings are cached unconditionally; *query* embeddings are
skipped by default (they are rarely repeated) — pass `cache_queries=True`
to cache them too.
* **Parameters:**
* **inner** ([`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)) – The embedder to memoize.
* **store** ([`MutableMapping`](https://docs.python.org/3/library/typing.html#typing.MutableMapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), `ndarray`]) – A `MutableMapping[str, ndarray]` keyed by [`cache_key()`](_autosummary/ef.html.md#ef.cache_key).
* **cache_queries** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to also cache `input_type="query"` calls.
#### *property* honored_input_types *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['query', 'document', 'classification', 'clustering'], ...]*
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple.
If iterable is specified the tuple is initialized from iterable’s items.
If the argument is a tuple, the return value is the same object.
#### *property* normalized *: [bool](https://docs.python.org/3/builtins/functions.html#bool)*
bool(x) -> bool
Returns True when the argument x is true, False otherwise.
The builtins True and False are the only two instances of the class bool.
The class bool is a subclass of the class int, and cannot be subclassed.
### *class* ef.ChangeDetectingCorpus(corpus=None, , on_change=None, hashes=None, content_keys=None, fingerprint=None)
Bases: [`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)
A corpus wrapper that content-hashes values and reports what changed.
Wraps any inner corpus (a `MutableMapping[source_id, Source]`) and adds
change detection *without* altering its mapping behavior — reads, writes and
deletes pass straight through to the inner store. It is to a corpus what
[`CachedEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.CachedEmbedder) is to an embedder: a
transparent, composable layer.
Changes are noticed two ways:
- **Through the wrapper.** A `corpus[id] = source` or `del corpus[id]`
that actually changes content fires the injected `on_change` callback
with a [`ChangeEvent`](_autosummary/ef.html.md#ef.ChangeEvent). Writing *identical* content is a no-op (no
event) — the basis of idempotent ingestion.
- **Out of band.** When the backing store is edited directly (a file
changed on disk, an S3 object replaced), [`scan()`](_autosummary/ef.html.md#ef.ChangeDetectingCorpus.scan) re-reads every
source, compares hashes, fires events for what differs and returns a
[`CorpusDiff`](_autosummary/ef.html.md#ef.CorpusDiff). [`diff()`](_autosummary/ef.html.md#ef.ChangeDetectingCorpus.diff) runs the same comparison with no side
> effects.
The `on_change` callback is the seam the downstream `ArtifactGraph`
plugs into: a `"modified"` / `"deleted"` event is exactly the trigger
for a cascade invalidation.
On construction a *baseline* is established: the current content hashes of
the inner corpus are recorded silently (no events). Pass an already-populated
`hashes` registry to instead resume from a persisted baseline — then call
[`scan()`](_autosummary/ef.html.md#ef.ChangeDetectingCorpus.scan) to reconcile any drift.
* **Parameters:**
* **corpus** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the inner corpus to wrap. Coerced via [`as_corpus()`](_autosummary/ef.html.md#ef.as_corpus), so a
directory path / `dict` / iterable of sources / `None` all work.
* **on_change** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`ChangeEvent`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeEvent)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – called with each [`ChangeEvent`](_autosummary/ef.html.md#ef.ChangeEvent). Defaults to a no-op.
It should be cheap and not raise; an exception propagates to the
caller *after* the corpus and registry are already consistent.
* **hashes** ([`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the hash registry — a `MutableMapping[source_id, str]` of the
last-known content hash per source. Injectable so it can be
persisted (e.g. a `dol` store); defaults to an in-RAM `dict`.
If passed already-populated it is trusted as the baseline.
* **content_keys** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – forwarded to [`content_hash()`](_autosummary/ef.html.md#ef.content_hash) — metadata keys that
participate in the hash.
* **fingerprint** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`Hashable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Hashable)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – an optional cheap `source_id -> Hashable` callback (an
mtime / etag prefilter). When given, [`scan()`](_autosummary/ef.html.md#ef.ChangeDetectingCorpus.scan) / [`diff()`](_autosummary/ef.html.md#ef.ChangeDetectingCorpus.diff)
skip re-hashing a source whose fingerprint is unchanged. The
content hash always remains the source of truth — the fingerprint
is only a speed optimization, never trusted to *declare* a change.
```pycon
>>> backing = {'a': 'one', 'b': 'two'}
>>> corpus = ChangeDetectingCorpus(backing) # baseline adopts 'a' and 'b'
>>> backing['a'] = 'ONE' # edited behind the wrapper's back
>>> backing['c'] = 'three'
>>> del backing['b']
>>> d = corpus.scan()
>>> (d.added, d.modified, d.deleted)
(('c',), ('a',), ('b',))
```
#### diff()
Compare current corpus content against the registry — **no side effects**.
Detects out-of-band edits (changes made to the backing store directly)
without firing events or updating the registry. [`scan()`](_autosummary/ef.html.md#ef.ChangeDetectingCorpus.scan) is the
effectful counterpart.
* **Return type:**
[`CorpusDiff`](_autosummary/ef.corpus.html.md#ef.corpus.CorpusDiff)
#### scan()
Reconcile the registry with the corpus, fire events, return the diff.
Use this to pick up **out-of-band** edits — changes made to the backing
store directly rather than through this wrapper. Every detected change
fires a [`ChangeEvent`](_autosummary/ef.html.md#ef.ChangeEvent) and the registry (and fingerprints) are
advanced to the new state, so a second [`scan()`](_autosummary/ef.html.md#ef.ChangeDetectingCorpus.scan) with nothing
further changed is a no-op. [`diff()`](_autosummary/ef.html.md#ef.ChangeDetectingCorpus.diff) is the side-effect-free version.
* **Return type:**
[`CorpusDiff`](_autosummary/ef.corpus.html.md#ef.corpus.CorpusDiff)
### *class* ef.ChangeEvent(source_id, kind, old_hash=None, new_hash=None)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
A single detected change to a source in a corpus.
Emitted by [`ChangeDetectingCorpus`](_autosummary/ef.html.md#ef.ChangeDetectingCorpus) to its `on_change` callback.
`old_hash` is `None` for an `"added"` event, `new_hash` is `None`
for a `"deleted"` one.
#### source_id
the key of the source that changed.
#### kind
`"added"`, `"modified"` or `"deleted"`.
#### old_hash
the source’s previous content hash, if any.
#### new_hash
the source’s new content hash, if any.
### ef.ConfigId
alias of [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
### *class* ef.CorpusDiff(added=(), modified=(), deleted=(), unchanged=())
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
The result of comparing a corpus’s content against known content hashes.
The four fields partition the corpus’s source ids; each is a sorted tuple,
so a [`CorpusDiff`](_autosummary/ef.html.md#ef.CorpusDiff) is deterministic and JSON-friendly. Truthiness
reports whether *anything* changed.
```pycon
>>> d = CorpusDiff(added=('b',), modified=('a',))
>>> bool(d), d.changed
(True, ('b', 'a'))
>>> bool(CorpusDiff(unchanged=('a', 'b')))
False
```
#### *property* changed *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), ...]*
The source ids that are new or modified — the set to (re-)materialize.
### *class* ef.CorpusInfo
Bases: [`TypedDict`](https://docs.python.org/3/library/typing.html#typing.TypedDict)
A JSON-friendly summary of one registered corpus.
The return shape shared by [`EfService.create_corpus()`](_autosummary/ef.html.md#ef.EfService.create_corpus),
[`EfService.corpus_info()`](_autosummary/ef.html.md#ef.EfService.corpus_info) and (as a list) [`EfService.list_corpora()`](_autosummary/ef.html.md#ef.EfService.list_corpora)
— a plain `dict` an HTTP client and `qh`’s schema both understand.
Keys:
: corpus_id: the registry handle — how every other method addresses it.
n_sources: the number of source documents in the corpus.
n_segments: the number of indexed segments (sources cut by the segmenter).
embedder: the embedder’s `model_id` (e.g. `"hashing:v1@512"`).
dim: the embedding dimensionality.
config_id: the content hash of the segmenter+embedder pipeline.
### *class* ef.EfService(, default_embedder=None)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
A registry-backed facade bridging stateful `ef` to a stateless transport.
Construct one `EfService` per process, register corpora with
[`create_corpus()`](_autosummary/ef.html.md#ef.EfService.create_corpus), then query them by `corpus_id`. Hand the seven
bound methods to `qh.mk_app()` to expose them over HTTP — see the module
docstring.
The handle registry is private instance state — never a module global.
Corpora are isolated: each [`create_corpus()`](_autosummary/ef.html.md#ef.EfService.create_corpus) builds its own
[`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager) over its own in-memory `vd`
backend, so one corpus’s vectors never leak into another’s search.
`default_embedder` sets the embedder [`create_corpus()`](_autosummary/ef.html.md#ef.EfService.create_corpus) resolves when
its own `embedder` argument is `None` — the per-instance hook a host
(e.g. `app_ef`) uses to pick a policy default without passing
`embedder=` on every call. It defaults to
[`DEFAULT_EMBEDDER`](_autosummary/ef.source_manager.html.md#ef.source_manager.DEFAULT_EMBEDDER) (`"hashing"`), so a bare
`EfService()` stays dependency-free and offline.
```pycon
>>> service = EfService()
>>> len(service)
0
>>> _ = service.create_corpus(['hello world'], corpus_id='greetings')
>>> 'greetings' in service
True
>>> len(service)
1
```
A host can swap the default embedder at construction (the string is resolved
lazily, per corpus, by the DI seam — constructing the service touches no
network):
```pycon
>>> EfService(default_embedder='openai:text-embedding-3-small')
```
#### corpus_info(corpus_id)
Return the [`CorpusInfo`](_autosummary/ef.html.md#ef.CorpusInfo) of a registered corpus.
* **Raises:**
[**KeyError**](https://docs.python.org/3/builtins/exceptions.html#KeyError) – if `corpus_id` is not registered.
* **Return type:**
[`CorpusInfo`](_autosummary/ef.service.html.md#ef.service.CorpusInfo)
#### create_corpus(sources, , embedder=None, segmenter=None, corpus_id=None, embedder_api_key=None)
Index `sources` into a new corpus, register it, return its [`CorpusInfo`](_autosummary/ef.html.md#ef.CorpusInfo).
* **Parameters:**
* **sources** ([`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – the corpus — a list of text documents.
* **embedder** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the embedder, as a string the DI seam resolves
([`as_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.as_embedder)) — `"hashing"`,
`"openai:text-embedding-3-small"`, `"cohere:..."`, an
`http(s)://` URL, …. `None` → the service’s
`default_embedder` (chosen at construction; itself
[`DEFAULT_EMBEDDER`](_autosummary/ef.source_manager.html.md#ef.source_manager.DEFAULT_EMBEDDER), the dependency-free
[`HashingEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.HashingEmbedder), unless a host overrode it).
* **segmenter** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the segmenter, as a string
([`as_segmenter()`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.as_segmenter)) — `None` → the
recursive-character default.
* **corpus_id** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the handle to register the corpus under; `None` → a
fresh random id. Reusing a live id is an error.
* **embedder_api_key** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – an optional API key for the embedder — the
bring-your-own-key seam. When given and `embedder` is a
hosted-API spec (`"openai:…"`, `"cohere:…"`, …), the key is
forwarded to that adapter, so the service itself never needs to
hold one. `None` → the adapter’s usual environment-variable
resolution. Ignored for key-less embedders (e.g. `"hashing"`).
The corpus’s embedder keeps the key for the life of the corpus,
so [`search()`](_autosummary/ef.html.md#ef.EfService.search) / [`retrieve()`](_autosummary/ef.html.md#ef.EfService.retrieve) / [`explore_corpus()`](_autosummary/ef.html.md#ef.EfService.explore_corpus)
need no key of their own.
* **Return type:**
[`CorpusInfo`](_autosummary/ef.service.html.md#ef.service.CorpusInfo)
* **Returns:**
the [`CorpusInfo`](_autosummary/ef.html.md#ef.CorpusInfo) of the freshly indexed corpus.
* **Raises:**
[**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if `corpus_id` is already registered.
#### delete_corpus(corpus_id)
Drop a corpus from the registry, releasing its index.
* **Raises:**
[**KeyError**](https://docs.python.org/3/builtins/exceptions.html#KeyError) – if `corpus_id` is not registered.
* **Return type:**
[`None`](https://docs.python.org/3/builtins/constants.html#None)
#### explore_corpus(corpus_id, , dims=2, projection_method='auto', cluster_method='kmeans', n_clusters=8, label=False)
Project & cluster a registered corpus — the corpus-map surface.
Runs [`ef.exploration.explore()`](_autosummary/ef.exploration.html.md#ef.exploration.explore) over the corpus: every indexed
segment is projected to `dims` coordinates and assigned a cluster,
returned as a row-aligned [`ExploreResult`](_autosummary/ef.exploration.html.md#ef.exploration.ExploreResult) (`ids` /
`coords` / `labels` / `cluster_titles`) — the JSON-friendly shape
an `app_ef` corpus map consumes.
* **Parameters:**
* **corpus_id** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the corpus to explore.
* **dims** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – projection target dimensionality — `2` or `3`.
* **projection_method** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – `"auto"` / `"umap"` / `"pca"`.
* **cluster_method** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – `"kmeans"` / `"hdbscan"`.
* **n_clusters** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – number of k-means clusters.
* **label** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – when `True`, also name each cluster with an LLM (needs the
`ef[imbed]` extra and a key); default `False`.
* **Raises:**
* [**KeyError**](https://docs.python.org/3/builtins/exceptions.html#KeyError) – if `corpus_id` is not registered.
* [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if the corpus has fewer than 2 indexed segments.
* **Return type:**
[`ExploreResult`](_autosummary/ef.exploration.html.md#ef.exploration.ExploreResult)
#### list_corpora()
Return the [`CorpusInfo`](_autosummary/ef.html.md#ef.CorpusInfo) of every registered corpus.
* **Return type:**
[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`CorpusInfo`](_autosummary/ef.service.html.md#ef.service.CorpusInfo)]
#### retrieve(corpus_id, query, , limit=10)
Retrieve ranked [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)s — the RAG-plug-in shape.
Like [`search()`](_autosummary/ef.html.md#ef.EfService.search), but returns plain segments in rank order (the
`score` dropped, the `source_id` folded into `metadata["source"]`)
— clean context to hand to an external RAG/agent framework. `ef`
returns context; it does not synthesize answers.
* **Raises:**
[**KeyError**](https://docs.python.org/3/builtins/exceptions.html#KeyError) – if `corpus_id` is not registered.
* **Return type:**
[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]
#### search(corpus_id, query, , limit=10)
Search a registered corpus — up to `limit` ranked [`SearchHit`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchHit)s.
Each hit carries the matched [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment), its
similarity `score` (higher = closer) and the `source_id` it was cut
from.
* **Raises:**
[**KeyError**](https://docs.python.org/3/builtins/exceptions.html#KeyError) – if `corpus_id` is not registered.
* **Return type:**
[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`SearchHit`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchHit)]
### *class* ef.Embedder(\*args, \*\*kwargs)
Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol)
Structural type for an embedder: a batch callable plus metadata.
An embedder maps `Iterable[str]` to an `ndarray` of shape `(n, dim)`.
The four metadata attributes are the single source of truth \*on the object
itself\* — no global registry, no config singleton.
#### model_id
Identity that bakes in everything affecting the vector —
`"openai:text-embedding-3-large@1024"` (`provider:model@dim`).
Works directly as a cache namespace.
#### dim
Output dimensionality. May be `None` until first call for
lazily-probed embedders (e.g. a bare callable of unknown width).
#### normalized
`True` iff `||v|| == 1` by construction.
#### honored_input_types
The `InputType` values this embedder acts on
(empty if it ignores the task hint).
#### embed_batch(texts, , input_type=None, \*\*backend)
Submit an embed job, returning a [`BatchHandle`](_autosummary/ef.html.md#ef.BatchHandle).
* **Return type:**
[`BatchHandle`](_autosummary/ef.embedders.html.md#ef.embedders.BatchHandle)
### *exception* ef.EmbedderError
Bases: [`Exception`](https://docs.python.org/3/builtins/exceptions.html#Exception)
Raised for embedder-specific failures (shape mismatch, batch job failure).
### *class* ef.ExploreResult
Bases: [`TypedDict`](https://docs.python.org/3/library/typing.html#typing.TypedDict)
A structured, JSON-friendly corpus-exploration result.
What [`explore()`](_autosummary/ef.html.md#ef.explore) returns and [`ef.service.EfService.explore_corpus()`](_autosummary/ef.service.html.md#ef.service.EfService.explore_corpus)
serves. Every list is **row-aligned**: `ids[i]`, `coords[i]` and
`labels[i]` all describe the same item.
Keys:
: ids: the per-item identifiers — a corpus’s keys, or positional `"0"`,
: `"1"`, … for an id-less vector matrix.
coords: the projected coordinates — one `[x, y]` (or `[x, y, z]`)
: row per item.
labels: the cluster id of each item (HDBSCAN marks noise `-1`).
cluster_titles: `{cluster_id: title}` — empty unless [`explore()`](_autosummary/ef.html.md#ef.explore)
> was called with `label=True`.
### *class* ef.FunctionEmbedder(func, , model_id=None, dim=None, normalized=False, honored_input_types=(), pass_backend=True)
Bases: [`BaseEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.BaseEmbedder)
Promote a bare `texts -> vectors` callable to a full [`Embedder`](_autosummary/ef.html.md#ef.Embedder).
This is the bridge for embedder *functions* that carry no metadata — most
importantly `imbed`’s registered embedders. The wrapped callable is asked
only for `func(texts)` (it knows nothing of `input_type`); the wrapper
advertises `honored_input_types=()` accordingly.
* **Parameters:**
* **func** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – A callable mapping a list of strings to an array-like of shape
`(n, dim)` (a numpy array or a nested sequence — both accepted).
* **model_id** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Vector-identity string. Defaults to `"function:"`.
* **dim** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Output width. If `None` it is inferred (and frozen) on the first
call.
* **normalized** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether `func` already returns unit vectors.
* **honored_input_types** ([`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'query'`, `'document'`, `'classification'`, `'clustering'`]]) – Task hints `func` honors (usually none).
* **pass_backend** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – If `True` (default), forward any `**backend` kwargs to
`func`; set `False` for callables with a strict `(texts,)`
signature.
```pycon
>>> import numpy as np
>>> e = FunctionEmbedder(lambda ts: np.ones((len(ts), 4)), model_id='ones@4')
>>> e(['a', 'b', 'c']).shape
(3, 4)
```
### *class* ef.FunctionSegmenter(func, \*, pass_doc=False, count_tokens=, tokenizer='char-approx')
Bases: [`BaseSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.BaseSegmenter)
Promote a bare `text -> pieces` callable to a full [`Segmenter`](_autosummary/ef.html.md#ef.Segmenter).
The bridge for segmenter *functions* that carry no segment schema — most
importantly `imbed`’s registered segmenters, which map text to an iterable
of plain strings. Each piece the callable yields is normalized:
- a `str` piece becomes a [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment); its character
offsets are recovered by locating it in the source text, and a token
count plus `tokenizer` metadata are attached;
- a `Mapping` / [`SegmentRecord`](_autosummary/ef.segments.html.md#ef.segments.SegmentRecord) piece is coerced with
[`as_segment()`](_autosummary/ef.segments.html.md#ef.segments.as_segment), gaining only an `index` / `parent_id`
> where it lacks them.
* **Parameters:**
* **func** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – The callable to wrap. By default it receives the document’s
*text*; with `pass_doc=True` it receives the whole document.
* **pass_doc** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Pass the raw document instead of its text.
* **count_tokens** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`int`](https://docs.python.org/3/builtins/functions.html#int)]]) – Token counter for `str` pieces (`None` disables the
`tokens` / `tokenizer` annotation).
* **tokenizer** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Name recorded as the `tokenizer` metadata key.
```pycon
>>> fs = FunctionSegmenter(lambda text: text.split(','))
>>> [s['text'] for s in fs('a,b,c')]
['a', 'b', 'c']
>>> s = list(fs('a,b,c'))[1]
>>> (s['start'], s['end'], s['index'])
(2, 3, 1)
```
### *class* ef.HashingEmbedder(, dim=512, ngram_range=(1, 2), sublinear_tf=True)
Bases: [`BaseEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.BaseEmbedder)
A dependency-free embedder — the feature-hashing trick, numpy only.
[`HashingEmbedder`](_autosummary/ef.html.md#ef.HashingEmbedder) is `ef`’s **zero-install default**: the embedder
[`ingest()`](_autosummary/ef.source_manager.html.md#ef.source_manager.ingest) resolves to when none is given, so the
headline `ingest([...]).search(query)` works on a bare `pip install ef`
— no torch, no model download, no network.
It is a *lexical* embedder, not a neural one. Each text is tokenized into
lowercased word n-grams; every token is hashed into one of `dim` buckets
with a sign (the hashing trick — no vocabulary, no fitting, fixed memory);
the signed, sublinear-tf-weighted bucket counts are L2-normalized. The
cosine similarity of two vectors then ranks by shared vocabulary — enough
for demos, doctests and offline tests, and a sane fallback in production.
For genuine semantic quality install a real embedder (`ef[sentence-transformers]`,
`ef[openai]`) and pass it explicitly.
Every parameter that changes a vector is baked into `model_id`, so a
[`HashingEmbedder`](_autosummary/ef.html.md#ef.HashingEmbedder)’s artifacts stay correctly content-addressed in the
[`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph).
* **Parameters:**
* **dim** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Output width. Wider → fewer hash collisions, larger vectors.
* **ngram_range** ([`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`int`](https://docs.python.org/3/builtins/functions.html#int), [`int`](https://docs.python.org/3/builtins/functions.html#int)]) – Inclusive `(min, max)` word-n-gram sizes. The default
`(1, 2)` mixes single words with adjacent pairs — a little phrase
sensitivity at no real cost.
* **sublinear_tf** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Weight a token by `1 + log(count)` rather than its raw
`count`, damping a word repeated many times in one text.
```pycon
>>> e = HashingEmbedder()
>>> e.model_id
'hashing:v1@512'
>>> import numpy as np
>>> v = e(['the quick brown fox'])
>>> v.shape
(1, 512)
>>> bool(np.isclose(np.linalg.norm(v[0]), 1.0)) # rows are L2-normalized
True
>>> a, b = e(['hello world']), e(['hello world'])
>>> round(float(a[0] @ b[0]), 5) # identical text → cosine similarity 1.0
1.0
>>> isinstance(e, Embedder)
True
```
### *class* ef.IndexedSource(source_id, doc_ids, source_hashes, config_hashes)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
The provenance a `vd` collection records for one source document.
A source is segmented into several documents; they all carry the same
`source_id` and *should* carry one `source_hash` / `config_hash` each.
`indexed_state()` groups a collection’s documents into one of these per
source — collecting the hashes as **sets** so a partially re-indexed source
(documents disagreeing on a hash) is detectable rather than papered over.
#### source_id
the corpus key of the source these documents were cut from.
#### doc_ids
the ids of every `vd` document belonging to the source.
#### source_hashes
the distinct `source_hash` values its documents record
— a single-element set for a cleanly indexed source.
#### config_hashes
likewise the distinct `config_hash` values.
### *class* ef.InstructionEmbedder(inner, , query_instruction=None, document_instruction=None, classification_instruction=None, clustering_instruction=None)
Bases: [`BaseEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.BaseEmbedder)
Prepend a per-`input_type` instruction before delegating to `inner`.
Instruction-tuned embedding models (E5-instruct, BGE, GTE, Qwen3-Embedding…)
are trained to read a task description in front of the text, typically on the
*query* side only, with documents embedded bare. That asymmetry is where the
gain comes from: the query says *what kind of relevance* is wanted (topic vs.
keyword), which a plain model cannot be told. This wrapper adds that to any
embedder: a text embedded with `input_type="query"` becomes
`query_instruction + text`, and so on per role. Roles without an instruction
(and calls with `input_type=None`) pass through unchanged.
**Only trust the prefix on a genuinely instruction-tuned model.** On a plain
model (e.g. `all-MiniLM-L6-v2`) the prefix is expected to be a near no-op or
noise: use that as the control arm of an experiment, not as an improvement.
The instruction strings are used verbatim, so include the model’s expected
separator (e.g. `"Instruct: ...\nQuery: "` for Qwen3/E5-instruct,
`"Represent this sentence for searching relevant passages: "` for BGE).
Metadata:
- `model_id` bakes in a digest of the instructions (they change the vector),
so [`CachedEmbedder`](_autosummary/ef.html.md#ef.CachedEmbedder) and content-addressed artifacts never mix vectors
made with different instructions.
- `honored_input_types` is the inner embedder’s plus every instructed role.
- `input_type` is still forwarded to `inner`; a backend that also acts on
it natively (Cohere, Voyage, Gemini, or a sentence-transformers adapter
built with `query_prompt=`) would then be conditioned twice, so use this
wrapper for backends that ignore the hint. For sentence-transformers models
prefer the adapter’s own `query_prompt=`.
* **Parameters:**
* **inner** ([`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)) – The embedder to condition.
* **query_instruction** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Prefix for `input_type="query"`.
* **document_instruction** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Prefix for `input_type="document"` (usually none).
* **classification_instruction** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Prefix for `input_type="classification"`.
* **clustering_instruction** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Prefix for `input_type="clustering"`.
```pycon
>>> import numpy as np
>>> from ef.embedders import FunctionEmbedder
>>> seen = []
>>> def toy(texts):
... seen.extend(texts)
... return np.ones((len(texts), 2))
>>> base = FunctionEmbedder(toy, model_id='toy@2')
>>> emb = InstructionEmbedder(base, query_instruction='Find packages about: ')
>>> _ = emb(['graphs'], input_type='query')
>>> _ = emb(['a DAG library'], input_type='document')
>>> seen
['Find packages about: graphs', 'a DAG library']
>>> emb.honored_input_types
('query',)
>>> emb.model_id.startswith('toy@2+instruct:')
True
```
#### embed_batch(texts, , input_type=None, \*\*backend)
Prefix, then use the inner embedder’s own (possibly async) batch API.
#### *property* honored_input_types *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['query', 'document', 'classification', 'clustering'], ...]*
The inner embedder’s honored roles plus every instructed role.
#### *property* model_id *: [str](https://docs.python.org/3/builtins/stdtypes.html#str)*
The inner `model_id` plus a digest of the instructions.
#### *property* normalized *: [bool](https://docs.python.org/3/builtins/functions.html#bool)*
bool(x) -> bool
Returns True when the argument x is true, False otherwise.
The builtins True and False are the only two instances of the class bool.
The class bool is a subclass of the class int, and cannot be subclassed.
### *class* ef.MultiEmbedder(routes, predicate, , default=None)
Bases: [`BaseEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.BaseEmbedder)
Route each text to one of several embedders by a predicate.
Useful for domain-specific models (e.g. a code embedder for code, a prose
embedder for prose). All routes must share the same `dim` — the output is
a single `(n, dim)` array, with each row produced by its text’s route.
* **Parameters:**
* **routes** ([`Mapping`](https://docs.python.org/3/library/typing.html#typing.Mapping)[[`Hashable`](https://docs.python.org/3/library/typing.html#typing.Hashable), [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)]) – `{route_key: Embedder}`.
* **predicate** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`Hashable`](https://docs.python.org/3/library/typing.html#typing.Hashable)]) – `text -> route_key` — picks a route per text.
* **default** ([`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Embedder for texts whose `route_key` is not in `routes`
(otherwise an unknown key raises [`EmbedderError`](_autosummary/ef.html.md#ef.EmbedderError)).
```pycon
>>> import numpy as np
>>> from ef.embedders import FunctionEmbedder
>>> short = FunctionEmbedder(lambda ts: np.zeros((len(ts), 2)), model_id='s@2')
>>> long = FunctionEmbedder(lambda ts: np.ones((len(ts), 2)), model_id='l@2')
>>> m = MultiEmbedder({'s': short, 'l': long},
... predicate=lambda t: 's' if len(t) < 4 else 'l')
>>> m(['hi', 'hello']).tolist()
[[0.0, 0.0], [1.0, 1.0]]
```
### *class* ef.NormalizingEmbedder(inner)
Bases: [`BaseEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.BaseEmbedder)
L2-normalize an inner embedder’s output.
Use this for MRL-truncated vectors (which lose unit norm when sliced to a
smaller dimension) or any backend that does not normalize by construction.
Reports `normalized=True`; [`cache_key()`](_autosummary/ef.html.md#ef.cache_key) keys on `normalized`, so a
normalized and an un-normalized view of the same model never collide in a
shared cache.
* **Parameters:**
**inner** ([`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)) – The embedder whose output to renormalize.
#### *property* honored_input_types *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['query', 'document', 'classification', 'clustering'], ...]*
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple.
If iterable is specified the tuple is initialized from iterable’s items.
If the argument is a tuple, the return value is the same object.
### ef.OpKey
alias of [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
### *class* ef.PipelineSpec(segment, embed)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
A whole indexing pipeline as data — the segment→embed chain.
A [`PipelineSpec`](_autosummary/ef.html.md#ef.PipelineSpec) is the declarative description of one *config*: a
[`TransformSpec`](_autosummary/ef.html.md#ef.TransformSpec) for the segmenter and one for the embedder. It is
fully serializable ([`as_dict()`](_autosummary/ef.html.md#ef.PipelineSpec.as_dict) / [`from_dict()`](_autosummary/ef.html.md#ef.PipelineSpec.from_dict)) and content-hashed
by [`config_id()`](_autosummary/ef.html.md#ef.config_id). Registering a second [`PipelineSpec`](_autosummary/ef.html.md#ef.PipelineSpec) that
differs only in its embed step shares the segment artifacts in the
[`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph) — that is config branching, and
the graph gives it for free.
The `segment``→``embed` ordering *is* the input wiring: the embed step
consumes the segment step’s output. (Phase 5 fixes this two-step shape; a
later phase may generalize it to an arbitrary step list.)
```pycon
>>> seg = TransformSpec(op='segment:r', op_version='1', params={})
>>> emb = TransformSpec(op='embed:m@8', op_version='1', params={'input_type': 'document'})
>>> spec = PipelineSpec(segment=seg, embed=emb)
>>> spec.embed.params['input_type']
'document'
>>> PipelineSpec.from_dict(spec.as_dict()) == spec
True
```
#### as_dict()
Return a plain-`dict` (JSON-serializable) form of the pipeline.
* **Return type:**
[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]
```pycon
>>> seg = TransformSpec(op='segment:r', op_version='1', params={})
>>> emb = TransformSpec(op='embed:m', op_version='1', params={})
>>> PipelineSpec(segment=seg, embed=emb).as_dict()['segment']['op']
'segment:r'
```
#### *classmethod* from_dict(d)
Rebuild a [`PipelineSpec`](_autosummary/ef.html.md#ef.PipelineSpec) from its [`as_dict()`](_autosummary/ef.html.md#ef.PipelineSpec.as_dict) form.
* **Return type:**
[`PipelineSpec`](_autosummary/ef.config.html.md#ef.config.PipelineSpec)
### *class* ef.ProducerSpec(op, op_version, inputs=(), params=)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
A declarative recipe for one produced artifact — the graph’s node type.
A [`ProducerSpec`](_autosummary/ef.html.md#ef.ProducerSpec) says: *this artifact is produced by op* `op` \*at
version\* `op_version`, *applied to the artifacts* `inputs` *(passed
positionally), with the keyword parameters* `params`. It does **not**
hold the producer callable — only the string `op` key — so a spec is
fully serializable ([`as_dict()`](_autosummary/ef.html.md#ef.ProducerSpec.as_dict) / [`from_dict()`](_autosummary/ef.html.md#ef.ProducerSpec.from_dict)) and the graph’s
`producers` registry can be persisted.
The recipe is the artifact’s *identity*: [`artifact_id()`](_autosummary/ef.html.md#ef.artifact_id) hashes exactly
these four fields. Treat an instance as immutable — it is frozen, and
mutating `params` in place would silently desynchronize it from its id.
#### op
the op key, resolved to a callable via the graph’s `ops` registry.
#### op_version
the op’s version string. It participates in the id, so a
behavior change to an op **must** be paired with a version bump —
reusing a version for changed behavior silently serves stale cached
values (the graph trusts the version; it cannot inspect the code).
#### inputs
the input artifact ids, in the order the op receives them
positionally. Order is significant — it is part of the id.
#### params
the op’s keyword arguments. Key order is *not* significant
(the id is computed over canonical, sorted-key JSON); pass
defaults-filled, fully-named params so that two semantically equal
calls hash equal.
```pycon
>>> spec = producer_spec('embed', 'seg-1', op_version='3', model='small')
>>> spec.op, spec.inputs, spec.op_version
('embed', ('seg-1',), '3')
>>> spec.params == {'model': 'small'}
True
```
#### as_dict()
Return a plain-`dict` form of the spec — JSON-serializable.
The inverse of [`from_dict()`](_autosummary/ef.html.md#ef.ProducerSpec.from_dict). Use it to persist a `producers`
registry to a `dol` JSON store.
* **Return type:**
[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]
```pycon
>>> producer_spec('embed', 'a', 'b', op_version='1', dim=256).as_dict()
{'op': 'embed', 'op_version': '1', 'inputs': ['a', 'b'], 'params': {'dim': 256}}
```
#### *classmethod* from_dict(d)
Rebuild a [`ProducerSpec`](_autosummary/ef.html.md#ef.ProducerSpec) from its [`as_dict()`](_autosummary/ef.html.md#ef.ProducerSpec.as_dict) form.
* **Return type:**
[`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec)
```pycon
>>> spec = producer_spec('embed', 'a', 'b', op_version='1', dim=256)
>>> ProducerSpec.from_dict(spec.as_dict()) == spec
True
```
### *class* ef.RagEvalReport(metrics, coverage, n_samples)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
The outcome of [`evaluate_rag()`](_autosummary/ef.html.md#ef.evaluate_rag) — lexical RAG metrics over a sample set.
`metrics` maps each metric to its mean over the samples it applied to;
`coverage` records how many samples that was (a metric needing a
`reference` skips samples without one). `n_samples` is the total scored.
```pycon
>>> report = RagEvalReport(
... metrics={'exact_match': 0.5}, coverage={'exact_match': 4}, n_samples=4)
>>> report.metrics['exact_match']
0.5
```
### *class* ef.RagSample
Bases: [`TypedDict`](https://docs.python.org/3/library/typing.html#typing.TypedDict)
One end-to-end RAG interaction — the unit [`evaluate_rag()`](_autosummary/ef.html.md#ef.evaluate_rag) scores.
The field names match Ragas’ `SingleTurnSample` exactly, so a
[`RagSample`](_autosummary/ef.html.md#ef.RagSample) is the interchange shape `app_ef` / `srag` / `raglab`
produce and [`as_ragas_dataset()`](_autosummary/ef.html.md#ef.as_ragas_dataset) forwards unchanged. `ef` itself fills
only `retrieved_contexts` (from [`retrieve()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.retrieve));
`response` is the *caller’s* LLM output — `ef` does not synthesize it.
Keys:
: user_input: the user’s question.
response: the generated answer (the caller’s LLM produced it).
retrieved_contexts: the context passages handed to the LLM — what
> [`retrieve()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.retrieve) returned, as text.
reference: the ground-truth answer, if known.
reference_contexts: the ground-truth context passages, if known.
### *class* ef.RecursiveCharacterSegmenter(\*, chunk_size=512, chunk_overlap=64, separators=('\\\\n\\\\n', '\\\\n', '. ', ', ', ' ', ''), count_tokens=, tokenizer='char-approx')
Bases: [`BaseSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.BaseSegmenter)
The default segmenter — recursive character splitting.
Splits a document on a descending ladder of separators (paragraph → line →
sentence → clause → word → character), merging the resulting pieces into
chunks of about `chunk_size` tokens with `chunk_overlap` tokens shared
between neighbours. This “good enough” default is, on realistic corpora,
within a point or two of far more elaborate methods — upgrade only if you
can measure the win.
Sizes are measured with `count_tokens`. The default counter is the
dependency-free [`approx_token_count()`](_autosummary/ef.html.md#ef.approx_token_count); inject a real tokenizer (and its
`tokenizer` name) when an exact count matters. Every emitted segment
records the tokenizer in its metadata and carries exact `start` / `end`
character offsets into the source.
* **Parameters:**
* **chunk_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Target chunk size, in `count_tokens` units.
* **chunk_overlap** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Tokens of overlap between consecutive chunks (must be
smaller than `chunk_size`).
* **separators** ([`Sequence`](https://docs.python.org/3/library/typing.html#typing.Sequence)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – The separator ladder, coarsest first; the last should be
`""` so an unsplittable run is still broken between characters.
* **count_tokens** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`int`](https://docs.python.org/3/builtins/functions.html#int)]) – Maps a string to a token count.
* **tokenizer** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Name recorded as the `tokenizer` metadata key.
```pycon
>>> seg = RecursiveCharacterSegmenter(chunk_size=3, chunk_overlap=1)
>>> [c['text'] for c in seg('one two three four')]
['one two three', 'three four']
```
### *class* ef.RefreshPlan(to_materialize=(), to_delete=())
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
What a refresh will do — [`plan_refresh()`](_autosummary/ef.html.md#ef.plan_refresh)’s output.
The two fields are disjoint in *intent* but may overlap in membership: a
stale source is in **both** — its old documents are deleted *and* it is
re-materialized. An orphan is only in `to_delete`; a brand-new source only
in `to_materialize`.
#### to_materialize
source ids to (re-)run through the pipeline.
#### to_delete
source ids whose currently-indexed documents are removed
(before re-materializing, for the ones that also reappear above).
### *class* ef.RefreshReport(config, mode, added=(), modified=(), deleted=(), unchanged=(), documents_written=0, documents_removed=0, artifacts_removed=0)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
The outcome of a refresh — what [`SourceManager.refresh()`](_autosummary/ef.html.md#ef.SourceManager.refresh) returns.
`added` / `modified` / `deleted` / `unchanged` partition the
source ids the refresh considered (mirroring [`CorpusDiff`](_autosummary/ef.corpus.html.md#ef.corpus.CorpusDiff)
field names): newly indexed, re-indexed (was stale/misconfigured), removed
(orphan), and already-fresh-left-alone. The three counts report the work
done. Truthiness reports whether anything changed.
```pycon
>>> r = RefreshReport(config='c0', mode='full', added=('a',), modified=('b',))
>>> bool(r), r.changed
(True, ('a', 'b'))
>>> bool(RefreshReport(config='c0', mode='full', unchanged=('a',)))
False
```
#### *property* changed *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), ...]*
The source ids that were added, modified or deleted.
### *class* ef.Reranker(\*args, \*\*kwargs)
Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol)
A callable that re-scores `(query, segments)` — higher means more relevant.
The structural contract a reranker satisfies: given the query and a list of
candidate [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)s, return \*\*one score per
segment\*\*, in the same order. [`rerank()`](_autosummary/ef.html.md#ef.rerank) sorts by these scores. It is a
`@runtime_checkable Protocol`, not a base class — a plain function of the
right shape already *is* a [`Reranker`](_autosummary/ef.html.md#ef.Reranker).
```pycon
>>> def by_length(query, segments):
... return [len(s['text']) for s in segments]
>>> isinstance(by_length, Reranker)
True
```
### *class* ef.RetrievalEvalReport(metrics, per_query, n_queries, k_values)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
The outcome of [`evaluate_retrieval()`](_autosummary/ef.html.md#ef.evaluate_retrieval) — metrics over a query set.
`metrics` maps a `"@"` key (e.g. `"ndcg@10"`) to the mean of
that metric over every scored query; `per_query` keeps the same keys per
query for drill-down. `n_queries` is how many queries were actually
scored (a query with no positive judgement is skipped).
```pycon
>>> report = RetrievalEvalReport(
... metrics={'ndcg@10': 0.83}, per_query={}, n_queries=12, k_values=(10,))
>>> report.primary
0.83
```
#### *property* primary *: [float](https://docs.python.org/3/builtins/functions.html#float) | [None](https://docs.python.org/3/builtins/constants.html#None)*
The headline number — mean [NDCG@10](mailto:NDCG@10), or `None` if it was not computed.
#### primary_at(k)
Mean NDCG at cutoff `k` — `None` if NDCG@\`\`k\`\` was not computed.
* **Return type:**
[`float`](https://docs.python.org/3/builtins/functions.html#float) | [`None`](https://docs.python.org/3/builtins/constants.html#None)
### *class* ef.RetryPolicy(max_attempts=4, base_delay=1.0, max_delay=30.0, jitter=0.3, retry_on=)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
Exponential-backoff retry policy for [`RetryingEmbedder`](_autosummary/ef.html.md#ef.RetryingEmbedder).
#### max_attempts
Total tries (the first call counts as attempt 1).
#### base_delay
Seconds before the first retry; doubles each attempt.
#### max_delay
Upper bound on any single backoff sleep.
#### jitter
Random fraction (`0..1`) of the delay added on top, to
de-correlate concurrent clients.
#### retry_on
Predicate deciding whether an exception is worth retrying.
* **Return type:**
[*bool*](https://docs.python.org/3/builtins/functions.html#bool)
#### retry_on()
Heuristic: retry transient failures (429, 5xx, connection/timeout).
Per-request 400-class errors are *not* retried — the same request would fail
identically. Recognizes the `status_code`/`status` attribute carried by
most HTTP-client SDK exceptions (incl. the OpenAI SDK).
* **Return type:**
[`bool`](https://docs.python.org/3/builtins/functions.html#bool)
### *class* ef.RetryingEmbedder(inner, policy=None, \*, sleep=)
Bases: [`BaseEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.BaseEmbedder)
Retry an inner embedder’s calls on transient failures.
* **Parameters:**
* **inner** ([`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)) – The embedder to guard.
* **policy** ([`RetryPolicy`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.RetryPolicy) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – A [`RetryPolicy`](_autosummary/ef.html.md#ef.RetryPolicy) (sensible defaults if omitted).
* **sleep** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`float`](https://docs.python.org/3/builtins/functions.html#float)], [`None`](https://docs.python.org/3/builtins/constants.html#None)]) – The sleep function — injectable so tests run instantly.
#### *property* honored_input_types *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['query', 'document', 'classification', 'clustering'], ...]*
Built-in immutable sequence.
If no argument is given, the constructor returns an empty tuple.
If iterable is specified the tuple is initialized from iterable’s items.
If the argument is a tuple, the return value is the same object.
#### *property* normalized *: [bool](https://docs.python.org/3/builtins/functions.html#bool)*
bool(x) -> bool
Returns True when the argument x is true, False otherwise.
The builtins True and False are the only two instances of the class bool.
The class bool is a subclass of the class int, and cannot be subclassed.
### *class* ef.SearchHit(segment, score, source_id=None)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
One ranked search result — a [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) and its score.
[`SearchableCorpus.search()`](_autosummary/ef.html.md#ef.SearchableCorpus.search) / [`SourceManager.search()`](_autosummary/ef.html.md#ef.SourceManager.search) return a
ranked `list` of these. Keeping [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) (the
interchange `TypedDict`) pure of a result-only `score` key, a
[`SearchHit`](_autosummary/ef.html.md#ef.SearchHit) *wraps* it alongside the similarity `score` and the
`source_id` of the document the segment was cut from.
#### segment
the matched segment — the canonical [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment).
#### score
the similarity score from the vector store (higher = closer).
#### source_id
the corpus key of the source document, if recorded.
```pycon
>>> hit = SearchHit(segment={'text': 'hi', 'id': 'x'}, score=0.9, source_id='doc-1')
>>> hit.segment['text'], hit.score, hit.source_id
('hi', 0.9, 'doc-1')
```
### *class* ef.SearchableCorpus(collection, embedder, , config=None)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
A ready-to-search view over one indexed config — the thin facade.
What [`ingest()`](_autosummary/ef.html.md#ef.ingest) returns and what [`SourceManager.searchable()`](_autosummary/ef.html.md#ef.SourceManager.searchable) hands
out: a small, read-mostly object that knows just enough to answer a query —
the `vd` collection holding the vectors and the [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)
that embeds the query. It does **not** re-index; mutation, refresh and
multi-config live on [`SourceManager`](_autosummary/ef.html.md#ef.SourceManager).
* **Parameters:**
* **collection** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the `vd` collection holding this config’s vectors.
* **embedder** ([`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)) – the embedder — used to embed queries with `input_type="query"`.
* **config** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the [`ConfigId`](_autosummary/ef.config.html.md#ef.config.ConfigId) of the indexed pipeline, if known.
```pycon
>>> # built for you by ingest(); see the module docstring for a full example.
```
#### retrieve(query, , limit=10, filter=None)
Retrieve ranked [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)s — the RAG-plug-in surface.
The name an external RAG/agent framework reaches for. Where
[`search()`](_autosummary/ef.html.md#ef.SearchableCorpus.search) returns scored [`SearchHit`](_autosummary/ef.html.md#ef.SearchHit)s (for inspecting the
ranking), `retrieve` returns **plain segments** in rank order — clean
context to hand to an LLM, with no `ef`-specific type to learn (design
notes §F5). `ef` returns the context; it does *not* synthesize answers.
See [`hits_to_segments()`](_autosummary/ef.html.md#ef.hits_to_segments) for how the score is dropped and the
`source_id` provenance is preserved in `metadata["source"]`.
* **Return type:**
[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]
```pycon
>>> # idx.retrieve('a query') -> [{'text': ..., 'id': ..., 'metadata': ...}, ...]
```
#### search(query, , limit=10, filter=None)
Search the indexed corpus — return up to `limit` ranked [`SearchHit`](_autosummary/ef.html.md#ef.SearchHit)s.
`query` is text (embedded with `input_type="query"`) or an already
computed query vector. `filter` is a `vd` metadata filter
(MongoDB-style) — see `vd.filters`.
* **Return type:**
[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`SearchHit`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchHit)]
### *class* ef.Segment
Bases: [`TypedDict`](https://docs.python.org/3/library/typing.html#typing.TypedDict)
A piece of text carved from a source document — the interchange type.
A `TypedDict`: a [`Segment`](_autosummary/ef.html.md#ef.Segment) *is* a plain `dict`, which keeps it
cheap to create and stream. `total=False` because most keys are optional;
by convention `text` is always present and `id` is always set (derived
from `text` if not supplied).
Keys:
: text: The segment’s text. **Required.**
id: Stable identifier — content-derived by default ([`segment_id()`](_autosummary/ef.html.md#ef.segment_id)).
parent_id: Id of the source document or parent segment this came from.
start: Character offset of the segment’s start in the source text.
end: Character offset of the segment’s end (exclusive) in the source.
index: Ordinal position of the segment in its segmenter’s output.
tokens: Token count of `text` (meaningful only with `metadata` key
> `tokenizer` recording which tokenizer counted it).
metadata: Free-form mapping; framework-specific keys live here. See
: `PROMOTED_METADATA_KEYS` for the conventional keys.
### *class* ef.SegmentRecord(text, id='', parent_id=None, start=None, end=None, index=None, tokens=None, metadata=)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
Frozen-dataclass view over a [`Segment`](_autosummary/ef.html.md#ef.Segment) — the convenience surface.
Use this when attribute access and immutability read better than dict
access; use the [`Segment`](_autosummary/ef.html.md#ef.Segment) `TypedDict` directly on the hot path.
The two convert losslessly via [`to_segment()`](_autosummary/ef.html.md#ef.SegmentRecord.to_segment) and [`segment_record()`](_autosummary/ef.html.md#ef.segment_record).
`id` is derived from `text` (and `metadata`) if left empty.
```pycon
>>> rec = SegmentRecord('hello', index=2)
>>> rec.id == segment_id('hello')
True
>>> rec.to_segment()['index']
2
```
#### to_segment()
Return an equivalent [`Segment`](_autosummary/ef.html.md#ef.Segment) `dict` (dropping empty keys).
* **Return type:**
[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)
### *class* ef.Segmenter(\*args, \*\*kwargs)
Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol)
Structural type for a segmenter: `str | Mapping -> Iterable[Segment]`.
A segmenter is *just a callable*. It accepts a document — a bare `str` or
a `Mapping` with a `text` key (and optionally an `id`, which becomes
the emitted segments’ `parent_id`) — and yields [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)
pieces. It returns an **iterable**, not a list: streaming is the default so
the heavy case scales.
Because the protocol’s only member is `__call__`, *every* callable
satisfies `isinstance(x, Segmenter)`. Treat it as documentation of intent,
not a discriminating check — [`as_segmenter()`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.as_segmenter) is
the seam that turns an arbitrary callable into a *well-behaved* segmenter.
### *class* ef.SourceManager(corpus=None, , segmenter=None, embedder=None, store=None, cache=None, auto_refresh=False, embedder_api_key=None)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
The heavy search facade — multi-config corpus indexing over one corpus.
A [`SourceManager`](_autosummary/ef.html.md#ef.SourceManager) holds a corpus, an [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph)
and one or more **configs**. A config is a named segmenter+embedder pipeline
([`PipelineSpec`](_autosummary/ef.config.html.md#ef.config.PipelineSpec)); [`register_config()`](_autosummary/ef.html.md#ef.SourceManager.register_config) adds one,
[`materialize()`](_autosummary/ef.html.md#ef.SourceManager.materialize) runs the corpus through it into a `vd` collection, and
[`search()`](_autosummary/ef.html.md#ef.SourceManager.search) queries it. Because every artifact is content-addressed in
the shared graph, two configs that share a step (e.g. the same segmenter)
compute that step’s artifacts **once** — config branching is free.
The constructor optionally registers a `"default"` config: pass an
`embedder` (and optionally a `segmenter`) and it is registered eagerly;
omit `embedder` to build every config explicitly via [`register_config()`](_autosummary/ef.html.md#ef.SourceManager.register_config).
* **Parameters:**
* **corpus** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the source corpus — coerced by [`as_corpus()`](_autosummary/ef.corpus.html.md#ef.corpus.as_corpus)
(a mapping, a directory path, an iterable of sources, or `None`).
* **segmenter** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the default config’s segmenter — coerced by
[`as_segmenter()`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.as_segmenter) (`None` → the recursive
default). Used only when `embedder` is also given.
* **embedder** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the default config’s embedder — coerced by
[`as_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.as_embedder). If given, a `"default"`
config is registered; if `None`, no default config is registered.
* **embedder_api_key** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – an optional API key for the default config’s embedder
— forwarded to a hosted-API adapter (`"openai:…"` etc.) when the
`embedder` is such a spec. The bring-your-own-key seam; ignored
for key-less embedders. See [`register_config()`](_autosummary/ef.html.md#ef.SourceManager.register_config).
* **store** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – where the vectors go. `None` → an in-memory `vd` backend; a
`vd` client → collections are created on it; a `vd` collection →
used directly (then only one config is supported); a backend-name
string → `vd.connect` of that backend.
* **cache** ([`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph)’s value cache — any
`MutableMapping`; `None` → an in-RAM `dict`.
* **auto_refresh** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – when `True`, the corpus is wrapped in a
[`ChangeDetectingCorpus`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus) and every edit made
*through* it is incrementally re-indexed into each materialized
config — the index stays live without an explicit
[`refresh()`](_autosummary/ef.refresh.html.md#module-ef.refresh). [`scan()`](_autosummary/ef.html.md#ef.SourceManager.scan) then also picks up out-of-band edits.
```pycon
>>> import numpy as np
>>> from ef import as_embedder
>>> e1 = as_embedder(lambda ts: np.array([[len(t)] for t in ts], float), model_id='len@1')
>>> sm = SourceManager(['hello', 'hi there'], embedder=e1)
>>> report = sm.materialize()
>>> report['segments']
2
>>> len(sm.search('hello')) >= 1
True
```
#### *property* configs *: [dict](https://docs.python.org/3/builtins/stdtypes.html#dict)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), [str](https://docs.python.org/3/builtins/stdtypes.html#str)]*
The registered configs as a `{name: config_id}` mapping.
#### diagnose(config=None)
Report how far one config’s index has drifted from the corpus.
Computes the four staleness conditions — `orphan` / `missing` /
`stale` / `misconfigured` — by comparing the config’s `vd`
collection against the current corpus (see [`ef.diagnostics`](_autosummary/ef.diagnostics.html.md#module-ef.diagnostics)).
Read-only: it changes nothing. [`refresh()`](_autosummary/ef.refresh.html.md#module-ef.refresh) acts on the result.
* **Return type:**
[`StalenessReport`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.StalenessReport)
```pycon
>>> sm = SourceManager({'a': 'alpha', 'b': 'beta'}, embedder='hashing')
>>> _ = sm.materialize()
>>> bool(sm.diagnose()) # freshly indexed — in sync
False
>>> sm.corpus['a'] = 'alpha rewritten'
>>> sm.diagnose().stale
('a',)
```
#### gc_orphans(config=None)
Delete the documents of sources that left the corpus — return the count.
The garbage-collection slice of [`refresh()`](_autosummary/ef.refresh.html.md#module-ef.refresh): it removes only
**orphan** documents (a source gone from the corpus) and prunes the
dead leaves out of the [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph),
touching nothing else. `gc_orphans()` is the orphan-deletion half of
`refresh(mode='full')`.
* **Return type:**
[`int`](https://docs.python.org/3/builtins/functions.html#int)
#### lineage(key)
The artifacts `key` was produced from — its provenance.
A thin pass-through to [`ArtifactGraph.ancestors`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.ancestors): “what produced this
vector / segment?”. The leaves of the returned set are the source
content hashes the artifact ultimately derives from.
* **Return type:**
[`frozenset`](https://docs.python.org/3/builtins/stdtypes.html#frozenset)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]
#### materialize(config=None, , sources=None)
Index the corpus through one config (or all) — corpus → segment → embed → `vd`.
For each source and each selected config: the source is added to the
[`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph) as a content-addressed leaf,
its `segment` and `embed` artifacts are [`materialize()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.materialize)d
(a cache hit if already computed — so a shared segment step runs once
across configs), and every `(segment, vector)` pair is written into
the config’s `vd` collection. Idempotent: re-materializing rewrites
identical documents.
* **Parameters:**
* **config** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the config to index — a name, a [`ConfigId`](_autosummary/ef.config.html.md#ef.config.ConfigId),
or `None` for *every* registered config.
* **sources** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – an optional subset of source ids to index — `None` (the
default) indexes the whole corpus. [`refresh()`](_autosummary/ef.refresh.html.md#module-ef.refresh) uses this to
re-index only the sources that changed; ids not in the corpus
are skipped.
* **Return type:**
[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]
* **Returns:**
a report `dict` with `configs` / `sources` / `segments` counts.
#### pipeline(config=None)
The [`PipelineSpec`](_autosummary/ef.config.html.md#ef.config.PipelineSpec) of a registered config.
* **Return type:**
[`PipelineSpec`](_autosummary/ef.config.html.md#ef.config.PipelineSpec)
#### rebuild(config=None)
Drop one config’s index entirely and re-index the whole corpus.
The heavy hammer: every document in the config’s `vd` collection is
deleted, then the whole corpus is re-[`materialize()`](_autosummary/ef.html.md#ef.SourceManager.materialize)d from
scratch. Use it to recover from a config whose `vd` documents were
produced by an earlier pipeline (every source `misconfigured`), or
whenever a guaranteed-clean index is wanted over the surgical
[`refresh()`](_autosummary/ef.refresh.html.md#module-ef.refresh). Every source is reported as `added` — the collection
was emptied and rebuilt.
* **Return type:**
[`RefreshReport`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshReport)
* **Returns:**
a [`RefreshReport`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshReport).
#### refresh(config=None, , sources=None, mode='full')
Re-sync one config’s `vd` index with the current corpus.
The explicit-refresh entry point. It [`diagnose()`](_autosummary/ef.html.md#ef.diagnose)s the config,
turns the staleness report into a [`RefreshPlan`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshPlan) for
the chosen `mode` ([`plan_refresh()`](_autosummary/ef.refresh.html.md#ef.refresh.plan_refresh)), deletes the
stale/orphan documents, re-[`materialize()`](_autosummary/ef.html.md#ef.SourceManager.materialize)s what changed, and
prunes the dead leaves out of the [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph).
Because it re-reads and re-hashes the corpus, it catches out-of-band
edits without any change-detection wrapper.
* **Parameters:**
* **config** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the config to refresh (name / id / `None` for the default
or sole config).
* **sources** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – an optional subset of source ids to refresh — sources
outside it are left untouched. Pair it with `mode='scoped_full'`
(or `'incremental'`); `'full'` rejects it.
* **mode** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'none'`, `'incremental'`, `'full'`, `'scoped_full'`]) – one of [`REFRESH_MODES`](_autosummary/ef.refresh.html.md#ef.refresh.REFRESH_MODES) — `'none'` (index
new/changed content, delete nothing), `'incremental'` (also
replace changed sources), `'full'` (also delete orphans — the
corpus is authoritative), `'scoped_full'` (`full` over the
`sources` subset). Default `'full'`.
* **Return type:**
[`RefreshReport`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshReport)
* **Returns:**
a [`RefreshReport`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshReport).
```pycon
>>> sm = SourceManager({'a': 'alpha text', 'b': 'beta text'}, embedder='hashing')
>>> _ = sm.materialize()
>>> sm.corpus['b'] = 'beta rewritten' # edit
>>> sm.corpus['c'] = 'gamma text' # add
>>> del sm.corpus['a'] # delete
>>> report = sm.refresh()
>>> report.added, report.modified, report.deleted
(('c',), ('b',), ('a',))
>>> bool(sm.diagnose()) # in sync again
False
```
#### register_config(name='default', , segmenter=None, embedder=None, embedder_api_key=None)
Register a named segmenter+embedder pipeline; return its [`ConfigId`](_autosummary/ef.config.html.md#ef.config.ConfigId).
The segmenter and embedder are coerced through `ef`’s DI seams
([`as_segmenter()`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.as_segmenter) /
[`as_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.as_embedder); an absent `embedder`
defaults to `DEFAULT_EMBEDDER`). Their ops are registered into the
shared [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph) and a `vd` collection
is created for the config. Registering a config does *not* index it —
call [`materialize()`](_autosummary/ef.html.md#ef.SourceManager.materialize).
`embedder_api_key` is the per-call bring-your-own-key seam: when given
and `embedder` is a hosted-API provider spec (`"openai:…"` etc.),
the key is forwarded to that adapter instead of relying on a server
environment variable. Ignored for key-less embedders.
Re-registering the same `name` replaces it. Two configs that resolve
to the same [`PipelineSpec`](_autosummary/ef.config.html.md#ef.config.PipelineSpec) share one
[`ConfigId`](_autosummary/ef.config.html.md#ef.config.ConfigId) (and one collection).
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
```pycon
>>> import numpy as np
>>> from ef import as_embedder
>>> sm = SourceManager(['a', 'b'])
>>> cid = sm.register_config(
... 'mini', embedder=as_embedder(
... lambda ts: np.ones((len(ts), 2)), model_id='ones@2'))
>>> len(cid)
64
```
#### retrieve(query, , config=None, limit=10, filter=None)
Retrieve ranked [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)s — the RAG-plug-in surface.
The [`SourceManager`](_autosummary/ef.html.md#ef.SourceManager) counterpart of
[`SearchableCorpus.retrieve()`](_autosummary/ef.html.md#ef.SearchableCorpus.retrieve): plain segments in rank order, the
clean context shape an external RAG/agent framework consumes (`ef`
does not synthesize answers). [`search()`](_autosummary/ef.html.md#ef.SourceManager.search) is the scored counterpart.
* **Return type:**
[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]
#### scan()
Re-scan the corpus for out-of-band edits — needs `auto_refresh=True`.
Picks up changes made to the backing store *directly* (a file edited on
disk, an S3 object replaced) rather than through the corpus wrapper.
When the manager was built with `auto_refresh=True` the corpus is a
[`ChangeDetectingCorpus`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus); this calls its
[`scan()`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus.scan), which fires a
[`ChangeEvent`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeEvent) per drift — each one auto-applied to
every materialized config. Returns the [`CorpusDiff`](_autosummary/ef.corpus.html.md#ef.corpus.CorpusDiff).
For a large bulk of out-of-band edits, prefer [`refresh()`](_autosummary/ef.refresh.html.md#module-ef.refresh) (one
corpus pass) over `scan` (one collection pass per changed source).
* **Raises:**
[**TypeError**](https://docs.python.org/3/builtins/exceptions.html#TypeError) – if the corpus is not change-detecting (the manager was
not built with `auto_refresh=True`).
* **Return type:**
[`CorpusDiff`](_autosummary/ef.corpus.html.md#ef.corpus.CorpusDiff)
#### search(query, , config=None, limit=10, filter=None)
Search one config’s index — return up to `limit` ranked [`SearchHit`](_autosummary/ef.html.md#ef.SearchHit)s.
`config` selects which config to query (a name / [`ConfigId`](_autosummary/ef.config.html.md#ef.config.ConfigId),
or `None` for the default or sole config). `query` is text or a
pre-computed query vector; `filter` is a `vd` metadata filter.
* **Return type:**
[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`SearchHit`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchHit)]
#### searchable(config=None)
Return a thin [`SearchableCorpus`](_autosummary/ef.html.md#ef.SearchableCorpus) bound to one config’s index.
* **Return type:**
[`SearchableCorpus`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus)
### *class* ef.StalenessReport(config, orphan=(), missing=(), stale=(), misconfigured=(), fresh=())
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
The four staleness conditions of one config — what [`diagnose()`](_autosummary/ef.html.md#ef.diagnose) returns.
Each field is a sorted tuple of `source_id`s, so a report is
deterministic and JSON-friendly. `orphan` and `missing` are disjoint
from the rest (a source absent from one side cannot also be compared); a
source can be **both** `stale` and `misconfigured` (its content changed
*and* the config did) — the conditions are reported independently.
`fresh` is the complement: indexed, current and built by this config.
Truthiness reports whether *anything is wrong* — `bool(report)` is
`False` only for a perfectly in-sync index.
```pycon
>>> r = StalenessReport(config='c0', stale=('a',), fresh=('b',))
>>> bool(r), r.needs_reindexing
(True, ('a',))
>>> bool(StalenessReport(config='c0', fresh=('a', 'b')))
False
```
#### *property* needs_reindexing *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), ...]*
Sources whose vectors must be recomputed — `stale` ∪ `misconfigured`.
### *class* ef.TransformSpec(op, op_version, params=)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
One step of a pipeline — a serializable recipe for an op application.
A [`TransformSpec`](_autosummary/ef.html.md#ef.TransformSpec) names *which op* (`op`, a key resolved through
the [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph)’s `ops` registry), \*at which
version\* (`op_version` — it participates in every downstream hash), and
*with which keyword params* (`params`). It does **not** record the step’s
inputs: those are wired structurally by the [`PipelineSpec`](_autosummary/ef.html.md#ef.PipelineSpec) (the embed
step consumes the segment step’s output). It is the per-step counterpart of
a [`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec).
The op key carries the *component identity* — e.g. `"embed:openai:text-
embedding-3-large@1024"` or `"segment:"` (see
[`ef.ops`](_autosummary/ef.ops.html.md#module-ef.ops)) — so two configs with different components get disjoint
artifact cones automatically.
```pycon
>>> ts = TransformSpec(op='embed:m@8', op_version='2', params={'input_type': 'document'})
>>> ts.op, ts.op_version
('embed:m@8', '2')
>>> TransformSpec.from_dict(ts.as_dict()) == ts
True
```
#### as_dict()
Return a plain-`dict` (JSON-serializable) form of the spec.
* **Return type:**
[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]
```pycon
>>> TransformSpec(op='segment:r', op_version='1', params={}).as_dict()
{'op': 'segment:r', 'op_version': '1', 'params': {}}
```
#### *classmethod* from_dict(d)
Rebuild a [`TransformSpec`](_autosummary/ef.html.md#ef.TransformSpec) from its [`as_dict()`](_autosummary/ef.html.md#ef.TransformSpec.as_dict) form.
* **Return type:**
[`TransformSpec`](_autosummary/ef.config.html.md#ef.config.TransformSpec)
### ef.approx_token_count(text)
Rough token count — the common `~4 characters per token` heuristic.
A dependency-free default so `ef` can talk in “tokens” without pulling in
`tiktoken` or a model tokenizer. Inject a real counter (and its name) into
a segmenter when an exact count matters.
* **Return type:**
[`int`](https://docs.python.org/3/builtins/functions.html#int)
```pycon
>>> approx_token_count('a' * 40)
10
>>> approx_token_count('')
0
```
### ef.artifact_id(spec)
Content-addressed id of `spec` — `sha256` over its four fields.
The id is `sha256` of canonical (sorted-key) JSON of
`{op, op_version, inputs, params}`. Two specs that describe the same
computation hash *identically*; any change to the op, its version, its
inputs (including their order) or its params yields a *different* id. That
is what makes the graph content-addressed: the id *is* the recipe.
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
```pycon
>>> a = producer_spec('embed', 'seg-1', op_version='2', model='small')
>>> artifact_id(a) == artifact_id(producer_spec(
... 'embed', 'seg-1', op_version='2', model='small'))
True
>>> len(artifact_id(a))
64
>>> artifact_id(a) == artifact_id(producer_spec(
... 'embed', 'seg-1', op_version='3', model='small')) # version bump
False
>>> artifact_id(producer_spec('op', 'x', 'y', op_version='1')) == artifact_id(
... producer_spec('op', 'y', 'x', op_version='1')) # input order matters
False
```
### ef.as_corpus(x=None, , \*\*kwargs)
Normalize `x` into a corpus (`MutableMapping[source_id, Source]`).
The single dependency-injection seam through which every `ef` entry point
accepts a user-supplied corpus — the mirror of
[`as_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.as_embedder) and
[`as_segmenter()`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.as_segmenter). Accepts, in order:
1. `None` — a fresh empty in-RAM `dict`;
2. a `Mapping` — returned **unchanged** (a `dict`, a `dol` store,
another corpus: it already *is* a corpus);
3. a directory path (`str` / `os.PathLike` naming an existing
directory) — a `dol` filesystem store of the text files under it
(`dol` is imported lazily; only this branch needs it);
4. an iterable of sources (strings or mappings) — an in-RAM `dict` keyed
by each source’s [`content_hash()`](_autosummary/ef.html.md#ef.content_hash) (content-addressed, so duplicate
sources collapse to one entry).
Extra `**kwargs` are forwarded to the `dol` filesystem store (the
directory branch only) and ignored otherwise.
* **Raises:**
[**TypeError**](https://docs.python.org/3/builtins/exceptions.html#TypeError) – if `x` is a non-directory string, or none of the above.
```pycon
>>> as_corpus()
{}
>>> as_corpus({'a': 'x'}) # a mapping passes straight through
{'a': 'x'}
>>> sorted(as_corpus(['alpha', 'beta']).values())
['alpha', 'beta']
>>> len(as_corpus(['dup', 'dup'])) # content-addressed: duplicates collapse
1
```
### ef.as_embedder(x, \*\*kwargs)
Normalize `x` into an [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) — the DI seam.
The single place every `ef` entry point coerces a user-supplied embedder
argument. Accepts, in order:
1. an existing [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) — returned unchanged;
2. a URL string (`http://` / `https://`) — [`http_embedder()`](_autosummary/ef.html.md#ef.http_embedder);
3. a provider-prefixed string — `"openai:"`, `"cohere:"`,
`"voyage:"` or `"gemini:"` — the matching hosted-API
adapter;
4. the bare string `"hashing"` — the dependency-free
[`HashingEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.HashingEmbedder) (`ef`’s zero-install default);
5. an `"st:"` string, or any other bare model name —
[`sentence_transformers_embedder()`](_autosummary/ef.html.md#ef.sentence_transformers_embedder) (the local neural default);
6. a bare callable — wrapped in
[`FunctionEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.FunctionEmbedder) (the `imbed`-embedder bridge).
Extra `**kwargs` are forwarded to the chosen factory.
* **Raises:**
[**TypeError**](https://docs.python.org/3/builtins/exceptions.html#TypeError) – if `x` is none of the above.
* **Return type:**
[*Embedder*](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)
```pycon
>>> import numpy as np
>>> e = as_embedder(lambda ts: np.zeros((len(ts), 2)), model_id='zero@2')
>>> type(e).__name__
'FunctionEmbedder'
```
* **Return type:**
[`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)
### ef.as_ragas_dataset(samples)
Convert [`RagSample`](_autosummary/ef.html.md#ef.RagSample)s into a Ragas `EvaluationDataset`.
The hookpoint for the *LLM-judged* RAG metrics `ef` deliberately does not
compute itself (faithfulness, answer relevancy, …). [`RagSample`](_autosummary/ef.html.md#ef.RagSample)
already mirrors Ragas’ `SingleTurnSample` field-for-field, so this is a
thin, lossless adapter. Run the evaluation with your own LLM:
```default
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy
dataset = as_ragas_dataset(samples)
result = evaluate(dataset, metrics=[faithfulness, answer_relevancy])
```
* **Parameters:**
**samples** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`RagSample`](_autosummary/ef.evaluation.html.md#ef.evaluation.RagSample) | [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]]) – an iterable of [`RagSample`](_autosummary/ef.html.md#ef.RagSample)s (or plain mappings).
* **Return type:**
[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)
* **Returns:**
a `ragas.EvaluationDataset`.
* **Raises:**
[**ImportError**](https://docs.python.org/3/builtins/exceptions.html#ImportError) – if the optional `ragas` package is not installed.
### ef.as_segment(x, \*\*defaults)
Coerce `x` into a [`Segment`](_autosummary/ef.html.md#ef.Segment) — accepts a `str`, `Mapping` or record.
This is the segment-side normalizer that lets a [`Segmenter`](_autosummary/ef.html.md#ef.Segmenter) accept
loosely-typed output (a bare string, a partial `dict`) and still emit a
well-formed [`Segment`](_autosummary/ef.html.md#ef.Segment). `**defaults` supplies *fallback* values:
a default fills a field only when `x` does not already carry it.
Accepts:
1. a [`SegmentRecord`](_autosummary/ef.html.md#ef.SegmentRecord) — converted via [`SegmentRecord.to_segment()`](_autosummary/ef.html.md#ef.SegmentRecord.to_segment);
2. a `str` — becomes the `text` of a fresh segment;
3. a `Mapping` with a `text` key — its keys are carried through.
The `id` is always present on the result (derived if absent).
* **Raises:**
* [**TypeError**](https://docs.python.org/3/builtins/exceptions.html#TypeError) – if `x` is none of the accepted forms.
* [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if `x` is a `Mapping` without a `text` key.
* **Return type:**
[*Segment*](_autosummary/ef.segments.html.md#ef.segments.Segment)
```pycon
>>> as_segment('plain text', index=3)['index']
3
>>> as_segment({'text': 'kept', 'index': 0}, index=9)['index'] # x wins
0
```
* **Return type:**
[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)
### ef.as_segmenter(x=None, , \*\*kwargs)
Normalize `x` into a [`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter) — the DI seam.
The single place every `ef` entry point coerces a user-supplied segmenter
argument. Accepts, in order:
1. `None` or `"default"` / `"recursive"` — a
[`RecursiveCharacterSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.RecursiveCharacterSegmenter) (the default);
2. `"lines"` — [`line_segmenter()`](_autosummary/ef.segmenters.html.md#ef.segmenters.line_segmenter);
3. any other string — looked up in `imbed`’s registry via
[`imbed_segmenter()`](_autosummary/ef.html.md#ef.imbed_segmenter);
4. a ready-made segmenter (a [`BaseSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.BaseSegmenter) or a
callable marked by `ef`) — returned unchanged;
5. a bare callable — wrapped in [`FunctionSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.FunctionSegmenter).
Extra `**kwargs` are forwarded to the chosen factory (ignored when `x`
is already a segmenter).
* **Raises:**
[**TypeError**](https://docs.python.org/3/builtins/exceptions.html#TypeError) – if `x` is none of the above.
* **Return type:**
[*Segmenter*](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter)
```pycon
>>> seg = as_segmenter(chunk_size=256, chunk_overlap=32)
>>> (seg.chunk_size, seg.chunk_overlap)
(256, 32)
```
* **Return type:**
[`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter)
### ef.average_precision(ranked, relevant, k=None)
Average precision of `ranked` — the per-query term of MAP.
The mean of the precision measured at each relevant result’s rank. `k`
optionally caps the ranking.
* **Return type:**
[`float`](https://docs.python.org/3/builtins/functions.html#float)
```pycon
>>> average_precision(['d1', 'x', 'd2'], {'d1': 1.0, 'd2': 1.0})
0.8333333333333333
```
### ef.cache_key(embedder, text, input_type=None)
Deterministic cache key for one `(embedder, text, input_type)` triple.
The key pins everything that changes the resulting vector. `model_id` is
the embedder-identity SSOT — it already bakes in provider, model and `dim`
— so `dim` is *not* keyed separately (keying it would also break
[`FunctionEmbedder`](_autosummary/ef.html.md#ef.FunctionEmbedder)’s lazy `dim` inference). `normalized` is keyed
because a [`NormalizingEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.NormalizingEmbedder) shares its
inner’s `model_id` yet yields different vectors.
Returns `"/"` — a namespaced key safe for any
`MutableMapping` store.
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
```pycon
>>> class _E:
... model_id, normalized = 'm@8', False
>>> k1 = cache_key(_E(), 'hello', 'document')
>>> k2 = cache_key(_E(), 'hello', 'query')
>>> k1 == k2 # input_type participates
False
>>> k1.startswith('m@8/')
True
```
### ef.cluster(data, , method='kmeans', n_clusters=8, min_cluster_size=5, embedder=None, normalize=True, random_state=42)
Cluster a corpus’s embeddings (use cases §G2).
`method='kmeans'` is the default and is **dependency-free** — a numpy
Lloyd’s algorithm with k-means++ seeding. `method='hdbscan'` finds a
density-based clustering (and a noise label `-1`) and needs the
`ef[explore]` extra.
* **Parameters:**
* **data** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – what to cluster — see [`project()`](_autosummary/ef.html.md#ef.project) / `_resolve_vectors()`.
* **method** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'kmeans'`, `'hdbscan'`]) – `'kmeans'` (numpy-only) or `'hdbscan'` (optional extra).
* **n_clusters** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – number of clusters for k-means (clamped to `n_samples`).
* **min_cluster_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – smallest cluster HDBSCAN will form.
* **embedder** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – embedder used when `data` is raw text.
* **normalize** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – L2-normalize the vectors first, so Euclidean distance
behaves like cosine similarity — the right default for semantic
embeddings. Set `False` to cluster raw vectors.
* **random_state** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – seed for k-means initialization.
* **Return type:**
`ndarray`
* **Returns:**
An `ndarray` of integer cluster labels, shape `(n_samples,)`, in
input order. HDBSCAN labels unclustered points `-1`.
```pycon
>>> import numpy as np
>>> vectors = np.random.RandomState(2).rand(15, 8)
>>> labels = cluster(vectors, method='kmeans', n_clusters=3, random_state=0)
>>> labels.shape
(15,)
>>> set(labels.tolist()) <= {0, 1, 2}
True
```
### ef.cohere_embedder(model='embed-v4.0', , dim=None, api_key=None, batch_size=96, timeout=60.0)
Build a Cohere-embeddings [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder).
Talks to Cohere’s `/v2/embed` REST endpoint directly — no `cohere` SDK,
no extra dependency. Cohere’s v3+ models *require* a task hint; this adapter
supplies one for every canonical [`InputType`](_autosummary/ef.embedders.html.md#ef.embedders.InputType) and
defaults to the document role when none is given.
* **Parameters:**
* **model** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – A Cohere embedding model name.
* **dim** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Matryoshka-truncated width (`embed-v4.0` only). Omit for the
model’s native width. The dim is part of the embedder’s identity —
set it once here, not per call.
* **api_key** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Cohere API key. Falls back to `CO_API_KEY` / `COHERE_API_KEY`.
* **batch_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Texts per request (capped at Cohere’s 96 limit).
* **timeout** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Per-request timeout in seconds.
* **Return type:**
[`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)
* **Returns:**
An [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) over the Cohere API.
### ef.config_id(spec)
Content-addressed id of a [`PipelineSpec`](_autosummary/ef.html.md#ef.PipelineSpec) — `sha256` over its steps.
The id is `sha256` of the canonical (sorted-key) JSON of
[`PipelineSpec.as_dict()`](_autosummary/ef.html.md#ef.PipelineSpec.as_dict). Two pipelines that describe the same
computation hash identically; any change to a step’s op, version or params
yields a different id. `ef` writes this digest into `vd` index metadata
as `config_hash`.
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
```pycon
>>> seg = TransformSpec(op='segment:r', op_version='1', params={})
>>> emb = TransformSpec(op='embed:m@8', op_version='1', params={})
>>> cid = config_id(PipelineSpec(segment=seg, embed=emb))
>>> len(cid)
64
>>> cid == config_id(PipelineSpec(segment=seg, embed=emb))
True
```
### ef.content_hash(source, , content_keys=None)
Content hash of a source document — `sha256` of its normalized content.
This is the basis of change detection: re-reading an unchanged source
yields an identical hash, so a refresh that finds the same hash does
nothing. Hashing *normalized* content (NFC, no BOM, `\n` line endings —
see [`ef.hashing`](_autosummary/ef.hashing.html.md#module-ef.hashing)) means cosmetic encoding differences never look like
a change.
The treatment of a `Mapping` source is deliberate:
- a plain `str` source — the hash covers the text;
- a `Mapping` with a `text` key — the hash covers `source['text']`
**only**. A change to title, tags, ACL or any other metadata does *not*
change the hash, so it never triggers a spurious re-segment/re-embed
(embedding is the pipeline’s most expensive step). Name the metadata keys
that *do* feed downstream in `content_keys` and changes to those keys
will be detected;
- a `Mapping` *without* a `text` key — treated as a structured document
and hashed via canonical (sorted-key) JSON of the whole mapping;
- `bytes` — hashed raw (no text normalization is possible pre-parse).
* **Parameters:**
* **source** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – a `str`, `bytes`, or a `Mapping` (ideally with a `text`
key).
* **content_keys** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – metadata keys whose values join the text in the hash.
Each key is looked up at the top level of `source` or inside a
`source['metadata']` sub-mapping. Use this *only* for metadata
that genuinely affects the L1–L4 pipeline (e.g. a title prepended
to the text before embedding); key order does not matter.
* **Raises:**
[**TypeError**](https://docs.python.org/3/builtins/exceptions.html#TypeError) – if `source` is not a `str`, `bytes` or `Mapping`.
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
```pycon
>>> content_hash('hello') == content_hash('hello')
True
>>> content_hash('hello') == content_hash('hello world')
False
>>> content_hash('a\r\nb') == content_hash('a\nb') # line endings normalized
True
>>> a = {'text': 'body', 'metadata': {'tags': ['x']}}
>>> b = {'text': 'body', 'metadata': {'tags': ['y']}}
>>> content_hash(a) == content_hash(b) # metadata ignored by default
True
>>> content_hash(a, content_keys=['tags']) == content_hash(b, content_keys=['tags'])
False
>>> content_hash({'text': 'body'}) == content_hash('body') # wrapper is transparent
True
```
### ef.context_precision(reference, retrieved_contexts)
Fraction of the retrieved contexts that lexically overlap the `reference`.
A deterministic *proxy* for retrieval precision — a context counts as useful
if it shares at least one (normalized) token with the reference answer.
* **Return type:**
[`float`](https://docs.python.org/3/builtins/functions.html#float)
```pycon
>>> context_precision('paris', ['Paris is in France.', 'Unrelated text.'])
0.5
```
### ef.context_recall(reference, retrieved_contexts)
Lexical recall of the `reference` tokens by the retrieved contexts.
The fraction of the (normalized) reference’s distinct tokens that appear
anywhere in `retrieved_contexts` — a deterministic *proxy* for “did
retrieval surface the information the answer needs”. For the LLM-judged
context recall, use [`as_ragas_dataset()`](_autosummary/ef.html.md#ef.as_ragas_dataset) and Ragas.
* **Return type:**
[`float`](https://docs.python.org/3/builtins/functions.html#float)
```pycon
>>> context_recall('paris france', ['Paris is the capital of France.'])
1.0
>>> context_recall('paris france', ['London is in England.'])
0.0
```
### ef.cross_encoder_reranker(model_name='cross-encoder/ms-marco-MiniLM-L-6-v2', , device=None, batch_size=32, \*\*predict_kwargs)
Build a [`Reranker`](_autosummary/ef.html.md#ef.Reranker) backed by a sentence-transformers `CrossEncoder`.
A cross-encoder scores a `(query, passage)` pair *jointly* — it is the
accurate, slower second stage two-stage retrieval is built for. The model is
a heavy, optional dependency: it is imported only when this factory is
called, so importing [`ef.reranking`](_autosummary/ef.reranking.html.md#module-ef.reranking) itself stays cheap.
Example:
```default
from ef import ingest, with_reranker, cross_encoder_reranker
index = ingest(corpus, embedder="st:all-MiniLM-L6-v2")
retrieve = with_reranker(index.retrieve, cross_encoder_reranker())
segments = retrieve("a query", limit=5)
```
* **Parameters:**
* **model_name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – a sentence-transformers `CrossEncoder` model name.
* **device** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – torch device (`"cuda"` / `"cpu"` / `"mps"`); `None`
auto-selects.
* **batch_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – how many pairs to score per forward pass.
* **predict_kwargs** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – extra keyword arguments forwarded to `CrossEncoder.predict`.
* **Return type:**
[`Reranker`](_autosummary/ef.reranking.html.md#ef.reranking.Reranker)
* **Returns:**
a [`Reranker`](_autosummary/ef.html.md#ef.Reranker) — `(query, segments) -> scores`.
* **Raises:**
[**ImportError**](https://docs.python.org/3/builtins/exceptions.html#ImportError) – if the optional `sentence-transformers` package is absent.
### ef.dcg_at_k(ranked, relevant, k)
Discounted cumulative gain of `ranked` at cutoff `k`.
`sum(grade_i / log2(i + 2))` over the top `k` results (`i` 0-indexed),
with `grade_i` the relevance of the `i`-th doc (`0` if unjudged) — the
linear-gain DCG that BEIR / `pytrec_eval` use.
* **Return type:**
[`float`](https://docs.python.org/3/builtins/functions.html#float)
```pycon
>>> round(dcg_at_k(['d1', 'x', 'd2'], {'d1': 1.0, 'd2': 1.0}, 3), 4)
1.5
```
### ef.diagnose(corpus, collection, config)
Compute the four staleness conditions of `collection` against `corpus`.
The pure core of `ef`’s diagnostics — no side effects. It reads what the
`vd` `collection` records (`indexed_state()`), what the `corpus`
currently holds (`corpus_state()`), and partitions every source id into
exactly the buckets it belongs in:
- **orphan** — in the collection, not in the corpus;
- **missing** — in the corpus, not in the collection;
- **stale** — in both, but the recorded `source_hash` ≠ the current one;
- **misconfigured** — in both, but a recorded `config_hash` ≠ `config`;
- **fresh** — in both, content current and config matching.
* **Parameters:**
* **corpus** ([`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]] | [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – the source corpus.
* **collection** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the `vd` collection backing the config being diagnosed.
* **config** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the [`ConfigId`](_autosummary/ef.config.html.md#ef.config.ConfigId) the collection *should* hold
— documents recording any other `config_hash` are misconfigured.
* **Return type:**
[`StalenessReport`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.StalenessReport)
* **Returns:**
a [`StalenessReport`](_autosummary/ef.html.md#ef.StalenessReport).
### ef.embed_length_sorted(texts, encode, , batch_size=128)
Encode `texts` in length-sorted batches; return vectors in caller order.
Transformer self-attention is O(n²) in sequence length and a batch is padded
to its longest member, so grouping similar-length texts together avoids
wasted compute — a 5–10× speedup on skewed corpora. `ef` owns this so
callers never sort, pad, or un-sort themselves.
* **Parameters:**
* **texts** ([`Sequence`](https://docs.python.org/3/library/typing.html#typing.Sequence)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – The strings to embed.
* **encode** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Sequence`](https://docs.python.org/3/library/typing.html#typing.Sequence)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – Maps a list of strings to an array-like of shape `(k, dim)`.
* **batch_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Texts per call to `encode`.
* **Return type:**
`ndarray`
* **Returns:**
A `float32` array of shape `(len(texts), dim)` in the *original*
order of `texts`.
```pycon
>>> import numpy as np
>>> def enc(batch): # toy encoder: vector = [length]
... return np.array([[len(t)] for t in batch], dtype=np.float32)
>>> out = embed_length_sorted(['aaa', 'a', 'aaaaa', 'aa'], enc, batch_size=2)
>>> out.ravel().tolist() # order preserved despite internal sorting
[3.0, 1.0, 5.0, 2.0]
```
### ef.evaluate_rag(samples, , metrics=('exact_match', 'token_f1', 'context_recall', 'context_precision', 'non_empty_rate'))
Score RAG samples with deterministic, reference-based lexical metrics.
Each sample is a [`RagSample`](_autosummary/ef.html.md#ef.RagSample) — `(user_input, response,
retrieved_contexts, reference)`. `evaluate_rag` computes only metrics
that need **no LLM**: [`exact_match()`](_autosummary/ef.html.md#ef.exact_match), [`token_f1()`](_autosummary/ef.html.md#ef.token_f1),
[`context_recall()`](_autosummary/ef.html.md#ef.context_recall), [`context_precision()`](_autosummary/ef.html.md#ef.context_precision) and `non_empty_rate`
(the share of samples that retrieved any context). This keeps `ef` a
facade — it never synthesizes an answer and never calls a model (§6).
For the LLM-judged metrics — faithfulness, answer relevancy, … — pass the
same samples to [`as_ragas_dataset()`](_autosummary/ef.html.md#ef.as_ragas_dataset) and run Ragas with your own LLM.
* **Parameters:**
* **samples** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`RagSample`](_autosummary/ef.evaluation.html.md#ef.evaluation.RagSample) | [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]]) – an iterable of [`RagSample`](_autosummary/ef.html.md#ef.RagSample)s (or plain mappings of the
same shape).
* **metrics** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – which metrics to compute — any of `exact_match` /
`token_f1` / `context_recall` / `context_precision` /
`non_empty_rate`. The reference-based ones skip samples that carry
no `reference`.
* **Return type:**
[`RagEvalReport`](_autosummary/ef.evaluation.html.md#ef.evaluation.RagEvalReport)
* **Returns:**
a [`RagEvalReport`](_autosummary/ef.html.md#ef.RagEvalReport).
* **Raises:**
[**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – for an unknown metric name, or if `samples` is empty.
```pycon
>>> samples = [
... {'user_input': 'capital of France?', 'response': 'Paris',
... 'reference': 'Paris',
... 'retrieved_contexts': ['Paris is the capital of France.']},
... {'user_input': '2 + 2?', 'response': 'five', 'reference': 'four',
... 'retrieved_contexts': []},
... ]
>>> report = evaluate_rag(samples)
>>> report.metrics['exact_match']
0.5
>>> report.metrics['non_empty_rate']
0.5
```
### ef.evaluate_retrieval(retriever, qrels, queries, , k_values=(1, 5, 10, 100), metrics=('ndcg', 'recall', 'precision', 'mrr', 'map'), limit=None)
Score a retriever against a BEIR-shaped evaluation set.
For every query judged in `qrels`, the `retriever` is run, its results
are mapped back to *source document ids* (`_result_doc_id()`,
de-duplicated to the best rank per source), and each requested metric is
computed at each cutoff in `k_values`. The report holds the mean of each
`"@"` over all scored queries — primary metric **NDCG@10**.
The natural pairing with [`ingest()`](_autosummary/ef.source_manager.html.md#ef.source_manager.ingest):
```default
corpus, queries, qrels = read_beir("path/to/beir/dataset")
index = ingest(corpus, embedder="st:all-MiniLM-L6-v2")
report = evaluate_retrieval(index, qrels, queries)
report.primary # mean NDCG@10
```
* **Parameters:**
* **retriever** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – what answers a query — a [`SearchableCorpus`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus),
a [`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager), or any callable
`query -> ranked results`. Results may be
[`SearchHit`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchHit)s, [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)s
or bare doc-id strings.
* **qrels** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`float`](https://docs.python.org/3/builtins/functions.html#float)]]) – the relevance judgements — `{query_id: {doc_id: relevance}}`.
Queries with no positive judgement are skipped.
* **queries** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – the query texts — `{query_id: query_text}`. A judged query
absent here cannot be run and is skipped.
* **k_values** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`int`](https://docs.python.org/3/builtins/functions.html#int)]) – the rank cutoffs to evaluate at.
* **metrics** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – which metrics to compute — any of `ndcg` / `recall` /
`precision` / `mrr` / `map`.
* **limit** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – how many results to retrieve per query. `None` uses
`max(k_values)`; raise it when sources segment into many pieces
(de-duplication to source level can otherwise shorten the ranking).
* **Return type:**
[`RetrievalEvalReport`](_autosummary/ef.evaluation.html.md#ef.evaluation.RetrievalEvalReport)
* **Returns:**
a [`RetrievalEvalReport`](_autosummary/ef.html.md#ef.RetrievalEvalReport).
* **Raises:**
[**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – for an unknown metric name, or if no query can be scored.
```pycon
>>> qrels = {'q1': {'d1': 1.0}, 'q2': {'d2': 1.0}}
>>> queries = {'q1': 'alpha', 'q2': 'beta'}
>>> def retriever(query, *, limit=10):
... return {'alpha': ['d1', 'd2'], 'beta': ['d1', 'd2']}[query][:limit]
>>> report = evaluate_retrieval(retriever, qrels, queries, k_values=(1, 2))
>>> report.metrics['recall@2'], report.n_queries
(1.0, 2)
```
### ef.exact_match(response, reference)
`1.0` if `response` equals `reference` after normalization, else `0.0`.
* **Return type:**
[`float`](https://docs.python.org/3/builtins/functions.html#float)
```pycon
>>> exact_match('The Eiffel Tower', 'eiffel tower')
1.0
>>> exact_match('Paris', 'London')
0.0
```
### ef.explore(data, , dims=2, projection_method='auto', cluster_method='kmeans', n_clusters=8, min_cluster_size=5, label=False, context=' ', n_words=4, embedder=None, random_state=42)
Project, cluster and (optionally) label a corpus in one call.
The layer-L5 orchestrator — [`project()`](_autosummary/ef.html.md#ef.project) + [`cluster()`](_autosummary/ef.html.md#ef.cluster) (+ optionally
[`label_clusters()`](_autosummary/ef.html.md#ef.label_clusters)) wired into one structured result. Where
[`project()`](_autosummary/ef.html.md#ef.project) / [`cluster()`](_autosummary/ef.html.md#ef.cluster) return *bare arrays in input order*,
`explore` keeps every row tied to its id: it returns an
[`ExploreResult`](_autosummary/ef.html.md#ef.ExploreResult) with `ids` / `coords` / `labels` row-aligned
(plus `cluster_titles`) — the JSON-friendly shape an `app_ef` corpus map
(or [`ef.service.EfService.explore_corpus()`](_autosummary/ef.service.html.md#ef.service.EfService.explore_corpus)) consumes directly.
* **Parameters:**
* **data** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the corpus to explore — anything `_resolve_explorable()`
understands (a [`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager) /
[`SearchableCorpus`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus), a
[`Corpus`](_autosummary/ef.corpus.html.md#ef.corpus.Corpus) mapping, an iterable of texts or
[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)s, or a vector matrix).
* **dims** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – projection target dimensionality — `2` or `3`.
* **projection_method** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'auto'`, `'umap'`, `'pca'`]) – forwarded to [`project()`](_autosummary/ef.html.md#ef.project) — `'auto'` /
`'umap'` / `'pca'`.
* **cluster_method** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'kmeans'`, `'hdbscan'`]) – forwarded to [`cluster()`](_autosummary/ef.html.md#ef.cluster) — `'kmeans'` /
`'hdbscan'`.
* **n_clusters** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – number of k-means clusters.
* **min_cluster_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – smallest cluster HDBSCAN will form.
* **label** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – when `True`, name each cluster with [`label_clusters()`](_autosummary/ef.html.md#ef.label_clusters)
(needs the `ef[imbed]` extra, an LLM key, and text-bearing
`data`); when `False` (the default) `cluster_titles` is empty.
* **context** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the corpus topic passed to [`label_clusters()`](_autosummary/ef.html.md#ef.label_clusters).
* **n_words** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – maximum cluster-title length, in words.
* **embedder** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – embedder used when `data` is raw text.
* **random_state** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – seed — projection and clustering are reproducible.
* **Return type:**
[`ExploreResult`](_autosummary/ef.exploration.html.md#ef.exploration.ExploreResult)
* **Returns:**
an [`ExploreResult`](_autosummary/ef.html.md#ef.ExploreResult) — `ids`, `coords` and `labels`
row-aligned, and `cluster_titles` (empty unless `label=True`).
* **Raises:**
[**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if the corpus is empty, has fewer than 2 samples, or
`label=True` is asked of a text-less vector matrix.
```pycon
>>> import numpy as np
>>> vectors = np.random.RandomState(0).rand(12, 16)
>>> result = explore(vectors, projection_method='pca', n_clusters=3, random_state=0)
>>> len(result['ids']), len(result['coords']), len(result['labels'])
(12, 12, 12)
>>> result['ids'][:3]
['0', '1', '2']
>>> len(result['coords'][0]) # each row has `dims` coordinates
2
>>> set(result['labels']) <= {0, 1, 2}
True
>>> result['cluster_titles'] # empty unless label=True
{}
```
### ef.full_kwargs(op, args=(), kwargs=None)
Normalize a call into a canonical, fully-named, defaults-filled kwargs dict.
The `i2.Sig` recipe of design notes §3.8. A call expressed as positional
`args` + keyword `kwargs` is mapped onto `op`’s signature: every
parameter is named, and any parameter the call left implicit is filled with
its default. The result is therefore *stable* — `op(x)` and
`op(x, size=512)` (`512` the default) yield the **same** dict, which is
what makes a content hash built from it correct.
* **Parameters:**
* **op** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – the callable whose signature defines the parameter names/defaults.
* **args** ([`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any), [`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis)]) – the call’s positional arguments — for an [`ef.ops`](_autosummary/ef.ops.html.md#module-ef.ops) op these
are the upstream artifact ids; [`step_params()`](_autosummary/ef.html.md#ef.step_params) drops them again.
* **kwargs** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the call’s keyword arguments.
* **Return type:**
[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]
```pycon
>>> def embed(segments, *, input_type='document', dim=None): ...
>>> full_kwargs(embed, ('seg-1',), {'input_type': 'query'})
{'segments': 'seg-1', 'input_type': 'query', 'dim': None}
>>> full_kwargs(embed, ('seg-1',)) # defaults filled in
{'segments': 'seg-1', 'input_type': 'document', 'dim': None}
```
### ef.gemini_embedder(model='gemini-embedding-001', , dim=None, api_key=None, batch_size=100, timeout=60.0)
Build a Google Gemini [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder).
Talks to the Generative Language `batchEmbedContents` REST endpoint
directly — no `google-genai` SDK, no extra dependency.
* **Parameters:**
* **model** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – A Gemini embedding model name (a leading `models/` is
tolerated and stripped).
* **dim** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Matryoshka-truncated width (`gemini-embedding-001` only). Omit
for the model’s native width. Note that Gemini, unlike OpenAI, does
*not* re-normalize MRL-truncated output — a truncated embedder
therefore reports `normalized=False`.
* **api_key** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Gemini API key. Falls back to `GEMINI_API_KEY` /
`GOOGLE_API_KEY`.
* **batch_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Texts per request (capped at Gemini’s 100 limit).
* **timeout** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Per-request timeout in seconds.
* **Return type:**
[`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)
* **Returns:**
An [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) over the Gemini API.
### ef.hierarchical(segmenters,)
Compose segmenters into levels — each segments the previous level’s output.
Level 0 applies `segmenters[0]` to the document. Level *k* applies
`segmenters[k]` to every level- *(k-1)* segment’s text, setting each
child’s `parent_id` to that segment’s `id` and shifting its offsets to
stay absolute. The result is a flat stream of every segment of every level
(level 0 first), with the tree expressed purely through `parent_id`
pointers.
Each entry of `segmenters` is passed through
[`as_segmenter()`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.as_segmenter), so strings (e.g. `"recursive"`)
and bare callables are accepted alongside ready-made segmenters.
* **Return type:**
[`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter)
### ef.hits_to_segments(hits)
Project [`SearchHit`](_autosummary/ef.html.md#ef.SearchHit)s to plain [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)s.
The shape [`SearchableCorpus.retrieve()`](_autosummary/ef.html.md#ef.SearchableCorpus.retrieve) / [`SourceManager.retrieve()`](_autosummary/ef.html.md#ef.SourceManager.retrieve)
hand to an external RAG/agent framework — the RAG-plug-in surface (design
notes §F5). A [`SearchHit`](_autosummary/ef.html.md#ef.SearchHit) carries an `ef`-specific `score` and
`source_id`; a [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) is a plain `TypedDict` an
external framework already understands (and trivially adapts to a LangChain
`Document` / LlamaIndex `TextNode` / a Ragas `retrieved_contexts`
`list[str]` via `[s["text"] for s in segments]`).
The similarity `score` is dropped — [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) is the
interchange type and stays pure of a result-only key (call `search()`
when the score matters). Provenance is *not* lost: each hit’s `source_id`
is folded into the segment’s `metadata` under the conventional `"source"`
key (one of [`PROMOTED_METADATA_KEYS`](_autosummary/ef.segments.html.md#ef.segments.PROMOTED_METADATA_KEYS)), so a plain segment
still records which source document it was retrieved from. A segment that
already carries `metadata["source"]` keeps its own value.
* **Return type:**
[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]
```pycon
>>> hit = SearchHit(segment={'text': 'hi', 'id': 'x'}, score=0.9, source_id='doc-1')
>>> segs = hits_to_segments([hit])
>>> segs[0]['text'], segs[0]['metadata']['source']
('hi', 'doc-1')
>>> 'score' in segs[0] # the score is not a Segment key
False
```
### ef.http_embedder(url, \*, model_id=None, dim=None, normalized=False, batch_size=64, headers=None, timeout=30.0, payload_builder=, response_parser=)
Build an [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) over a remote HTTP service.
Defaults target the Text-Embeddings-Inference / infinity shape (a POST of
`{"inputs": [...]}` returning a JSON list of vectors). For a different
service, override `payload_builder` and/or `response_parser`.
* **Parameters:**
* **url** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The embedding endpoint.
* **model_id** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Vector-identity string (defaults to `"http:"`).
* **dim** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Output width; inferred from the first response if omitted.
* **normalized** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether the service returns unit vectors.
* **batch_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Texts per request (used with length-sorted batching).
* **headers** ([`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Extra HTTP headers (e.g. an auth token).
* **timeout** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Per-request timeout in seconds.
* **payload_builder** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Sequence`](https://docs.python.org/3/library/typing.html#typing.Sequence)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]], [`bytes`](https://docs.python.org/3/builtins/stdtypes.html#bytes)]) – `texts -> request body bytes`.
* **response_parser** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – `decoded JSON -> array-like (n, dim)`.
* **Return type:**
[`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)
Uses only the stdlib — no extra dependency.
### ef.imbed_segmenter(name='default', , \*\*kwargs)
Build a [`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter) from `imbed`’s segmenter registry.
`imbed`’s registered segmenters are bare `text -> Iterable[str]`
callables; this wraps the named one in a [`FunctionSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.FunctionSegmenter)
so it emits well-formed [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) pieces.
* **Parameters:**
* **name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – A key in `imbed.components.segmentation.segmenters` (`"default"`
resolves to `imbed`’s configured default segmenter).
* **\*\*kwargs** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Forwarded to [`FunctionSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.FunctionSegmenter).
* **Raises:**
* [**ImportError**](https://docs.python.org/3/builtins/exceptions.html#ImportError) – if `imbed` is not installed.
* [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if `name` is not a registered segmenter.
* **Return type:**
[`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter)
Requires the `imbed` package (`pip install 'ef[imbed]'`).
### ef.ingest(sources, , segmenter=None, embedder=None, store=None, cache=None)
Index `sources` and return a ready-to-search [`SearchableCorpus`](_autosummary/ef.html.md#ef.SearchableCorpus) — one call.
The light path: corpus in, `search`-ready object out, every component
defaulted. `ingest(['a', 'b', 'c']).search('query')` is the whole story.
It builds a single-config [`SourceManager`](_autosummary/ef.html.md#ef.SourceManager), [`materialize()`](_autosummary/ef.html.md#ef.SourceManager.materialize)s
it, and hands back the thin [`SearchableCorpus`](_autosummary/ef.html.md#ef.SearchableCorpus).
* **Parameters:**
* **sources** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the corpus — coerced by [`as_corpus()`](_autosummary/ef.corpus.html.md#ef.corpus.as_corpus) (a mapping,
a directory path, an iterable of strings/mappings, or `None`).
* **segmenter** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the segmenter — [`as_segmenter()`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.as_segmenter)
coerces it (`None` → the recursive default).
* **embedder** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the embedder — [`as_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.as_embedder)
coerces it; `None` → `DEFAULT_EMBEDDER`, the dependency-free
[`HashingEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.HashingEmbedder) (so this call needs nothing
beyond `pip install ef`). Pass an explicit embedder — e.g.
`"st:all-MiniLM-L6-v2"` — for neural semantic quality.
* **store** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the vector store — `None` → an in-memory `vd` backend. See
[`SourceManager`](_autosummary/ef.html.md#ef.SourceManager) for the other accepted forms.
* **cache** ([`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the artifact-graph value cache — any `MutableMapping`.
* **Return type:**
[`SearchableCorpus`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus)
* **Returns:**
a [`SearchableCorpus`](_autosummary/ef.html.md#ef.SearchableCorpus) over the freshly indexed corpus.
See the module docstring for a complete offline example.
### ef.label_clusters(segments, labels, , context=' ', n_words=4, n_samples=None, \*\*labeler_kwargs)
Name each cluster with a short LLM-generated title (use cases §G3).
A thin wrapper over `imbed`’s `imbed.tools.ClusterLabeler`: it
samples each cluster’s segments, asks an LLM for a few-word title, and
returns `{cluster_id: title}`. `ef` does not reimplement this — it
reuses `imbed` (design notes §7).
* **Parameters:**
* **segments** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – the segment texts — plain strings or [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)
mappings (the `"text"` field is used). Aligned with `labels`.
* **labels** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – the cluster id of each segment — typically the output of
[`cluster()`](_autosummary/ef.html.md#ef.cluster).
* **context** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the corpus’s overall topic, given to the LLM so titles
describe how a cluster *differs* from the rest, not the shared topic.
* **n_words** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – maximum title length, in words.
* **n_samples** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – how many segments to sample per cluster when prompting;
`None` keeps `ClusterLabeler`’s default.
* **\*\*labeler_kwargs** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – forwarded to `imbed.tools.ClusterLabeler`
(e.g. `max_unique_clusters`, `prompt`).
* **Return type:**
[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`int`](https://docs.python.org/3/builtins/functions.html#int), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]
* **Returns:**
`{cluster_id: title}` for every distinct label.
* **Raises:**
* [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if `segments` and `labels` differ in length.
* [**ImportError**](https://docs.python.org/3/builtins/exceptions.html#ImportError) – if `imbed` (and its `pandas`/`oa` deps) is missing.
```pycon
>>> # needs ef[imbed] + an LLM key — illustrative only:
>>> # label_clusters(['neural net training ...', ...], [0, 1, 0, ...])
>>> # -> {0: 'Neural network training', 1: 'Dataset preprocessing'}
```
### ef.line_segmenter(doc,)
Segment a document into its non-empty lines.
Each line is stripped of surrounding whitespace; blank lines are dropped.
Character offsets point at the *stripped* content within the source.
A plain function — and therefore already a [`Segmenter`](_autosummary/ef.html.md#ef.Segmenter). It carries
the marker [`as_segmenter()`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.as_segmenter) uses to recognize a
ready-made segmenter.
* **Return type:**
[`Iterator`](https://docs.python.org/3/library/typing.html#typing.Iterator)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]
```pycon
>>> [s['text'] for s in line_segmenter(' hello \n\nworld\n')]
['hello', 'world']
```
### ef.make_segment(text, , id=None, parent_id=None, start=None, end=None, index=None, tokens=None, metadata=None)
Build a [`Segment`](_autosummary/ef.html.md#ef.Segment), deriving a content-based `id` if none given.
Optional fields are omitted from the result when `None` — a segment
`dict` carries only the keys it actually has.
* **Return type:**
[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)
```pycon
>>> seg = make_segment('chunk one', index=0, start=0, end=9)
>>> sorted(seg)
['end', 'id', 'index', 'start', 'text']
>>> seg['id'] == segment_id('chunk one')
True
```
### ef.materialise(x)
Force a streaming segmenter (or segment stream) to a concrete list.
Streaming is the default, but the light case often wants everything in
hand. [`materialise()`](_autosummary/ef.html.md#ef.materialise) is polymorphic:
- given a **segmenter** (a callable), it returns a new segmenter whose
output is a `list` rather than a lazy iterator;
- given an **iterable of segments**, it returns a `list` of them.
```pycon
>>> eager = materialise(line_segmenter)
>>> out = eager('a\nb')
>>> isinstance(out, list)
True
>>> [s['text'] for s in out]
['a', 'b']
>>> materialise(s for s in [{'text': 'x'}])
[{'text': 'x'}]
```
* **Return type:**
[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)
### ef.ndcg_at_k(ranked, relevant, k)
Normalized DCG at cutoff `k` — `ef`’s **primary** retrieval metric.
[`dcg_at_k()`](_autosummary/ef.html.md#ef.dcg_at_k) divided by the ideal DCG (the DCG of the best possible
ranking of the judged docs). `1.0` is a perfect ranking; `0.0` when no
relevant doc was judged.
* **Return type:**
[`float`](https://docs.python.org/3/builtins/functions.html#float)
```pycon
>>> ndcg_at_k(['d1', 'd2'], {'d1': 1.0, 'd2': 1.0}, 2) # perfect order
1.0
>>> round(ndcg_at_k(['x', 'd1'], {'d1': 1.0}, 2), 4) # relevant doc at rank 2
0.6309
```
### ef.openai_embedder(model='text-embedding-3-small', , dim=None, api_key=None, client=None, batch_size=2048, poll_interval=30.0)
Build an OpenAI-embeddings [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder).
* **Parameters:**
* **model** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – An OpenAI embedding model name.
* **dim** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Matryoshka-truncated output width (`text-embedding-3-*` only).
Omit for the model’s native dimension. The dim is part of the
embedder’s identity — set it once here, not per call.
* **api_key** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – OpenAI API key (else the SDK’s usual env-var resolution).
* **client** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – A pre-built `openai.OpenAI` client (overrides `api_key`).
* **batch_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Texts per request (capped at OpenAI’s 2048 limit).
* **poll_interval** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Seconds between polls when blocking on a Batch-API job.
* **Return type:**
[`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)
* **Returns:**
An embedder whose `embed_batch` uses the async OpenAI Batch API.
Requires the `openai` package (`pip install 'ef[openai]'`).
### ef.plan_refresh(report, , mode='full', scope=None)
Turn a [`StalenessReport`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.StalenessReport) into a [`RefreshPlan`](_autosummary/ef.html.md#ef.RefreshPlan).
The pure, side-effect-free core of the refresh policy. Every source that is
`missing`, `stale` or `misconfigured` is (re-)materialized; what gets
*deleted* is what the `mode` dictates:
- `none` — delete nothing (stale content is added alongside the old);
- `incremental` — delete the stale/misconfigured sources’ old documents;
- `full` / `scoped_full` — also delete orphans.
* **Parameters:**
* **report** ([`StalenessReport`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.StalenessReport)) – the staleness report to act on.
* **mode** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'none'`, `'incremental'`, `'full'`, `'scoped_full'`]) – one of `REFRESH_MODES`.
* **scope** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – an optional subset of source ids the plan is restricted to —
sources outside it are left entirely alone. `None` means the whole
report. (With `scope=None`, `full` and `scoped_full` coincide.)
* **Raises:**
[**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if `mode` is not a recognized `RefreshMode`.
* **Return type:**
[`RefreshPlan`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshPlan)
```pycon
>>> from ef.diagnostics import StalenessReport
>>> report = StalenessReport(config='c0', orphan=('x',), missing=('a',),
... stale=('b',), misconfigured=('c',), fresh=('d',))
>>> plan_refresh(report, mode='full').to_materialize
('a', 'b', 'c')
>>> plan_refresh(report, mode='full').to_delete
('b', 'c', 'x')
>>> plan_refresh(report, mode='incremental').to_delete
('b', 'c')
>>> plan_refresh(report, mode='none').to_delete
()
>>> plan_refresh(report, mode='scoped_full', scope=['a', 'b']).to_delete
('b',)
```
### ef.precision_at_k(ranked, relevant, k)
Fraction of the top `k` results that are relevant.
* **Return type:**
[`float`](https://docs.python.org/3/builtins/functions.html#float)
```pycon
>>> precision_at_k(['d1', 'x', 'd2'], {'d1': 1.0, 'd2': 1.0}, 2)
0.5
```
### ef.producer_spec(op, \*inputs, op_version, \*\*params)
Build a [`ProducerSpec`](_autosummary/ef.html.md#ef.ProducerSpec) ergonomically — inputs positional, params keyword.
The natural way to write a recipe: the input artifact ids are positional
(their order is significant — it is the order the op receives them), and
every keyword argument is an op parameter. `op_version` is keyword-only
and required — an op without a stated version cannot be content-addressed
safely.
* **Return type:**
[`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec)
```pycon
>>> spec = producer_spec('segment', 'doc-1', op_version='1', size=512, overlap=64)
>>> spec.op, spec.inputs
('segment', ('doc-1',))
>>> spec.params == {'size': 512, 'overlap': 64}
True
```
### ef.project(data, , dims=2, method='auto', embedder=None, n_neighbors=15, min_dist=0.1, metric='cosine', pca_components=50, random_state=42)
Project a corpus’s embeddings to `dims` coordinates for visualization.
The canonical “show me the shape of the corpus” operation (use cases §G1).
The default `method='auto'` runs the design-notes §8.4 recipe — \*\*PCA →
UMAP\*\*, `cosine` metric, seeded: PCA first compresses high-dimensional
embeddings to `pca_components` (cheap denoising), then UMAP lays them out
in `dims` dimensions.
* **Parameters:**
* **data** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – what to project — anything `_resolve_vectors()` understands:
an `ndarray` (or nested sequence) of vectors, an iterable of
texts (embedded with `embedder`), a [`Corpus`](_autosummary/ef.corpus.html.md#ef.corpus.Corpus)
mapping, or a [`SearchableCorpus`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus) /
[`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager) (vectors pulled from the
index).
* **dims** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – target dimensionality — `2` or `3` for plotting.
* **method** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'auto'`, `'umap'`, `'pca'`]) – `'auto'` (UMAP when available and the corpus is large enough,
else PCA), `'umap'` (force UMAP — needs the `ef[explore]`
extra), or `'pca'` (numpy-only, always available).
* **embedder** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – embedder used when `data` is raw text; defaults to the
dependency-free [`DEFAULT_EMBEDDER`](_autosummary/ef.source_manager.html.md#ef.source_manager.DEFAULT_EMBEDDER).
* **n_neighbors** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – UMAP neighborhood size (clamped to `n_samples - 1`).
* **min_dist** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – UMAP minimum point separation in the embedding.
* **metric** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – UMAP distance metric — `'cosine'` for semantic embeddings.
* **pca_components** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – PCA width for the pre-UMAP compression step; PCA is
skipped when the source dimensionality is already smaller.
* **random_state** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – seed — projection is reproducible.
* **Return type:**
`ndarray`
* **Returns:**
An `ndarray` of shape `(n_samples, dims)`, rows in input order.
* **Raises:**
* [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if there are fewer than 2 samples.
* [**ImportError**](https://docs.python.org/3/builtins/exceptions.html#ImportError) – if `method='umap'` and `umap-learn` is not installed.
```pycon
>>> import numpy as np
>>> vectors = np.random.RandomState(1).rand(20, 32)
>>> project(vectors, dims=3, method='pca').shape
(20, 3)
```
### ef.read_beir(directory, , qrels=None)
Load a BEIR-format dataset from disk into the in-memory evaluation triple.
Reads `corpus.jsonl` (records `{"_id", "title", "text"}`),
`queries.jsonl` (records `{"_id", "text"}`) and a `qrels` TSV
(`query-id`, `corpus-id`, `score` columns). The corpus text is the
record’s `title` and `text` joined — the BEIR convention.
* **Parameters:**
* **directory** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the dataset directory.
* **qrels** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the qrels file, relative to `directory`. `None` tries
`_DEFAULT_QRELS_PATHS` in order.
* **Return type:**
[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`float`](https://docs.python.org/3/builtins/functions.html#float)]]]
* **Returns:**
a `(corpus, queries, qrels)` triple — `corpus` and `queries` are
`{id: text}` dicts, `qrels` is `{query_id: {doc_id: relevance}}`.
Feed it straight to [`evaluate_retrieval()`](_autosummary/ef.html.md#ef.evaluate_retrieval) (after indexing `corpus`
with [`ingest()`](_autosummary/ef.source_manager.html.md#ef.source_manager.ingest)).
* **Raises:**
[**FileNotFoundError**](https://docs.python.org/3/builtins/exceptions.html#FileNotFoundError) – if `corpus.jsonl`, `queries.jsonl` or the qrels
file is missing.
### ef.ready_handle(array)
Wrap an already-computed array in a finished [`BatchHandle`](_autosummary/ef.html.md#ef.BatchHandle).
* **Return type:**
[`BatchHandle`](_autosummary/ef.embedders.html.md#ef.embedders.BatchHandle)
```pycon
>>> import numpy as np
>>> h = ready_handle(np.zeros((2, 3)))
>>> h.poll()
'done'
>>> h.result().shape
(2, 3)
```
### ef.recall_at_k(ranked, relevant, k)
Fraction of the relevant docs that appear in the top `k` results.
* **Return type:**
[`float`](https://docs.python.org/3/builtins/functions.html#float)
```pycon
>>> recall_at_k(['d1', 'x', 'd2'], {'d1': 1.0, 'd2': 1.0}, 2)
0.5
>>> recall_at_k(['d1', 'x', 'd2'], {'d1': 1.0, 'd2': 1.0}, 3)
1.0
```
### ef.reciprocal_rank(ranked, relevant, k=None)
Reciprocal of the rank of the first relevant result (`0` if none).
Averaged over queries this is MRR. `k` optionally caps how far down the
ranking to look.
* **Return type:**
[`float`](https://docs.python.org/3/builtins/functions.html#float)
```pycon
>>> reciprocal_rank(['x', 'd1', 'd2'], {'d1': 1.0})
0.5
>>> reciprocal_rank(['x', 'd1'], {'d1': 1.0}, k=1) # relevant doc past the cutoff
0.0
```
### ef.refresh_on_change(manager)
Build a [`ChangeEvent`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeEvent) handler that auto-refreshes `manager`.
The *auto* refresh seam. The returned callable is meant to be a
[`ChangeDetectingCorpus`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus)’s `on_change` — each detected
edit (through the wrapper, or surfaced out-of-band by `scan()`) is
incrementally applied to every materialized config of `manager`:
an `added` source is indexed, a `modified` one is re-indexed
(delete-then-add), a `deleted` one is removed.
[`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager) wires this automatically when
constructed with `auto_refresh=True`; call this directly only to attach a
manager to a corpus you build yourself.
* **Parameters:**
**manager** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – a [`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager) (duck-typed — any
object with an `_apply_change(event)` method).
* **Return type:**
[`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`ChangeEvent`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeEvent)], [`None`](https://docs.python.org/3/builtins/constants.html#None)]
* **Returns:**
a `Callable[[ChangeEvent], None]` for `on_change`.
### ef.rerank(query, segments, reranker, , limit=None)
Re-score `segments` against `query` with `reranker` and re-order them.
The pure heart of the module — no I/O, no retrieval. The `reranker` is
called once with the whole candidate list; the segments are returned sorted
by descending score (ties keep their incoming order), optionally trimmed to
`limit`. Each returned segment is a copy carrying the reranker’s score in
`metadata["rerank_score"]` — so the new ranking is auditable.
* **Parameters:**
* **query** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the search query the segments are scored against.
* **segments** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]) – the candidate segments (from a first-stage retriever).
* **reranker** ([`Reranker`](_autosummary/ef.reranking.html.md#ef.reranking.Reranker)) – a [`Reranker`](_autosummary/ef.html.md#ef.Reranker) — a callable `(query, segments) -> scores`.
* **limit** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – keep only the top `limit` after reranking; `None` keeps all.
* **Return type:**
[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]
* **Returns:**
the reranked segments — copies, best first.
* **Raises:**
[**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if the reranker returns a number of scores that does not
match the number of segments.
```pycon
>>> segs = [{'text': 'a', 'id': '1'}, {'text': 'bbb', 'id': '2'}]
>>> ranked = rerank('q', segs, lambda q, s: [len(x['text']) for x in s])
>>> [r['text'] for r in ranked]
['bbb', 'a']
>>> ranked[0]['metadata']['rerank_score']
3.0
```
### ef.segment_id(text, metadata=None)
Content-derived id for a segment — `sha256` of its normalized content.
The id is `sha256(normalize(text) + canonical_json(metadata))` — the
normalization and hashing primitives live in [`ef.hashing`](_autosummary/ef.hashing.html.md#module-ef.hashing), the single
source of truth `ef` content-addresses through. Because the id depends
only on content, re-segmenting an unchanged document produces identical ids
— which is what makes ingestion idempotent (a re-run is a no-op, not a
duplicate). Pass `metadata` only when two segments could share identical
`text` yet must stay distinct (e.g. same boilerplate from different
sources).
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
```pycon
>>> segment_id('hello') == segment_id('hello')
True
>>> segment_id('hello') == segment_id('world')
False
>>> segment_id('x', {'source': 'a'}) == segment_id('x', {'source': 'b'})
False
```
### ef.segment_record(x)
Coerce `x` (`str` / `Mapping` / [`SegmentRecord`](_autosummary/ef.html.md#ef.SegmentRecord)) into a record.
The dataclass counterpart of [`as_segment()`](_autosummary/ef.html.md#ef.as_segment) — the convenience-surface
constructor.
* **Return type:**
[`SegmentRecord`](_autosummary/ef.segments.html.md#ef.segments.SegmentRecord)
```pycon
>>> segment_record('hello').text
'hello'
>>> segment_record({'text': 'hi', 'tokens': 1}).tokens
1
```
### ef.sentence_transformers_embedder(model_name='all-MiniLM-L6-v2', , device=None, normalize=False, batch_size=128, model_path=None, query_prompt=None, document_prompt=None, prompt_names=None, \*\*st_kwargs)
Build a local sentence-transformers [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder).
Instruction-tuned models (bge, gte, e5-instruct, Qwen3-Embedding…) are
conditioned per `input_type`: pass a literal `query_prompt` (and, rarely,
`document_prompt`), or map input types to prompts the model defines itself
with `prompt_names` (e.g. `{"query": "query"}` for Qwen3-Embedding). Those
roles are then advertised in `honored_input_types`, and the prompt text is
baked into `model_id`. Only trust prompts on genuinely instruction-tuned
models; on `all-MiniLM` they are expected to be a near no-op. Prefer these
over wrapping the adapter in [`InstructionEmbedder`](_autosummary/ef.embedder_wrappers.html.md#ef.embedder_wrappers.InstructionEmbedder):
sentence-transformers knows the prompt’s length, so models that pool without
the prompt tokens (INSTRUCTOR, GRIT) stay correct. Needs
sentence-transformers >= 2.3.
* **Parameters:**
* **model_name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – A sentence-transformers model name (also used in
`model_id`).
* **device** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Torch device (`"cuda"` / `"cpu"` / `"mps"`); `None`
auto-selects.
* **normalize** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to L2-normalize outputs. sentence-transformers does
**not** normalize by default — set this truthfully.
* **batch_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Encoding batch size (smaller on CPU).
* **model_path** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – A local filesystem path to load from instead of the hub —
for air-gapped use. Loading still records `model_name` in
`model_id`.
* **query_prompt** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Prompt prepended to `input_type="query"` texts (passed
as `encode(prompt=...)`).
* **document_prompt** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Prompt for `input_type="document"` texts (usually
none: the asymmetric pattern leaves documents bare). `""` means an
explicit empty prompt, which stops a model’s `default_prompt_name`
from being applied to documents.
* **prompt_names** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Mapping`](https://docs.python.org/3/library/typing.html#typing.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]]) – `{input_type: name}` of prompts defined in the model’s
own `prompts` config (passed as `encode(prompt_name=...)`).
An explicit `prompt=` / `prompt_name=` in a call overrides both.
* **\*\*st_kwargs** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Forwarded to the `SentenceTransformer` constructor.
* **Return type:**
[`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)
Requires `sentence-transformers` (`pip install 'ef[sentence-transformers]'`).
### ef.step_params(op, raw_params=None, , n_inputs=1)
The canonical *keyword* params of an op step — its inputs dropped.
An [`ef.ops`](_autosummary/ef.ops.html.md#module-ef.ops) op is called `op(*input_values, **params)`: the first
`n_inputs` positional parameters are upstream artifacts (they belong in
[`ProducerSpec.inputs`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec.inputs), not in
the params), and every remaining parameter is a hashable step param. This
runs [`full_kwargs()`](_autosummary/ef.html.md#ef.full_kwargs) with placeholder inputs, then drops those input
slots — leaving exactly the params, defaults filled, ready for a
[`TransformSpec`](_autosummary/ef.html.md#ef.TransformSpec).
* **Parameters:**
* **op** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – the op callable whose signature is the param contract.
* **raw_params** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the keyword params the caller actually supplied; anything
omitted is defaulted.
* **n_inputs** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – how many leading positional parameters are upstream inputs.
* **Return type:**
[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]
```pycon
>>> def embed(segments, *, input_type='document'): ...
>>> step_params(embed, {'input_type': 'query'})
{'input_type': 'query'}
>>> step_params(embed) # default filled in
{'input_type': 'document'}
>>> def segment(source): ...
>>> step_params(segment) # no params at all
{}
```
### ef.token_f1(response, reference)
Token-overlap F1 of `response` against `reference` (the SQuAD F1).
The harmonic mean of token precision and recall over the normalized token
bags — a graded credit where [`exact_match()`](_autosummary/ef.html.md#ef.exact_match) is all-or-nothing.
* **Return type:**
[`float`](https://docs.python.org/3/builtins/functions.html#float)
```pycon
>>> token_f1('the quick brown fox', 'a quick brown fox')
1.0
>>> round(token_f1('quick brown fox', 'quick brown dog'), 4)
0.6667
```
### ef.voyage_embedder(model='voyage-3.5', , dim=None, api_key=None, batch_size=1000, timeout=60.0)
Build a Voyage AI [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder).
Talks to Voyage’s `/v1/embeddings` REST endpoint directly — no
`voyageai` SDK, no extra dependency. Voyage embeddings are L2-normalized.
Voyage distinguishes only `query` from `document`; a `classification`
or `clustering` hint is simply not sent.
* **Parameters:**
* **model** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – A Voyage embedding model name.
* **dim** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Matryoshka-truncated width (the `voyage-3.5` family and
`voyage-code-3` only). Omit for the model’s native width.
* **api_key** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Voyage API key. Falls back to `VOYAGE_API_KEY`.
* **batch_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Texts per request (capped at Voyage’s 1000 limit).
* **timeout** ([`float`](https://docs.python.org/3/builtins/functions.html#float)) – Per-request timeout in seconds.
* **Return type:**
[`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)
* **Returns:**
An [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) over the Voyage API.
### ef.with_overlap(segmenter, n_chars, /, \*, count_tokens=, tokenizer='char-approx')
Wrap `segmenter` so consecutive segments overlap by `n_chars` characters.
Each segment is extended *forward* by up to `n_chars` characters (clamped
to the source length), so segment *K* shares its tail with segment *K+1*’s
head. A segment whose text changes has its `id` re-derived and its
`tokens` recounted; segments lacking `start` / `end` offsets pass
through untouched.
Overlap is expressed in *characters* here (this is a generic post-hoc
helper) — for token-measured overlap use a segmenter’s own `chunk_overlap`.
* **Return type:**
[`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter)
```pycon
>>> base = RecursiveCharacterSegmenter(chunk_size=3, chunk_overlap=0)
>>> [c['text'] for c in base('one two three four')]
['one two three', 'four']
>>> [c['text'] for c in with_overlap(base, 100)('one two three four')]
['one two three four', 'four']
```
### ef.with_reranker(retriever, reranker, , fetch_k=50)
Decorate a retrieve-style callable with a two-stage reranking pass.
Wraps `retriever` (e.g. [`SourceManager.retrieve`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.retrieve), or any callable
`query -> list[Segment]`) so every call over-fetches `fetch_k`
candidates cheaply, [`rerank()`](_autosummary/ef.html.md#ef.rerank)s them, and returns the top `limit`.
The decorated callable keeps the retriever’s interface — `query` plus a
keyword-only `limit` — so it is a drop-in replacement, including as the
`retriever` argument of [`evaluate_retrieval()`](_autosummary/ef.evaluation.html.md#ef.evaluation.evaluate_retrieval).
* **Parameters:**
* **retriever** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]]) – the first-stage retriever — a callable
`retriever(query, *, limit=...) -> segments`.
* **reranker** ([`Reranker`](_autosummary/ef.reranking.html.md#ef.reranking.Reranker)) – the [`Reranker`](_autosummary/ef.html.md#ef.Reranker) applied to the over-fetched candidates.
* **fetch_k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – how many candidates to retrieve before reranking. Larger trades
latency for recall headroom; it should comfortably exceed the
`limit` callers ask for.
* **Return type:**
[`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]]
* **Returns:**
a new callable `(query, *, limit=10, **kwargs) -> list[Segment]` —
extra keyword arguments (e.g. `filter`) pass straight through to
`retriever`.
```pycon
>>> pool = [{'text': t, 'id': t} for t in ['ab', 'abcd', 'abc']]
>>> def base(query, *, limit=10):
... return pool[:limit]
>>> retrieve = with_reranker(
... base, lambda q, segs: [len(s['text']) for s in segs], fetch_k=10)
>>> [s['text'] for s in retrieve('a query', limit=2)]
['abcd', 'abc']
```
### ef.write_beir(directory, corpus, queries, qrels)
Write an evaluation triple to disk in BEIR format — the inverse of [`read_beir()`](_autosummary/ef.html.md#ef.read_beir).
Creates `directory` (and parents) and writes `corpus.jsonl`,
`queries.jsonl` and `qrels.tsv`. Use it to snapshot an `ef` corpus as
a reusable benchmark (`ef_use_cases.md` §H4).
* **Parameters:**
* **directory** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the output directory — created if absent.
* **corpus** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – the documents — `{doc_id: text}`.
* **queries** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – the queries — `{query_id: query_text}`.
* **qrels** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`float`](https://docs.python.org/3/builtins/functions.html#float)]]) – the relevance judgements — `{query_id: {doc_id: relevance}}`.
* **Return type:**
[`None`](https://docs.python.org/3/builtins/constants.html#None)
### Modules
| [`artifact_graph`](_autosummary/ef.artifact_graph.html.md#module-ef.artifact_graph) | The `ArtifactGraph` — `ef`'s content-addressed corpus-indexing core. |
|--------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [`config`](_autosummary/ef.config.html.md#module-ef.config) | The config layer — declarative, content-addressable pipeline specs. |
| [`corpus`](_autosummary/ef.corpus.html.md#module-ef.corpus) | The `Corpus` abstraction — `ef`'s source layer (L0) — and change detection. |
| [`diagnostics`](_autosummary/ef.diagnostics.html.md#module-ef.diagnostics) | Staleness diagnostics — has an indexed corpus drifted from its source? |
| [`embedder_adapters`](_autosummary/ef.embedder_adapters.html.md#module-ef.embedder_adapters) | Concrete [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder) adapters and the DI seam. |
| [`embedder_wrappers`](_autosummary/ef.embedder_wrappers.html.md#module-ef.embedder_wrappers) | Composition wrappers for [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder). |
| [`embedders`](_autosummary/ef.embedders.html.md#module-ef.embedders) | The `Embedder` facade — `ef`'s embed layer (L3). |
| [`evaluation`](_autosummary/ef.evaluation.html.md#module-ef.evaluation) | Evaluation hookpoints — measuring retrieval and RAG quality. |
| [`exploration`](_autosummary/ef.exploration.html.md#module-ef.exploration) | Layer L5 — *explore* a corpus: project to 2-D/3-D, cluster, label clusters. |
| [`hashing`](_autosummary/ef.hashing.html.md#module-ef.hashing) | Content-hashing primitives shared across `ef`'s layers. |
| [`ops`](_autosummary/ef.ops.html.md#module-ef.ops) | The pipeline ops — the callables an [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph) runs. |
| [`refresh`](_autosummary/ef.refresh.html.md#module-ef.refresh) | Refresh — keeping an indexed corpus in sync as its sources change. |
| [`reranking`](_autosummary/ef.reranking.html.md#module-ef.reranking) | Reranking — a precision pass over retrieved segments (use case F6). |
| [`segmenter_adapters`](_autosummary/ef.segmenter_adapters.html.md#module-ef.segmenter_adapters) | The segmenter dependency-injection seam and the `imbed` registry bridge. |
| [`segmenters`](_autosummary/ef.segmenters.html.md#module-ef.segmenters) | The `Segmenter` facade — `ef`'s segment layer (L2). |
| [`segments`](_autosummary/ef.segments.html.md#module-ef.segments) | The canonical `Segment` data model — `ef`'s interchange type for L2. |
| [`service`](_autosummary/ef.service.html.md#module-ef.service) | The HTTP-service bridge — [`EfService`](_autosummary/ef.html.md#ef.EfService), a stateless-friendly facade. |
| [`source_manager`](_autosummary/ef.source_manager.html.md#module-ef.source_manager) | The search facade — one-shot [`ingest()`](_autosummary/ef.html.md#ef.ingest), [`SearchableCorpus`](_autosummary/ef.html.md#ef.SearchableCorpus), [`SourceManager`](_autosummary/ef.html.md#ef.SourceManager). |
# _autosummary/ef.ops.html.md
# ef.ops
The pipeline ops — the callables an [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph) runs.
The [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph) resolves a [`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec)’s string `op` key to a callable through its
injectable `ops` registry, then calls `op(*input_values, **params)`. This
module supplies the two ops Phase 5’s pipeline needs and the machinery that
turns a *live component* (a [`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter) /
[`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)) into a serializable [`TransformSpec`](_autosummary/ef.config.html.md#ef.config.TransformSpec)
*plus* the bound callable to register.
The split is deliberate:
- a [`TransformSpec`](_autosummary/ef.config.html.md#ef.config.TransformSpec) is **data** — an op *key*, a version,
params — so it can be persisted (the recipe is content-addressed);
- the callable is **not** data — it closes over a live component — so it is
re-registered into the graph’s `ops` registry each process.
The op key carries the **component identity** ([`segmenter_identity()`](_autosummary/ef.ops.html.md#ef.ops.segmenter_identity),
`embedder.model_id`): `"segment:"` / `"embed:"`.
Because `op` participates in [`artifact_id()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.artifact_id), two
configs with different components get disjoint artifact cones automatically,
and two configs with the *same* component share every upstream artifact — which
is what makes config branching free (design notes §3.1).
```pycon
>>> import numpy as np
>>> from ef import as_embedder, as_segmenter
>>> seg = as_segmenter('recursive', chunk_size=128)
>>> seg_spec, seg_fn = segment_step(seg)
>>> seg_spec.op.startswith('segment:recursive-char')
True
>>> [s['text'] for s in seg_fn('hello world')]
['hello world']
>>> emb = as_embedder(lambda ts: np.ones((len(ts), 4)), model_id='ones@4')
>>> emb_spec, emb_fn = embed_step(emb)
>>> emb_spec.op, emb_spec.params
('embed:ones@4', {'input_type': 'document'})
>>> emb_fn([{'text': 'a'}, {'text': 'b'}]).shape
(2, 4)
```
### Module Attributes
| [`SEGMENT_OP_VERSION`](_autosummary/ef.ops.html.md#ef.ops.SEGMENT_OP_VERSION) | Version of the [`make_segment_op()`](_autosummary/ef.ops.html.md#ef.ops.make_segment_op) wrapper. |
|-----------------------------------------------------------------------|----------------------------------------------------------------------------------------------|
| [`EMBED_OP_VERSION`](_autosummary/ef.ops.html.md#ef.ops.EMBED_OP_VERSION) | Version of the [`make_embed_op()`](_autosummary/ef.ops.html.md#ef.ops.make_embed_op) wrapper. |
### Functions
| [`segmenter_identity`](_autosummary/ef.ops.html.md#ef.ops.segmenter_identity)(segmenter) | A stable identity string for a segmenter — the content-addressing key. |
|----------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------|
| [`embedder_identity`](_autosummary/ef.ops.html.md#ef.ops.embedder_identity)(embedder) | The content-addressing identity of an embedder — its `model_id`. |
| [`make_segment_op`](_autosummary/ef.ops.html.md#ef.ops.make_segment_op)(segmenter) | Build the **segment** op — a leaf source → its list of `Segment`s. |
| [`make_embed_op`](_autosummary/ef.ops.html.md#ef.ops.make_embed_op)(embedder) | Build the **embed** op — a list of `Segment`s → an `(n, dim)` array. |
| [`segment_step`](_autosummary/ef.ops.html.md#ef.ops.segment_step)(segmenter) | A segmenter → its [`TransformSpec`](_autosummary/ef.config.html.md#ef.config.TransformSpec) and bound op callable. |
| [`embed_step`](_autosummary/ef.ops.html.md#ef.ops.embed_step)(embedder) | An embedder → its [`TransformSpec`](_autosummary/ef.config.html.md#ef.config.TransformSpec) and bound op callable. |
### ef.ops.EMBED_OP_VERSION *= '1'*
Version of the [`make_embed_op()`](_autosummary/ef.ops.html.md#ef.ops.make_embed_op) wrapper. Bump on any behavior change —
see [`SEGMENT_OP_VERSION`](_autosummary/ef.ops.html.md#ef.ops.SEGMENT_OP_VERSION). (This is the *wrapper* version; the embedding
*model*’s identity already lives in the op key via `embedder.model_id`.)
### ef.ops.SEGMENT_OP_VERSION *= '1'*
Version of the [`make_segment_op()`](_autosummary/ef.ops.html.md#ef.ops.make_segment_op) wrapper. It participates in every
downstream [`artifact_id()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.artifact_id), so \*\*bump it on any
behavior change\*\* to how a segment artifact is produced — the graph trusts
the version and cannot inspect the code (a silent change serves stale cache).
### ef.ops.embed_step(embedder)
An embedder → its [`TransformSpec`](_autosummary/ef.config.html.md#ef.config.TransformSpec) and bound op callable.
The mirror of [`segment_step()`](_autosummary/ef.ops.html.md#ef.ops.segment_step) for the embed layer. The op key is
`"embed:"` — the model identity already content-addresses the
vector — and `params` carries the defaults-filled `input_type`.
* **Return type:**
[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`TransformSpec`](_autosummary/ef.config.html.md#ef.config.TransformSpec), [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]]
```pycon
>>> import numpy as np
>>> from ef import as_embedder
>>> spec, fn = embed_step(as_embedder(lambda ts: np.ones((len(ts), 2)), model_id='m@2'))
>>> spec.op, spec.params
('embed:m@2', {'input_type': 'document'})
```
### ef.ops.embedder_identity(embedder)
The content-addressing identity of an embedder — its `model_id`.
[`Embedder.model_id`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder.model_id) already bakes in
everything that changes the vector (provider, model, dimensionality), so it
*is* the identity — this function just names that fact for symmetry with
[`segmenter_identity()`](_autosummary/ef.ops.html.md#ef.ops.segmenter_identity).
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
```pycon
>>> from ef import as_embedder
>>> import numpy as np
>>> embedder_identity(as_embedder(lambda ts: np.ones((len(ts), 2)), model_id='m@2'))
'm@2'
```
### ef.ops.make_embed_op(embedder)
Build the **embed** op — a list of `Segment`s → an `(n, dim)` array.
The returned callable closes over `embedder` and embeds the segments’
`text` in caller order; row `i` of the result is the vector of segment
`i`. `input_type` is a hashable step param (it changes the vector for
task-aware embedders) — it defaults to `"document"`, the indexing role.
* **Return type:**
[`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]
```pycon
>>> import numpy as np
>>> from ef import as_embedder
>>> op = make_embed_op(as_embedder(lambda ts: np.ones((len(ts), 3)), model_id='m@3'))
>>> op([{'text': 'a'}, {'text': 'b'}]).shape
(2, 3)
>>> op([]).shape[0] # an empty doc embeds to zero rows
0
```
### ef.ops.make_segment_op(segmenter)
Build the **segment** op — a leaf source → its list of `Segment`s.
The returned callable closes over `segmenter`; it is what gets registered
into the [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph)’s `ops` registry under
the key [`segment_step()`](_autosummary/ef.ops.html.md#ef.ops.segment_step) computes. It materializes the (streaming)
segmenter output into a `list` — a graph artifact must be a concrete,
re-readable value.
* **Return type:**
[`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]]
```pycon
>>> from ef import as_segmenter
>>> op = make_segment_op(as_segmenter('lines'))
>>> [s['text'] for s in op('one\ntwo')]
['one', 'two']
```
### ef.ops.segment_step(segmenter)
A segmenter → its [`TransformSpec`](_autosummary/ef.config.html.md#ef.config.TransformSpec) and bound op callable.
The [`TransformSpec`](_autosummary/ef.config.html.md#ef.config.TransformSpec) is serializable data (op key + version
+ params); the callable closes over the live `segmenter` and is registered
into the graph’s `ops` registry under `spec.op`. Pair them: persist the
spec, re-register the callable each process.
```pycon
>>> from ef import as_segmenter
>>> spec, fn = segment_step(as_segmenter('recursive', chunk_size=128))
>>> spec.op_version, spec.params
('1', {})
```
* **Return type:**
[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`TransformSpec`](_autosummary/ef.config.html.md#ef.config.TransformSpec), [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]]
### ef.ops.segmenter_identity(segmenter)
A stable identity string for a segmenter — the content-addressing key.
The segmenter’s counterpart of [`Embedder.model_id`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder.model_id). Two segmenters that segment identically
must return the *same* identity; any difference that changes the segments
(chunk size, overlap, separators, tokenizer) must change it. Resolution
order:
1. an explicit `segmenter_id` attribute, if the segmenter advertises one;
2. a [`RecursiveCharacterSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.RecursiveCharacterSegmenter) — identified by its
config-bearing attributes (`chunk_size` / `chunk_overlap` /
`separators` / `tokenizer`);
3. a [`FunctionSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.FunctionSegmenter) — by its wrapped callable;
4. otherwise the segmenter’s own `__qualname__` / `__name__` / type name.
```pycon
>>> from ef import as_segmenter
>>> a = segmenter_identity(as_segmenter('recursive', chunk_size=256))
>>> b = segmenter_identity(as_segmenter('recursive', chunk_size=512))
>>> a == b # different chunk_size → different id
False
>>> a == segmenter_identity(as_segmenter('recursive', chunk_size=256))
True
```
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
# _autosummary/ef.refresh.html.md
# ef.refresh
Refresh — keeping an indexed corpus in sync as its sources change.
Phase 5 indexes a corpus *once*. This module makes the index **maintainable**:
when sources are added, edited or deleted, refresh brings the `vd` index back
into agreement with the corpus. It is the acting counterpart of the read-only
[`ef.diagnostics`](_autosummary/ef.diagnostics.html.md#module-ef.diagnostics) — diagnostics says *what drifted*, refresh *fixes it*.
The one idea, restated from [`ef.artifact_graph`](_autosummary/ef.artifact_graph.html.md#module-ef.artifact_graph):
> Refresh = diff source hashes → delete the stale/orphan documents →
> re-`materialize` what changed.
The `vd` collection is the single source of truth for *what is indexed*; the
[`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph) is only a *cache*. So refresh has two
kinds of effect, and they are not equally critical:
- **Collection edits are correctness.** Stale and orphaned documents must be
removed precisely (by `source_id`), or a search returns wrong hits.
- **Graph pruning is housekeeping.** A content-addressed artifact keyed by a
now-dead content hash is never *served* again (its source’s content — hence
its leaf id — has moved on); it is merely wasted cache. So [`prune_dead_leaves()`](_autosummary/ef.refresh.html.md#ef.refresh.prune_dead_leaves)
is best-effort and only ever cascades leaves verified dead — no surviving
source still hashes to them.
The four refresh modes (`ef_design_notes.md` §8.5, mirroring LangChain’s
indexing API) differ only in **how much they delete**:
[`plan_refresh()`](_autosummary/ef.refresh.html.md#ef.refresh.plan_refresh) turns a [`StalenessReport`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.StalenessReport) + a mode
into a [`RefreshPlan`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshPlan) (which sources to (re-)materialize, which documents
to delete) — a pure function, the testable heart of the policy. The orchestration
that applies a plan lives on [`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager)
([`refresh()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.refresh)).
**Explicit vs. auto.** [`SourceManager.refresh`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.refresh) is the *explicit* path — it re-reads
and re-hashes the corpus, so it catches out-of-band edits on its own.
[`refresh_on_change()`](_autosummary/ef.refresh.html.md#ef.refresh.refresh_on_change) is the *auto* path: a handler you hand to a
[`ChangeDetectingCorpus`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus)’s `on_change` so each edit is
incrementally re-indexed as it happens (`SourceManager(auto_refresh=True)`
wires this for you).
### Module Attributes
| [`RefreshMode`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshMode) | The four refresh modes — see the module docstring and `ef_design_notes.md` §8.5. |
|----------------------------------------------------------------|---------------------------------------------------------------------------------------------------|
| [`REFRESH_MODES`](_autosummary/ef.refresh.html.md#ef.refresh.REFRESH_MODES) | The accepted [`RefreshMode`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshMode) values, for validation. |
### Functions
| [`plan_refresh`](_autosummary/ef.refresh.html.md#ef.refresh.plan_refresh)(report, \*[, mode, scope]) | Turn a [`StalenessReport`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.StalenessReport) into a [`RefreshPlan`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshPlan). |
|----------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [`delete_source_documents`](_autosummary/ef.refresh.html.md#ef.refresh.delete_source_documents)(collection, indexed, ...) | Delete every `vd` document belonging to the given `source_ids`. |
| [`prune_dead_leaves`](_autosummary/ef.refresh.html.md#ef.refresh.prune_dead_leaves)(graph, live_leaves) | Cascade out of `graph` every leaf no live source still hashes to. |
| [`refresh_on_change`](_autosummary/ef.refresh.html.md#ef.refresh.refresh_on_change)(manager) | Build a [`ChangeEvent`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeEvent) handler that auto-refreshes `manager`. |
### Classes
| [`RefreshPlan`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshPlan)([to_materialize, to_delete]) | What a refresh will do — [`plan_refresh()`](_autosummary/ef.refresh.html.md#ef.refresh.plan_refresh)'s output. |
|---------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------|
| [`RefreshReport`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshReport)(config, mode[, added, ...]) | The outcome of a refresh — what `SourceManager.refresh()` returns. |
### ef.refresh.REFRESH_MODES *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[Literal](https://docs.python.org/3/library/typing.html#typing.Literal)['none', 'incremental', 'full', 'scoped_full'], ...]* *= ('none', 'incremental', 'full', 'scoped_full')*
The accepted [`RefreshMode`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshMode) values, for validation.
### ef.refresh.RefreshMode
The four refresh modes — see the module docstring and `ef_design_notes.md`
§8.5. They differ only in how much they delete.
alias of [`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[‘none’, ‘incremental’, ‘full’, ‘scoped_full’]
### *class* ef.refresh.RefreshPlan(to_materialize=(), to_delete=())
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
What a refresh will do — [`plan_refresh()`](_autosummary/ef.refresh.html.md#ef.refresh.plan_refresh)’s output.
The two fields are disjoint in *intent* but may overlap in membership: a
stale source is in **both** — its old documents are deleted *and* it is
re-materialized. An orphan is only in `to_delete`; a brand-new source only
in `to_materialize`.
#### to_materialize
source ids to (re-)run through the pipeline.
#### to_delete
source ids whose currently-indexed documents are removed
(before re-materializing, for the ones that also reappear above).
### *class* ef.refresh.RefreshReport(config, mode, added=(), modified=(), deleted=(), unchanged=(), documents_written=0, documents_removed=0, artifacts_removed=0)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
The outcome of a refresh — what `SourceManager.refresh()` returns.
`added` / `modified` / `deleted` / `unchanged` partition the
source ids the refresh considered (mirroring [`CorpusDiff`](_autosummary/ef.corpus.html.md#ef.corpus.CorpusDiff)
field names): newly indexed, re-indexed (was stale/misconfigured), removed
(orphan), and already-fresh-left-alone. The three counts report the work
done. Truthiness reports whether anything changed.
```pycon
>>> r = RefreshReport(config='c0', mode='full', added=('a',), modified=('b',))
>>> bool(r), r.changed
(True, ('a', 'b'))
>>> bool(RefreshReport(config='c0', mode='full', unchanged=('a',)))
False
```
#### *property* changed *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), ...]*
The source ids that were added, modified or deleted.
### ef.refresh.delete_source_documents(collection, indexed, source_ids)
Delete every `vd` document belonging to the given `source_ids`.
The correctness half of a refresh: a stale or orphaned source’s documents
must leave the collection. `indexed` — an [`indexed_state()`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.indexed_state)
grouping — supplies each source’s document ids, so no per-source scan is
needed. A document already gone is silently tolerated (idempotent).
* **Parameters:**
* **collection** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the `vd` collection (a `MutableMapping`) to delete from.
* **indexed** ([`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`IndexedSource`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.IndexedSource)]) – the `{source_id: IndexedSource}` map of what is indexed.
* **source_ids** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – the sources whose documents to remove.
* **Return type:**
[`int`](https://docs.python.org/3/builtins/functions.html#int)
* **Returns:**
the number of documents actually deleted.
```pycon
>>> from ef.diagnostics import IndexedSource
>>> collection = {'s1#0': ..., 's1#1': ..., 's2#0': ...}
>>> indexed = {
... 's1': IndexedSource('s1', ('s1#0', 's1#1'), frozenset(), frozenset()),
... 's2': IndexedSource('s2', ('s2#0',), frozenset(), frozenset()),
... }
>>> delete_source_documents(collection, indexed, ['s1'])
2
>>> sorted(collection)
['s2#0']
```
### ef.refresh.plan_refresh(report, , mode='full', scope=None)
Turn a [`StalenessReport`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.StalenessReport) into a [`RefreshPlan`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshPlan).
The pure, side-effect-free core of the refresh policy. Every source that is
`missing`, `stale` or `misconfigured` is (re-)materialized; what gets
*deleted* is what the `mode` dictates:
- `none` — delete nothing (stale content is added alongside the old);
- `incremental` — delete the stale/misconfigured sources’ old documents;
- `full` / `scoped_full` — also delete orphans.
* **Parameters:**
* **report** ([`StalenessReport`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.StalenessReport)) – the staleness report to act on.
* **mode** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'none'`, `'incremental'`, `'full'`, `'scoped_full'`]) – one of [`REFRESH_MODES`](_autosummary/ef.refresh.html.md#ef.refresh.REFRESH_MODES).
* **scope** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – an optional subset of source ids the plan is restricted to —
sources outside it are left entirely alone. `None` means the whole
report. (With `scope=None`, `full` and `scoped_full` coincide.)
* **Raises:**
[**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if `mode` is not a recognized [`RefreshMode`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshMode).
* **Return type:**
[`RefreshPlan`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshPlan)
```pycon
>>> from ef.diagnostics import StalenessReport
>>> report = StalenessReport(config='c0', orphan=('x',), missing=('a',),
... stale=('b',), misconfigured=('c',), fresh=('d',))
>>> plan_refresh(report, mode='full').to_materialize
('a', 'b', 'c')
>>> plan_refresh(report, mode='full').to_delete
('b', 'c', 'x')
>>> plan_refresh(report, mode='incremental').to_delete
('b', 'c')
>>> plan_refresh(report, mode='none').to_delete
()
>>> plan_refresh(report, mode='scoped_full', scope=['a', 'b']).to_delete
('b',)
```
### ef.refresh.prune_dead_leaves(graph, live_leaves)
Cascade out of `graph` every leaf no live source still hashes to.
The housekeeping half of a refresh. A graph *leaf* (a node with a cached
value but no producer recipe) is an L0 source keyed by its content hash.
After a refresh, the set of leaves the graph *should* hold is exactly the
content hashes of the current corpus; any other leaf is dead — the source
that produced it was edited (new content → new leaf) or deleted. Each dead
leaf and its whole downstream cone (its segment + embed artifacts) is
[`delete_cascade()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.delete_cascade)d.
This is purely a cache reclaim: a dead leaf is never *served* (nothing
addresses it any more), so pruning it can never change a result — only free
memory. Dead leaves’ cones are disjoint (each produced artifact traces to
exactly one leaf), so the removed-count is exact.
* **Parameters:**
* **graph** ([`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph)) – the [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph) to prune.
* **live_leaves** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – the content hashes that are still live (the current
corpus’s source hashes).
* **Return type:**
[`int`](https://docs.python.org/3/builtins/functions.html#int)
* **Returns:**
the number of artifacts (leaves + produced) removed.
```pycon
>>> from ef import ArtifactGraph, producer_spec
>>> graph = ArtifactGraph()
>>> _ = graph.register_op('id', lambda x: x)
>>> _ = graph.put('live', 'L')
>>> _ = graph.put('dead', 'D')
>>> artifact = graph.add(producer_spec('id', 'dead', op_version='1'))
>>> _ = graph.materialize(artifact)
>>> prune_dead_leaves(graph, ['live']) # the 'dead' leaf + its artifact
2
>>> 'dead' in graph, 'live' in graph
(False, True)
```
### ef.refresh.refresh_on_change(manager)
Build a [`ChangeEvent`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeEvent) handler that auto-refreshes `manager`.
The *auto* refresh seam. The returned callable is meant to be a
[`ChangeDetectingCorpus`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus)’s `on_change` — each detected
edit (through the wrapper, or surfaced out-of-band by `scan()`) is
incrementally applied to every materialized config of `manager`:
an `added` source is indexed, a `modified` one is re-indexed
(delete-then-add), a `deleted` one is removed.
[`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager) wires this automatically when
constructed with `auto_refresh=True`; call this directly only to attach a
manager to a corpus you build yourself.
* **Parameters:**
**manager** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – a [`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager) (duck-typed — any
object with an `_apply_change(event)` method).
* **Return type:**
[`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`ChangeEvent`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeEvent)], [`None`](https://docs.python.org/3/builtins/constants.html#None)]
* **Returns:**
a `Callable[[ChangeEvent], None]` for `on_change`.
# _autosummary/ef.reranking.html.md
# ef.reranking
Reranking — a precision pass over retrieved segments (use case F6).
First-stage retrieval (a dense ANN query over `vd`) is tuned for *recall*: it
casts a wide, cheap net. A **reranker** is the precision counterpart — a slower,
more accurate model that re-scores a handful of candidates against the query.
The standard two-stage pattern is “retrieve `fetch_k` cheaply, rerank, keep
the top `limit`”.
This module keeps reranking a *decorator over a retriever*, never a new pipeline
stage — it composes with the existing surface rather than complicating it:
- [`Reranker`](_autosummary/ef.reranking.html.md#ef.reranking.Reranker) — the structural protocol: a callable
`(query, segments) -> one score per segment`. Any plain function of that
shape *is* a reranker; no base class to subclass.
- [`rerank()`](_autosummary/ef.reranking.html.md#ef.reranking.rerank) — the pure operation: re-score and re-order a list of
[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)s. The reranker’s score is folded into each
> segment’s `metadata["rerank_score"]`.
- [`with_reranker()`](_autosummary/ef.reranking.html.md#ef.reranking.with_reranker) — the decorator: wrap a retrieve-style callable (such as
[`SourceManager.retrieve`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.retrieve)) so
> every call over-fetches, reranks, and trims — transparently.
- [`cross_encoder_reranker()`](_autosummary/ef.reranking.html.md#ef.reranking.cross_encoder_reranker) — a ready reranker backed by a
sentence-transformers `CrossEncoder` (an optional heavy dependency, imported
lazily — the protocol and [`rerank()`](_autosummary/ef.reranking.html.md#ef.reranking.rerank) cost nothing).
Example — rerank three segments by a toy length score, longest first:
```pycon
>>> segments = [
... {'text': 'cat', 'id': '1'},
... {'text': 'dog', 'id': '2'},
... {'text': 'cathedral', 'id': '3'},
... ]
>>> def by_length(query, segments):
... return [len(segment['text']) for segment in segments]
>>> [segment['text'] for segment in rerank('a query', segments, by_length, limit=2)]
['cathedral', 'cat']
```
### Functions
| [`rerank`](_autosummary/ef.reranking.html.md#ef.reranking.rerank)(query, segments, reranker, \*[, limit]) | Re-score `segments` against `query` with `reranker` and re-order them. |
|----------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
| [`with_reranker`](_autosummary/ef.reranking.html.md#ef.reranking.with_reranker)(retriever, reranker, \*[, fetch_k]) | Decorate a retrieve-style callable with a two-stage reranking pass. |
| [`cross_encoder_reranker`](_autosummary/ef.reranking.html.md#ef.reranking.cross_encoder_reranker)([model_name, device, ...]) | Build a [`Reranker`](_autosummary/ef.reranking.html.md#ef.reranking.Reranker) backed by a sentence-transformers `CrossEncoder`. |
### Classes
| [`Reranker`](_autosummary/ef.reranking.html.md#ef.reranking.Reranker)(\*args, \*\*kwargs) | A callable that re-scores `(query, segments)` — higher means more relevant. |
|---------------------------------------------------------------------------------|-------------------------------------------------------------------------------|
### *class* ef.reranking.Reranker(\*args, \*\*kwargs)
Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol)
A callable that re-scores `(query, segments)` — higher means more relevant.
The structural contract a reranker satisfies: given the query and a list of
candidate [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)s, return \*\*one score per
segment\*\*, in the same order. [`rerank()`](_autosummary/ef.reranking.html.md#ef.reranking.rerank) sorts by these scores. It is a
`@runtime_checkable Protocol`, not a base class — a plain function of the
right shape already *is* a [`Reranker`](_autosummary/ef.reranking.html.md#ef.reranking.Reranker).
```pycon
>>> def by_length(query, segments):
... return [len(s['text']) for s in segments]
>>> isinstance(by_length, Reranker)
True
```
### ef.reranking.cross_encoder_reranker(model_name='cross-encoder/ms-marco-MiniLM-L-6-v2', , device=None, batch_size=32, \*\*predict_kwargs)
Build a [`Reranker`](_autosummary/ef.reranking.html.md#ef.reranking.Reranker) backed by a sentence-transformers `CrossEncoder`.
A cross-encoder scores a `(query, passage)` pair *jointly* — it is the
accurate, slower second stage two-stage retrieval is built for. The model is
a heavy, optional dependency: it is imported only when this factory is
called, so importing [`ef.reranking`](_autosummary/ef.reranking.html.md#module-ef.reranking) itself stays cheap.
Example:
```default
from ef import ingest, with_reranker, cross_encoder_reranker
index = ingest(corpus, embedder="st:all-MiniLM-L6-v2")
retrieve = with_reranker(index.retrieve, cross_encoder_reranker())
segments = retrieve("a query", limit=5)
```
* **Parameters:**
* **model_name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – a sentence-transformers `CrossEncoder` model name.
* **device** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – torch device (`"cuda"` / `"cpu"` / `"mps"`); `None`
auto-selects.
* **batch_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – how many pairs to score per forward pass.
* **predict_kwargs** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – extra keyword arguments forwarded to `CrossEncoder.predict`.
* **Return type:**
[`Reranker`](_autosummary/ef.reranking.html.md#ef.reranking.Reranker)
* **Returns:**
a [`Reranker`](_autosummary/ef.reranking.html.md#ef.reranking.Reranker) — `(query, segments) -> scores`.
* **Raises:**
[**ImportError**](https://docs.python.org/3/builtins/exceptions.html#ImportError) – if the optional `sentence-transformers` package is absent.
### ef.reranking.rerank(query, segments, reranker, , limit=None)
Re-score `segments` against `query` with `reranker` and re-order them.
The pure heart of the module — no I/O, no retrieval. The `reranker` is
called once with the whole candidate list; the segments are returned sorted
by descending score (ties keep their incoming order), optionally trimmed to
`limit`. Each returned segment is a copy carrying the reranker’s score in
`metadata["rerank_score"]` — so the new ranking is auditable.
* **Parameters:**
* **query** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the search query the segments are scored against.
* **segments** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]) – the candidate segments (from a first-stage retriever).
* **reranker** ([`Reranker`](_autosummary/ef.reranking.html.md#ef.reranking.Reranker)) – a [`Reranker`](_autosummary/ef.reranking.html.md#ef.reranking.Reranker) — a callable `(query, segments) -> scores`.
* **limit** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – keep only the top `limit` after reranking; `None` keeps all.
* **Return type:**
[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]
* **Returns:**
the reranked segments — copies, best first.
* **Raises:**
[**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if the reranker returns a number of scores that does not
match the number of segments.
```pycon
>>> segs = [{'text': 'a', 'id': '1'}, {'text': 'bbb', 'id': '2'}]
>>> ranked = rerank('q', segs, lambda q, s: [len(x['text']) for x in s])
>>> [r['text'] for r in ranked]
['bbb', 'a']
>>> ranked[0]['metadata']['rerank_score']
3.0
```
### ef.reranking.with_reranker(retriever, reranker, , fetch_k=50)
Decorate a retrieve-style callable with a two-stage reranking pass.
Wraps `retriever` (e.g. [`SourceManager.retrieve`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.retrieve), or any callable
`query -> list[Segment]`) so every call over-fetches `fetch_k`
candidates cheaply, [`rerank()`](_autosummary/ef.reranking.html.md#ef.reranking.rerank)s them, and returns the top `limit`.
The decorated callable keeps the retriever’s interface — `query` plus a
keyword-only `limit` — so it is a drop-in replacement, including as the
`retriever` argument of [`evaluate_retrieval()`](_autosummary/ef.evaluation.html.md#ef.evaluation.evaluate_retrieval).
* **Parameters:**
* **retriever** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]]) – the first-stage retriever — a callable
`retriever(query, *, limit=...) -> segments`.
* **reranker** ([`Reranker`](_autosummary/ef.reranking.html.md#ef.reranking.Reranker)) – the [`Reranker`](_autosummary/ef.reranking.html.md#ef.reranking.Reranker) applied to the over-fetched candidates.
* **fetch_k** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – how many candidates to retrieve before reranking. Larger trades
latency for recall headroom; it should comfortably exceed the
`limit` callers ask for.
* **Return type:**
[`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]]
* **Returns:**
a new callable `(query, *, limit=10, **kwargs) -> list[Segment]` —
extra keyword arguments (e.g. `filter`) pass straight through to
`retriever`.
```pycon
>>> pool = [{'text': t, 'id': t} for t in ['ab', 'abcd', 'abc']]
>>> def base(query, *, limit=10):
... return pool[:limit]
>>> retrieve = with_reranker(
... base, lambda q, segs: [len(s['text']) for s in segs], fetch_k=10)
>>> [s['text'] for s in retrieve('a query', limit=2)]
['abcd', 'abc']
```
# _autosummary/ef.segmenter_adapters.html.md
# ef.segmenter_adapters
The segmenter dependency-injection seam and the `imbed` registry bridge.
[`as_segmenter()`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.as_segmenter) is the single place `ef` coerces a user-supplied
segmenter argument into a well-behaved [`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter) — the
exact mirror of [`as_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.as_embedder) for the embed layer.
[`imbed_segmenter()`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.imbed_segmenter) builds a segmenter from `imbed`’s segmenter registry
(`imbed.components.segmentation`); `imbed` is imported lazily, so importing
this module costs nothing extra and `ef` works without `imbed` installed.
Example — the DI seam in action (no `imbed` needed):
```pycon
>>> from ef.segmenters import Segmenter, RecursiveCharacterSegmenter
>>> seg = as_segmenter('recursive', chunk_size=256)
>>> isinstance(seg, RecursiveCharacterSegmenter)
True
>>> as_segmenter(seg) is seg # a ready segmenter passes straight through
True
>>> custom = as_segmenter(lambda text: text.split('|'))
>>> [s['text'] for s in custom('a|b|c')]
['a', 'b', 'c']
```
### Functions
| [`as_segmenter`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.as_segmenter)([x]) | Normalize `x` into a [`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter) — the DI seam. |
|--------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|
| [`imbed_segmenter`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.imbed_segmenter)([name]) | Build a [`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter) from `imbed`'s segmenter registry. |
### ef.segmenter_adapters.as_segmenter(x=None, , \*\*kwargs)
Normalize `x` into a [`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter) — the DI seam.
The single place every `ef` entry point coerces a user-supplied segmenter
argument. Accepts, in order:
1. `None` or `"default"` / `"recursive"` — a
[`RecursiveCharacterSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.RecursiveCharacterSegmenter) (the default);
2. `"lines"` — [`line_segmenter()`](_autosummary/ef.segmenters.html.md#ef.segmenters.line_segmenter);
3. any other string — looked up in `imbed`’s registry via
[`imbed_segmenter()`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.imbed_segmenter);
4. a ready-made segmenter (a [`BaseSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.BaseSegmenter) or a
callable marked by `ef`) — returned unchanged;
5. a bare callable — wrapped in [`FunctionSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.FunctionSegmenter).
Extra `**kwargs` are forwarded to the chosen factory (ignored when `x`
is already a segmenter).
* **Raises:**
[**TypeError**](https://docs.python.org/3/builtins/exceptions.html#TypeError) – if `x` is none of the above.
* **Return type:**
[*Segmenter*](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter)
```pycon
>>> seg = as_segmenter(chunk_size=256, chunk_overlap=32)
>>> (seg.chunk_size, seg.chunk_overlap)
(256, 32)
```
* **Return type:**
[`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter)
### ef.segmenter_adapters.imbed_segmenter(name='default', , \*\*kwargs)
Build a [`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter) from `imbed`’s segmenter registry.
`imbed`’s registered segmenters are bare `text -> Iterable[str]`
callables; this wraps the named one in a [`FunctionSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.FunctionSegmenter)
so it emits well-formed [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) pieces.
* **Parameters:**
* **name** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – A key in `imbed.components.segmentation.segmenters` (`"default"`
resolves to `imbed`’s configured default segmenter).
* **\*\*kwargs** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – Forwarded to [`FunctionSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.FunctionSegmenter).
* **Raises:**
* [**ImportError**](https://docs.python.org/3/builtins/exceptions.html#ImportError) – if `imbed` is not installed.
* [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if `name` is not a registered segmenter.
* **Return type:**
[`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter)
Requires the `imbed` package (`pip install 'ef[imbed]'`).
# _autosummary/ef.segmenters.html.md
# ef.segmenters
The `Segmenter` facade — `ef`’s segment layer (L2).
A *segmenter* turns a document into a stream of [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)
pieces. Like the embedder facade, its core is a tiny structural contract:
```default
Segmenter = Callable[[str | Mapping], Iterable[Segment]]
```
Everything else — overlap, hierarchy, batching, the recursive splitting
algorithm — is *layered scaffolding* over that one shape.
This module defines:
- [`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter) — the `@runtime_checkable` protocol (a callable). It is
a protocol, **not** an ABC: any matching callable *is* a segmenter.
- [`BatchedSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.BatchedSegmenter) — an optional optimization trait (a `batch`
method) for segmenters that can process many documents at once.
- [`BaseSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.BaseSegmenter) — an implementation-convenience base (a default
[`batch()`](_autosummary/ef.segmenters.html.md#ef.segmenters.BaseSegmenter.batch) and a `repr`). Subclassing is optional.
- [`RecursiveCharacterSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.RecursiveCharacterSegmenter) — the **default segmenter**: recursive
character splitting at ~512 tokens with ~12.5% overlap. The honest default —
within a point or two of fancier methods on realistic corpora.
- [`line_segmenter()`](_autosummary/ef.segmenters.html.md#ef.segmenters.line_segmenter) — a builtin that splits a document into its non-empty
lines (a plain function — proof that a bare callable *is* a segmenter).
- [`FunctionSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.FunctionSegmenter) — wraps a bare `text -> pieces` callable into a
full segmenter; the bridge for `imbed`’s registered segmenters.
- Composition helpers — [`with_overlap()`](_autosummary/ef.segmenters.html.md#ef.segmenters.with_overlap), [`hierarchical()`](_autosummary/ef.segmenters.html.md#ef.segmenters.hierarchical),
[`materialise()`](_autosummary/ef.segmenters.html.md#ef.segmenters.materialise).
Streaming is first-class: a segmenter returns an *iterable*, not a list, so the
heavy case (a huge corpus) never has to fit every segment in memory at once.
The dependency-injection seam [`as_segmenter()`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.as_segmenter) and
the `imbed` registry bridge live in [`ef.segmenter_adapters`](_autosummary/ef.segmenter_adapters.html.md#module-ef.segmenter_adapters).
Example — split a document into ~4-token chunks with one token of overlap:
```pycon
>>> seg = RecursiveCharacterSegmenter(chunk_size=4, chunk_overlap=1)
>>> [c['text'] for c in seg('alpha beta gamma delta epsilon')]
['alpha beta gamma delta', 'delta epsilon']
>>> isinstance(seg, Segmenter)
True
```
### Module Attributes
| [`APPROX_TOKENIZER`](_autosummary/ef.segmenters.html.md#ef.segmenters.APPROX_TOKENIZER) | Identifier recorded as the `tokenizer` metadata key when token counts come from [`approx_token_count()`](_autosummary/ef.segmenters.html.md#ef.segmenters.approx_token_count). |
|---------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|
| [`DEFAULT_SEPARATORS`](_autosummary/ef.segmenters.html.md#ef.segmenters.DEFAULT_SEPARATORS) | Separators tried, in order, by [`RecursiveCharacterSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.RecursiveCharacterSegmenter). |
### Functions
| [`line_segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.line_segmenter)(doc, /) | Segment a document into its non-empty lines. |
|-------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------|
| [`with_overlap`](_autosummary/ef.segmenters.html.md#ef.segmenters.with_overlap)(segmenter, n_chars, /, \*[, ...]) | Wrap `segmenter` so consecutive segments overlap by `n_chars` characters. |
| [`hierarchical`](_autosummary/ef.segmenters.html.md#ef.segmenters.hierarchical)(segmenters, /) | Compose segmenters into levels — each segments the previous level's output. |
| [`materialise`](_autosummary/ef.segmenters.html.md#ef.segmenters.materialise)(x) | Force a streaming segmenter (or segment stream) to a concrete list. |
| [`approx_token_count`](_autosummary/ef.segmenters.html.md#ef.segmenters.approx_token_count)(text) | Rough token count — the common `~4 characters per token` heuristic. |
### Classes
| [`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter)(\*args, \*\*kwargs) | Structural type for a segmenter: `str | Mapping -> Iterable[Segment]`. |
|-----------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------|
| [`BatchedSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.BatchedSegmenter)(\*args, \*\*kwargs) | Optional trait: a segmenter that can process many documents at once. |
| [`BaseSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.BaseSegmenter)() | Implementation base for segmenters (optional — [`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter) is structural). |
| [`FunctionSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.FunctionSegmenter)(func, \*[, pass_doc, ...]) | Promote a bare `text -> pieces` callable to a full [`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter). |
| [`RecursiveCharacterSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.RecursiveCharacterSegmenter)(\*[, chunk_size, ...]) | The default segmenter — recursive character splitting. |
### ef.segmenters.APPROX_TOKENIZER *= 'char-approx'*
Identifier recorded as the `tokenizer` metadata key when token counts come
from [`approx_token_count()`](_autosummary/ef.segmenters.html.md#ef.segmenters.approx_token_count). Recording *which* tokenizer counted a
segment is mandatory — a `tokens` number is meaningless without it.
### *class* ef.segmenters.BaseSegmenter
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
Implementation base for segmenters (optional — [`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter) is structural).
A subclass implements `__call__`; in return it gets a default
[`batch()`](_autosummary/ef.segmenters.html.md#ef.segmenters.BaseSegmenter.batch) (one-document-at-a-time — override it for a genuinely batched
backend) and a readable `repr`. Subclassing is purely a convenience;
satisfying [`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter) structurally is what matters.
#### batch(docs,)
Default: segment each document independently.
* **Return type:**
[`Iterator`](https://docs.python.org/3/library/typing.html#typing.Iterator)[[`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]]
### *class* ef.segmenters.BatchedSegmenter(\*args, \*\*kwargs)
Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol)
Optional trait: a segmenter that can process many documents at once.
Plain [`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter) handles one document per call. A segmenter that can
amortize per-call cost across documents (a shared model load, a vectorized
pass) also offers `batch`. [`BaseSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.BaseSegmenter) supplies a correct — if
un-optimized — default, so every [`BaseSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.BaseSegmenter) subclass is already
a [`BatchedSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.BatchedSegmenter).
#### batch(docs,)
Segment each document in `docs`, yielding one segment-stream each.
* **Return type:**
[`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]]
### ef.segmenters.DEFAULT_SEPARATORS *: [tuple](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), ...]* *= ('\\n\\n', '\\n', '. ', ', ', ' ', '')*
Separators tried, in order, by [`RecursiveCharacterSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.RecursiveCharacterSegmenter). Earlier
(coarser) boundaries are preferred; the empty string is the last resort,
splitting between characters so an oversized run is never left unsplit.
### *class* ef.segmenters.FunctionSegmenter(func, \*, pass_doc=False, count_tokens=, tokenizer='char-approx')
Bases: [`BaseSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.BaseSegmenter)
Promote a bare `text -> pieces` callable to a full [`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter).
The bridge for segmenter *functions* that carry no segment schema — most
importantly `imbed`’s registered segmenters, which map text to an iterable
of plain strings. Each piece the callable yields is normalized:
- a `str` piece becomes a [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment); its character
offsets are recovered by locating it in the source text, and a token
count plus `tokenizer` metadata are attached;
- a `Mapping` / [`SegmentRecord`](_autosummary/ef.segments.html.md#ef.segments.SegmentRecord) piece is coerced with
[`as_segment()`](_autosummary/ef.segments.html.md#ef.segments.as_segment), gaining only an `index` / `parent_id`
> where it lacks them.
* **Parameters:**
* **func** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[`...`](https://docs.python.org/3/builtins/constants.html#Ellipsis), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]) – The callable to wrap. By default it receives the document’s
*text*; with `pass_doc=True` it receives the whole document.
* **pass_doc** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Pass the raw document instead of its text.
* **count_tokens** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`int`](https://docs.python.org/3/builtins/functions.html#int)]]) – Token counter for `str` pieces (`None` disables the
`tokens` / `tokenizer` annotation).
* **tokenizer** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Name recorded as the `tokenizer` metadata key.
```pycon
>>> fs = FunctionSegmenter(lambda text: text.split(','))
>>> [s['text'] for s in fs('a,b,c')]
['a', 'b', 'c']
>>> s = list(fs('a,b,c'))[1]
>>> (s['start'], s['end'], s['index'])
(2, 3, 1)
```
### *class* ef.segmenters.RecursiveCharacterSegmenter(\*, chunk_size=512, chunk_overlap=64, separators=('\\\\n\\\\n', '\\\\n', '. ', ', ', ' ', ''), count_tokens=, tokenizer='char-approx')
Bases: [`BaseSegmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.BaseSegmenter)
The default segmenter — recursive character splitting.
Splits a document on a descending ladder of separators (paragraph → line →
sentence → clause → word → character), merging the resulting pieces into
chunks of about `chunk_size` tokens with `chunk_overlap` tokens shared
between neighbours. This “good enough” default is, on realistic corpora,
within a point or two of far more elaborate methods — upgrade only if you
can measure the win.
Sizes are measured with `count_tokens`. The default counter is the
dependency-free [`approx_token_count()`](_autosummary/ef.segmenters.html.md#ef.segmenters.approx_token_count); inject a real tokenizer (and its
`tokenizer` name) when an exact count matters. Every emitted segment
records the tokenizer in its metadata and carries exact `start` / `end`
character offsets into the source.
* **Parameters:**
* **chunk_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Target chunk size, in `count_tokens` units.
* **chunk_overlap** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Tokens of overlap between consecutive chunks (must be
smaller than `chunk_size`).
* **separators** ([`Sequence`](https://docs.python.org/3/library/typing.html#typing.Sequence)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – The separator ladder, coarsest first; the last should be
`""` so an unsplittable run is still broken between characters.
* **count_tokens** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`int`](https://docs.python.org/3/builtins/functions.html#int)]) – Maps a string to a token count.
* **tokenizer** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Name recorded as the `tokenizer` metadata key.
```pycon
>>> seg = RecursiveCharacterSegmenter(chunk_size=3, chunk_overlap=1)
>>> [c['text'] for c in seg('one two three four')]
['one two three', 'three four']
```
### *class* ef.segmenters.Segmenter(\*args, \*\*kwargs)
Bases: [`Protocol`](https://docs.python.org/3/library/typing.html#typing.Protocol)
Structural type for a segmenter: `str | Mapping -> Iterable[Segment]`.
A segmenter is *just a callable*. It accepts a document — a bare `str` or
a `Mapping` with a `text` key (and optionally an `id`, which becomes
the emitted segments’ `parent_id`) — and yields [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)
pieces. It returns an **iterable**, not a list: streaming is the default so
the heavy case scales.
Because the protocol’s only member is `__call__`, *every* callable
satisfies `isinstance(x, Segmenter)`. Treat it as documentation of intent,
not a discriminating check — [`as_segmenter()`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.as_segmenter) is
the seam that turns an arbitrary callable into a *well-behaved* segmenter.
### ef.segmenters.approx_token_count(text)
Rough token count — the common `~4 characters per token` heuristic.
A dependency-free default so `ef` can talk in “tokens” without pulling in
`tiktoken` or a model tokenizer. Inject a real counter (and its name) into
a segmenter when an exact count matters.
* **Return type:**
[`int`](https://docs.python.org/3/builtins/functions.html#int)
```pycon
>>> approx_token_count('a' * 40)
10
>>> approx_token_count('')
0
```
### ef.segmenters.hierarchical(segmenters,)
Compose segmenters into levels — each segments the previous level’s output.
Level 0 applies `segmenters[0]` to the document. Level *k* applies
`segmenters[k]` to every level- *(k-1)* segment’s text, setting each
child’s `parent_id` to that segment’s `id` and shifting its offsets to
stay absolute. The result is a flat stream of every segment of every level
(level 0 first), with the tree expressed purely through `parent_id`
pointers.
Each entry of `segmenters` is passed through
[`as_segmenter()`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.as_segmenter), so strings (e.g. `"recursive"`)
and bare callables are accepted alongside ready-made segmenters.
* **Return type:**
[`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter)
### ef.segmenters.line_segmenter(doc,)
Segment a document into its non-empty lines.
Each line is stripped of surrounding whitespace; blank lines are dropped.
Character offsets point at the *stripped* content within the source.
A plain function — and therefore already a [`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter). It carries
the marker [`as_segmenter()`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.as_segmenter) uses to recognize a
ready-made segmenter.
* **Return type:**
[`Iterator`](https://docs.python.org/3/library/typing.html#typing.Iterator)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]
```pycon
>>> [s['text'] for s in line_segmenter(' hello \n\nworld\n')]
['hello', 'world']
```
### ef.segmenters.materialise(x)
Force a streaming segmenter (or segment stream) to a concrete list.
Streaming is the default, but the light case often wants everything in
hand. [`materialise()`](_autosummary/ef.segmenters.html.md#ef.segmenters.materialise) is polymorphic:
- given a **segmenter** (a callable), it returns a new segmenter whose
output is a `list` rather than a lazy iterator;
- given an **iterable of segments**, it returns a `list` of them.
```pycon
>>> eager = materialise(line_segmenter)
>>> out = eager('a\nb')
>>> isinstance(out, list)
True
>>> [s['text'] for s in out]
['a', 'b']
>>> materialise(s for s in [{'text': 'x'}])
[{'text': 'x'}]
```
* **Return type:**
[`Any`](https://docs.python.org/3/library/typing.html#typing.Any)
### ef.segmenters.with_overlap(segmenter, n_chars, /, \*, count_tokens=, tokenizer='char-approx')
Wrap `segmenter` so consecutive segments overlap by `n_chars` characters.
Each segment is extended *forward* by up to `n_chars` characters (clamped
to the source length), so segment *K* shares its tail with segment *K+1*’s
head. A segment whose text changes has its `id` re-derived and its
`tokens` recounted; segments lacking `start` / `end` offsets pass
through untouched.
Overlap is expressed in *characters* here (this is a generic post-hoc
helper) — for token-measured overlap use a segmenter’s own `chunk_overlap`.
* **Return type:**
[`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter)
```pycon
>>> base = RecursiveCharacterSegmenter(chunk_size=3, chunk_overlap=0)
>>> [c['text'] for c in base('one two three four')]
['one two three', 'four']
>>> [c['text'] for c in with_overlap(base, 100)('one two three four')]
['one two three four', 'four']
```
# _autosummary/ef.segments.html.md
# ef.segments
The canonical `Segment` data model — `ef`’s interchange type for L2.
A *segment* is a piece of text carved out of a source document by a
[`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter). This module defines the one shape every
segmenter, adapter and store in `ef` agrees on, so segments round-trip
losslessly across the whole pipeline.
Two surfaces over the same data:
- [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) — a `TypedDict`: the lightweight, streaming-friendly
*interchange* type. A segment is just a `dict`; the type is passive and
every helper here is a free function over it.
- [`SegmentRecord`](_autosummary/ef.segments.html.md#ef.segments.SegmentRecord) — a frozen dataclass: the *convenience* surface for
code that prefers attribute access and immutability.
Only `text` is genuinely required; `id` is required by convention and is
**content-derived** by default ([`segment_id()`](_autosummary/ef.segments.html.md#ef.segments.segment_id)) so that re-segmenting the
same document yields the same ids — the basis of idempotent ingestion.
The two field names that are *promoted to top level* (rather than buried in
`metadata`) are `parent_id` and `index` — hierarchy is expressed by a
flat list with `parent_id` pointers, never a nested structure.
Example — build a segment, derive its id, view it as a record:
```pycon
>>> seg = make_segment('Hello world.', index=0, tokens=3)
>>> seg['text']
'Hello world.'
>>> seg['id'] == segment_id('Hello world.')
True
>>> rec = segment_record(seg)
>>> rec.text, rec.index
('Hello world.', 0)
>>> rec.to_segment() == seg
True
```
### Module Attributes
| [`PROMOTED_METADATA_KEYS`](_autosummary/ef.segments.html.md#ef.segments.PROMOTED_METADATA_KEYS) | Metadata keys that, by convention, every `ef` adapter must round-trip faithfully. |
|---------------------------------------------------------------------------|-------------------------------------------------------------------------------------|
### Functions
| [`segment_id`](_autosummary/ef.segments.html.md#ef.segments.segment_id)(text[, metadata]) | Content-derived id for a segment — `sha256` of its normalized content. |
|-----------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|
| [`make_segment`](_autosummary/ef.segments.html.md#ef.segments.make_segment)(text, \*[, id, parent_id, ...]) | Build a [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment), deriving a content-based `id` if none given. |
| [`as_segment`](_autosummary/ef.segments.html.md#ef.segments.as_segment)(x, \*\*defaults) | Coerce `x` into a [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) — accepts a `str`, `Mapping` or record. |
| [`segment_record`](_autosummary/ef.segments.html.md#ef.segments.segment_record)(x) | Coerce `x` (`str` / `Mapping` / [`SegmentRecord`](_autosummary/ef.segments.html.md#ef.segments.SegmentRecord)) into a record. |
### Classes
| [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) | A piece of text carved from a source document — the interchange type. |
|---------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------|
| [`SegmentRecord`](_autosummary/ef.segments.html.md#ef.segments.SegmentRecord)(text[, id, parent_id, start, ...]) | Frozen-dataclass view over a [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) — the convenience surface. |
### ef.segments.PROMOTED_METADATA_KEYS *= ('source', 'source_type', 'tokenizer', 'token_count', 'embedding_model', 'page', 'license', 'ingestion_run_id')*
Metadata keys that, by convention, every `ef` adapter must round-trip
faithfully. They live inside `Segment["metadata"]` (they are *not* promoted
to top-level fields) but carry meaning the pipeline relies on. Recording
`tokenizer` in particular is mandatory — omitting it is the single most
common silent bug in chunking pipelines (a segment’s `tokens` count is
meaningless without knowing which tokenizer produced it).
### *class* ef.segments.Segment
Bases: [`TypedDict`](https://docs.python.org/3/library/typing.html#typing.TypedDict)
A piece of text carved from a source document — the interchange type.
A `TypedDict`: a [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) *is* a plain `dict`, which keeps it
cheap to create and stream. `total=False` because most keys are optional;
by convention `text` is always present and `id` is always set (derived
from `text` if not supplied).
Keys:
: text: The segment’s text. **Required.**
id: Stable identifier — content-derived by default ([`segment_id()`](_autosummary/ef.segments.html.md#ef.segments.segment_id)).
parent_id: Id of the source document or parent segment this came from.
start: Character offset of the segment’s start in the source text.
end: Character offset of the segment’s end (exclusive) in the source.
index: Ordinal position of the segment in its segmenter’s output.
tokens: Token count of `text` (meaningful only with `metadata` key
> `tokenizer` recording which tokenizer counted it).
metadata: Free-form mapping; framework-specific keys live here. See
: [`PROMOTED_METADATA_KEYS`](_autosummary/ef.segments.html.md#ef.segments.PROMOTED_METADATA_KEYS) for the conventional keys.
### *class* ef.segments.SegmentRecord(text, id='', parent_id=None, start=None, end=None, index=None, tokens=None, metadata=)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
Frozen-dataclass view over a [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) — the convenience surface.
Use this when attribute access and immutability read better than dict
access; use the [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) `TypedDict` directly on the hot path.
The two convert losslessly via [`to_segment()`](_autosummary/ef.segments.html.md#ef.segments.SegmentRecord.to_segment) and [`segment_record()`](_autosummary/ef.segments.html.md#ef.segments.segment_record).
`id` is derived from `text` (and `metadata`) if left empty.
```pycon
>>> rec = SegmentRecord('hello', index=2)
>>> rec.id == segment_id('hello')
True
>>> rec.to_segment()['index']
2
```
#### to_segment()
Return an equivalent [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) `dict` (dropping empty keys).
* **Return type:**
[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)
### ef.segments.as_segment(x, \*\*defaults)
Coerce `x` into a [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) — accepts a `str`, `Mapping` or record.
This is the segment-side normalizer that lets a `Segmenter` accept
loosely-typed output (a bare string, a partial `dict`) and still emit a
well-formed [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment). `**defaults` supplies *fallback* values:
a default fills a field only when `x` does not already carry it.
Accepts:
1. a [`SegmentRecord`](_autosummary/ef.segments.html.md#ef.segments.SegmentRecord) — converted via [`SegmentRecord.to_segment()`](_autosummary/ef.segments.html.md#ef.segments.SegmentRecord.to_segment);
2. a `str` — becomes the `text` of a fresh segment;
3. a `Mapping` with a `text` key — its keys are carried through.
The `id` is always present on the result (derived if absent).
* **Raises:**
* [**TypeError**](https://docs.python.org/3/builtins/exceptions.html#TypeError) – if `x` is none of the accepted forms.
* [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if `x` is a `Mapping` without a `text` key.
* **Return type:**
[*Segment*](_autosummary/ef.segments.html.md#ef.segments.Segment)
```pycon
>>> as_segment('plain text', index=3)['index']
3
>>> as_segment({'text': 'kept', 'index': 0}, index=9)['index'] # x wins
0
```
* **Return type:**
[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)
### ef.segments.make_segment(text, , id=None, parent_id=None, start=None, end=None, index=None, tokens=None, metadata=None)
Build a [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment), deriving a content-based `id` if none given.
Optional fields are omitted from the result when `None` — a segment
`dict` carries only the keys it actually has.
* **Return type:**
[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)
```pycon
>>> seg = make_segment('chunk one', index=0, start=0, end=9)
>>> sorted(seg)
['end', 'id', 'index', 'start', 'text']
>>> seg['id'] == segment_id('chunk one')
True
```
### ef.segments.segment_id(text, metadata=None)
Content-derived id for a segment — `sha256` of its normalized content.
The id is `sha256(normalize(text) + canonical_json(metadata))` — the
normalization and hashing primitives live in [`ef.hashing`](_autosummary/ef.hashing.html.md#module-ef.hashing), the single
source of truth `ef` content-addresses through. Because the id depends
only on content, re-segmenting an unchanged document produces identical ids
— which is what makes ingestion idempotent (a re-run is a no-op, not a
duplicate). Pass `metadata` only when two segments could share identical
`text` yet must stay distinct (e.g. same boilerplate from different
sources).
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
```pycon
>>> segment_id('hello') == segment_id('hello')
True
>>> segment_id('hello') == segment_id('world')
False
>>> segment_id('x', {'source': 'a'}) == segment_id('x', {'source': 'b'})
False
```
### ef.segments.segment_record(x)
Coerce `x` (`str` / `Mapping` / [`SegmentRecord`](_autosummary/ef.segments.html.md#ef.segments.SegmentRecord)) into a record.
The dataclass counterpart of [`as_segment()`](_autosummary/ef.segments.html.md#ef.segments.as_segment) — the convenience-surface
constructor.
* **Return type:**
[`SegmentRecord`](_autosummary/ef.segments.html.md#ef.segments.SegmentRecord)
```pycon
>>> segment_record('hello').text
'hello'
>>> segment_record({'text': 'hi', 'tokens': 1}).tokens
1
```
# _autosummary/ef.service.html.md
# ef.service
The HTTP-service bridge — [`EfService`](_autosummary/ef.service.html.md#ef.service.EfService), a stateless-friendly facade.
`ef`’s query surface is **stateful**: [`ingest()`](_autosummary/ef.source_manager.html.md#ef.source_manager.ingest)
returns a live [`SearchableCorpus`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus), and a
[`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager) holds a live corpus, an
[`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph) and `vd` collections. HTTP is
**stateless**. To serve `ef` over HTTP — so a frontend (`app_ef`) can reach
it through `qh.mk_app()` — something must map a JSON-friendly `corpus_id`
back to the live indexed object across requests.
[`EfService`](_autosummary/ef.service.html.md#ef.service.EfService) is that bridge: a facade holding a **handle registry**
`{corpus_id: SourceManager}` **on the instance**. A single `EfService()` is
constructed once at server start-up, and its seven bound methods —
[`create_corpus()`](_autosummary/ef.service.html.md#ef.service.EfService.create_corpus), [`search()`](_autosummary/ef.service.html.md#ef.service.EfService.search),
[`retrieve()`](_autosummary/ef.service.html.md#ef.service.EfService.retrieve), [`explore_corpus()`](_autosummary/ef.service.html.md#ef.service.EfService.explore_corpus),
[`corpus_info()`](_autosummary/ef.service.html.md#ef.service.EfService.corpus_info), [`list_corpora()`](_autosummary/ef.service.html.md#ef.service.EfService.list_corpora),
[`delete_corpus()`](_autosummary/ef.service.html.md#ef.service.EfService.delete_corpus) — are handed to `qh.mk_app()` as the HTTP
surface. The registry lives on the instance,
never as a module global: a process-wide mutable registry would be the
`ServiceContext` singleton anti-pattern `ef` rejects (`.claude/CLAUDE.md`
§6) — inject an [`EfService`](_autosummary/ef.service.html.md#ef.service.EfService) explicitly instead.
Every method is JSON-friendly and fully type-annotated, because `qh` derives
the HTTP schema from the type hints. `ef`’s stateful objects never cross the
boundary: a corpus is addressed by its string `corpus_id`, an embedder /
segmenter by a string the DI seam resolves (`"hashing"`, `"openai:..."`,
`"cohere:..."`, …), and results are plain [`SearchHit`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchHit)
/ [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) data.
This is *only* a transport bridge — it adds no orchestration. `ef` stays a
facade (`.claude/CLAUDE.md` §6); answer synthesis, agents and UI live outside.
Example — create a corpus, search it, then drop it, all offline:
```pycon
>>> from ef.service import EfService
>>> service = EfService()
>>> info = service.create_corpus(
... ['the cat sat on the mat', 'dogs are loyal', 'felines and canines'],
... corpus_id='animals',
... )
>>> info['corpus_id'], info['n_sources'], info['n_segments']
('animals', 3, 3)
>>> info['embedder'], info['dim']
('hashing:v1@512', 512)
>>> hits = service.search('animals', 'cat', limit=2)
>>> len(hits)
2
>>> isinstance(hits[0].score, float)
True
>>> segments = service.retrieve('animals', 'cat', limit=1)
>>> isinstance(segments[0]['text'], str)
True
>>> [c['corpus_id'] for c in service.list_corpora()]
['animals']
>>> service.delete_corpus('animals')
>>> service.list_corpora()
[]
```
### Classes
| [`CorpusInfo`](_autosummary/ef.service.html.md#ef.service.CorpusInfo) | A JSON-friendly summary of one registered corpus. |
|------------------------------------------------------------------------------------|---------------------------------------------------------------------------|
| [`EfService`](_autosummary/ef.service.html.md#ef.service.EfService)(\*[, default_embedder]) | A registry-backed facade bridging stateful `ef` to a stateless transport. |
### *class* ef.service.CorpusInfo
Bases: [`TypedDict`](https://docs.python.org/3/library/typing.html#typing.TypedDict)
A JSON-friendly summary of one registered corpus.
The return shape shared by [`EfService.create_corpus()`](_autosummary/ef.service.html.md#ef.service.EfService.create_corpus),
[`EfService.corpus_info()`](_autosummary/ef.service.html.md#ef.service.EfService.corpus_info) and (as a list) [`EfService.list_corpora()`](_autosummary/ef.service.html.md#ef.service.EfService.list_corpora)
— a plain `dict` an HTTP client and `qh`’s schema both understand.
Keys:
: corpus_id: the registry handle — how every other method addresses it.
n_sources: the number of source documents in the corpus.
n_segments: the number of indexed segments (sources cut by the segmenter).
embedder: the embedder’s `model_id` (e.g. `"hashing:v1@512"`).
dim: the embedding dimensionality.
config_id: the content hash of the segmenter+embedder pipeline.
### *class* ef.service.EfService(, default_embedder=None)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
A registry-backed facade bridging stateful `ef` to a stateless transport.
Construct one `EfService` per process, register corpora with
[`create_corpus()`](_autosummary/ef.service.html.md#ef.service.EfService.create_corpus), then query them by `corpus_id`. Hand the seven
bound methods to `qh.mk_app()` to expose them over HTTP — see the module
docstring.
The handle registry is private instance state — never a module global.
Corpora are isolated: each [`create_corpus()`](_autosummary/ef.service.html.md#ef.service.EfService.create_corpus) builds its own
[`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager) over its own in-memory `vd`
backend, so one corpus’s vectors never leak into another’s search.
`default_embedder` sets the embedder [`create_corpus()`](_autosummary/ef.service.html.md#ef.service.EfService.create_corpus) resolves when
its own `embedder` argument is `None` — the per-instance hook a host
(e.g. `app_ef`) uses to pick a policy default without passing
`embedder=` on every call. It defaults to
[`DEFAULT_EMBEDDER`](_autosummary/ef.source_manager.html.md#ef.source_manager.DEFAULT_EMBEDDER) (`"hashing"`), so a bare
`EfService()` stays dependency-free and offline.
```pycon
>>> service = EfService()
>>> len(service)
0
>>> _ = service.create_corpus(['hello world'], corpus_id='greetings')
>>> 'greetings' in service
True
>>> len(service)
1
```
A host can swap the default embedder at construction (the string is resolved
lazily, per corpus, by the DI seam — constructing the service touches no
network):
```pycon
>>> EfService(default_embedder='openai:text-embedding-3-small')
```
#### corpus_info(corpus_id)
Return the [`CorpusInfo`](_autosummary/ef.service.html.md#ef.service.CorpusInfo) of a registered corpus.
* **Raises:**
[**KeyError**](https://docs.python.org/3/builtins/exceptions.html#KeyError) – if `corpus_id` is not registered.
* **Return type:**
[`CorpusInfo`](_autosummary/ef.service.html.md#ef.service.CorpusInfo)
#### create_corpus(sources, , embedder=None, segmenter=None, corpus_id=None, embedder_api_key=None)
Index `sources` into a new corpus, register it, return its [`CorpusInfo`](_autosummary/ef.service.html.md#ef.service.CorpusInfo).
* **Parameters:**
* **sources** ([`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – the corpus — a list of text documents.
* **embedder** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the embedder, as a string the DI seam resolves
([`as_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.as_embedder)) — `"hashing"`,
`"openai:text-embedding-3-small"`, `"cohere:..."`, an
`http(s)://` URL, …. `None` → the service’s
`default_embedder` (chosen at construction; itself
[`DEFAULT_EMBEDDER`](_autosummary/ef.source_manager.html.md#ef.source_manager.DEFAULT_EMBEDDER), the dependency-free
[`HashingEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.HashingEmbedder), unless a host overrode it).
* **segmenter** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the segmenter, as a string
([`as_segmenter()`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.as_segmenter)) — `None` → the
recursive-character default.
* **corpus_id** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the handle to register the corpus under; `None` → a
fresh random id. Reusing a live id is an error.
* **embedder_api_key** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – an optional API key for the embedder — the
bring-your-own-key seam. When given and `embedder` is a
hosted-API spec (`"openai:…"`, `"cohere:…"`, …), the key is
forwarded to that adapter, so the service itself never needs to
hold one. `None` → the adapter’s usual environment-variable
resolution. Ignored for key-less embedders (e.g. `"hashing"`).
The corpus’s embedder keeps the key for the life of the corpus,
so [`search()`](_autosummary/ef.service.html.md#ef.service.EfService.search) / [`retrieve()`](_autosummary/ef.service.html.md#ef.service.EfService.retrieve) / [`explore_corpus()`](_autosummary/ef.service.html.md#ef.service.EfService.explore_corpus)
need no key of their own.
* **Return type:**
[`CorpusInfo`](_autosummary/ef.service.html.md#ef.service.CorpusInfo)
* **Returns:**
the [`CorpusInfo`](_autosummary/ef.service.html.md#ef.service.CorpusInfo) of the freshly indexed corpus.
* **Raises:**
[**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if `corpus_id` is already registered.
#### delete_corpus(corpus_id)
Drop a corpus from the registry, releasing its index.
* **Raises:**
[**KeyError**](https://docs.python.org/3/builtins/exceptions.html#KeyError) – if `corpus_id` is not registered.
* **Return type:**
[`None`](https://docs.python.org/3/builtins/constants.html#None)
#### explore_corpus(corpus_id, , dims=2, projection_method='auto', cluster_method='kmeans', n_clusters=8, label=False)
Project & cluster a registered corpus — the corpus-map surface.
Runs [`ef.exploration.explore()`](_autosummary/ef.exploration.html.md#ef.exploration.explore) over the corpus: every indexed
segment is projected to `dims` coordinates and assigned a cluster,
returned as a row-aligned [`ExploreResult`](_autosummary/ef.exploration.html.md#ef.exploration.ExploreResult) (`ids` /
`coords` / `labels` / `cluster_titles`) — the JSON-friendly shape
an `app_ef` corpus map consumes.
* **Parameters:**
* **corpus_id** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – the corpus to explore.
* **dims** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – projection target dimensionality — `2` or `3`.
* **projection_method** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – `"auto"` / `"umap"` / `"pca"`.
* **cluster_method** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – `"kmeans"` / `"hdbscan"`.
* **n_clusters** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – number of k-means clusters.
* **label** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – when `True`, also name each cluster with an LLM (needs the
`ef[imbed]` extra and a key); default `False`.
* **Raises:**
* [**KeyError**](https://docs.python.org/3/builtins/exceptions.html#KeyError) – if `corpus_id` is not registered.
* [**ValueError**](https://docs.python.org/3/builtins/exceptions.html#ValueError) – if the corpus has fewer than 2 indexed segments.
* **Return type:**
[`ExploreResult`](_autosummary/ef.exploration.html.md#ef.exploration.ExploreResult)
#### list_corpora()
Return the [`CorpusInfo`](_autosummary/ef.service.html.md#ef.service.CorpusInfo) of every registered corpus.
* **Return type:**
[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`CorpusInfo`](_autosummary/ef.service.html.md#ef.service.CorpusInfo)]
#### retrieve(corpus_id, query, , limit=10)
Retrieve ranked [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)s — the RAG-plug-in shape.
Like [`search()`](_autosummary/ef.service.html.md#ef.service.EfService.search), but returns plain segments in rank order (the
`score` dropped, the `source_id` folded into `metadata["source"]`)
— clean context to hand to an external RAG/agent framework. `ef`
returns context; it does not synthesize answers.
* **Raises:**
[**KeyError**](https://docs.python.org/3/builtins/exceptions.html#KeyError) – if `corpus_id` is not registered.
* **Return type:**
[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]
#### search(corpus_id, query, , limit=10)
Search a registered corpus — up to `limit` ranked [`SearchHit`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchHit)s.
Each hit carries the matched [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment), its
similarity `score` (higher = closer) and the `source_id` it was cut
from.
* **Raises:**
[**KeyError**](https://docs.python.org/3/builtins/exceptions.html#KeyError) – if `corpus_id` is not registered.
* **Return type:**
[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`SearchHit`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchHit)]
# _autosummary/ef.source_manager.html.md
# ef.source_manager
The search facade — one-shot [`ingest()`](_autosummary/ef.source_manager.html.md#ef.source_manager.ingest), [`SearchableCorpus`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus), [`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager).
This is where `ef`’s layers come together. Phases 1–4 built the parts — a
[`Corpus`](_autosummary/ef.corpus.html.md#ef.corpus.Corpus), a [`Segmenter`](_autosummary/ef.segmenters.html.md#ef.segmenters.Segmenter), an
[`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder), the content-addressed
[`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph) — and this module wires
**corpus → segment → embed → vd** into a usable search facade with the
progressive disclosure `ef` promises:
- **The light path** — [`ingest()`](_autosummary/ef.source_manager.html.md#ef.source_manager.ingest) — one line. A list of strings (or any
corpus) in, a [`SearchableCorpus`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus) out, `search(query)` ready.
- **The heavy path** — [`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager) — the same machinery with the
components named explicitly and **multiple configs** (a config is one
segmenter+embedder pipeline). Two configs that share a step share that step’s
artifacts in the [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph) for free — config
branching costs only the divergent cone.
- **Keeping it fresh** — [`SourceManager.diagnose()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.diagnose) reports how far the
index has drifted from the corpus (the four staleness conditions);
[`SourceManager.refresh()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.refresh) re-syncs it. Pass `auto_refresh=True` to keep
> the index live as the corpus is edited. See [`ef.diagnostics`](_autosummary/ef.diagnostics.html.md#module-ef.diagnostics) and
[`ef.refresh`](_autosummary/ef.refresh.html.md#module-ef.refresh).
How the wiring works (one source, one config):
1. the source is a graph **leaf**, keyed by [`content_hash()`](_autosummary/ef.corpus.html.md#ef.corpus.content_hash);
2. a `segment` [`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec) over that leaf →
a `list` of [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)s;
3. an `embed` [`ProducerSpec`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ProducerSpec) over the segment
artifact → an `(n, dim)` array;
4. each `(segment, vector)` pair is written into a `vd` collection as a
`vd.Document`, with `source_id` / `source_hash` / `config_hash`
> recorded in its metadata.
`ef` owns the embedding: it always supplies vectors to `vd` (the index is
`vd`’s job, the vectors are `ef`’s — design notes §7), and at query time it
embeds the query with `input_type="query"` and hands `vd` the vector.
Example — the light path, end to end, offline:
```pycon
>>> import numpy as np
>>> from ef import ingest, as_embedder
>>> toy = as_embedder(
... lambda texts: np.array(
... [[t.count('a'), t.count('e'), t.count('o')] for t in texts], dtype=float
... ),
... model_id='toy-vowels@3',
... )
>>> idx = ingest(['apple orchard', 'ocean breeze', 'green meadow'], embedder=toy)
>>> hits = idx.search('ocean breeze', limit=2)
>>> hits[0].segment['text']
'ocean breeze'
>>> round(hits[0].score, 6)
1.0
>>> hits[0].source_id is not None
True
```
And the true light path — no `embedder` at all, nothing to install beyond
`ef` itself (the default is the dependency-free
[`HashingEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.HashingEmbedder)):
```pycon
>>> idx = ingest(['apple orchard', 'ocean breeze', 'green meadow'])
>>> idx.search('ocean breeze', limit=1)[0].segment['text']
'ocean breeze'
```
### Module Attributes
| [`DEFAULT_EMBEDDER`](_autosummary/ef.source_manager.html.md#ef.source_manager.DEFAULT_EMBEDDER) | The model name [`ingest()`](_autosummary/ef.source_manager.html.md#ef.source_manager.ingest) / [`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager) resolve an absent `embedder` to — [`HashingEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.HashingEmbedder), the dependency-free default (numpy only, no extra to install). |
|---------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
### Functions
| [`hits_to_segments`](_autosummary/ef.source_manager.html.md#ef.source_manager.hits_to_segments)(hits) | Project [`SearchHit`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchHit)s to plain [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)s. |
|--------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [`ingest`](_autosummary/ef.source_manager.html.md#ef.source_manager.ingest)(sources, \*[, segmenter, embedder, ...]) | Index `sources` and return a ready-to-search [`SearchableCorpus`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus) — one call. |
### Classes
| [`SearchHit`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchHit)(segment, score[, source_id]) | One ranked search result — a [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) and its score. |
|----------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------|
| [`SearchableCorpus`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus)(collection, embedder, \*[, ...]) | A ready-to-search view over one indexed config — the thin facade. |
| [`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager)([corpus, segmenter, embedder, ...]) | The heavy search facade — multi-config corpus indexing over one corpus. |
### ef.source_manager.DEFAULT_EMBEDDER *= 'hashing'*
The model name [`ingest()`](_autosummary/ef.source_manager.html.md#ef.source_manager.ingest) / [`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager) resolve an absent
`embedder` to — [`HashingEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.HashingEmbedder), the dependency-free
default (numpy only, no extra to install). It is a lexical embedder; pass an
explicit `embedder` (`"st:..."`, `"openai:..."`, …) for neural
semantic quality.
### *class* ef.source_manager.SearchHit(segment, score, source_id=None)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
One ranked search result — a [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) and its score.
[`SearchableCorpus.search()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus.search) / [`SourceManager.search()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.search) return a
ranked `list` of these. Keeping [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) (the
interchange `TypedDict`) pure of a result-only `score` key, a
[`SearchHit`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchHit) *wraps* it alongside the similarity `score` and the
`source_id` of the document the segment was cut from.
#### segment
the matched segment — the canonical [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment).
#### score
the similarity score from the vector store (higher = closer).
#### source_id
the corpus key of the source document, if recorded.
```pycon
>>> hit = SearchHit(segment={'text': 'hi', 'id': 'x'}, score=0.9, source_id='doc-1')
>>> hit.segment['text'], hit.score, hit.source_id
('hi', 0.9, 'doc-1')
```
### *class* ef.source_manager.SearchableCorpus(collection, embedder, , config=None)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
A ready-to-search view over one indexed config — the thin facade.
What [`ingest()`](_autosummary/ef.source_manager.html.md#ef.source_manager.ingest) returns and what [`SourceManager.searchable()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.searchable) hands
out: a small, read-mostly object that knows just enough to answer a query —
the `vd` collection holding the vectors and the [`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)
that embeds the query. It does **not** re-index; mutation, refresh and
multi-config live on [`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager).
* **Parameters:**
* **collection** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the `vd` collection holding this config’s vectors.
* **embedder** ([`Embedder`](_autosummary/ef.embedders.html.md#ef.embedders.Embedder)) – the embedder — used to embed queries with `input_type="query"`.
* **config** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the [`ConfigId`](_autosummary/ef.config.html.md#ef.config.ConfigId) of the indexed pipeline, if known.
```pycon
>>> # built for you by ingest(); see the module docstring for a full example.
```
#### retrieve(query, , limit=10, filter=None)
Retrieve ranked [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)s — the RAG-plug-in surface.
The name an external RAG/agent framework reaches for. Where
[`search()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus.search) returns scored [`SearchHit`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchHit)s (for inspecting the
ranking), `retrieve` returns **plain segments** in rank order — clean
context to hand to an LLM, with no `ef`-specific type to learn (design
notes §F5). `ef` returns the context; it does *not* synthesize answers.
See [`hits_to_segments()`](_autosummary/ef.source_manager.html.md#ef.source_manager.hits_to_segments) for how the score is dropped and the
`source_id` provenance is preserved in `metadata["source"]`.
* **Return type:**
[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]
```pycon
>>> # idx.retrieve('a query') -> [{'text': ..., 'id': ..., 'metadata': ...}, ...]
```
#### search(query, , limit=10, filter=None)
Search the indexed corpus — return up to `limit` ranked [`SearchHit`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchHit)s.
`query` is text (embedded with `input_type="query"`) or an already
computed query vector. `filter` is a `vd` metadata filter
(MongoDB-style) — see `vd.filters`.
* **Return type:**
[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`SearchHit`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchHit)]
### *class* ef.source_manager.SourceManager(corpus=None, , segmenter=None, embedder=None, store=None, cache=None, auto_refresh=False, embedder_api_key=None)
Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)
The heavy search facade — multi-config corpus indexing over one corpus.
A [`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager) holds a corpus, an [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph)
and one or more **configs**. A config is a named segmenter+embedder pipeline
([`PipelineSpec`](_autosummary/ef.config.html.md#ef.config.PipelineSpec)); [`register_config()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.register_config) adds one,
[`materialize()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.materialize) runs the corpus through it into a `vd` collection, and
[`search()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.search) queries it. Because every artifact is content-addressed in
the shared graph, two configs that share a step (e.g. the same segmenter)
compute that step’s artifacts **once** — config branching is free.
The constructor optionally registers a `"default"` config: pass an
`embedder` (and optionally a `segmenter`) and it is registered eagerly;
omit `embedder` to build every config explicitly via [`register_config()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.register_config).
* **Parameters:**
* **corpus** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the source corpus — coerced by [`as_corpus()`](_autosummary/ef.corpus.html.md#ef.corpus.as_corpus)
(a mapping, a directory path, an iterable of sources, or `None`).
* **segmenter** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the default config’s segmenter — coerced by
[`as_segmenter()`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.as_segmenter) (`None` → the recursive
default). Used only when `embedder` is also given.
* **embedder** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the default config’s embedder — coerced by
[`as_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.as_embedder). If given, a `"default"`
config is registered; if `None`, no default config is registered.
* **embedder_api_key** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – an optional API key for the default config’s embedder
— forwarded to a hosted-API adapter (`"openai:…"` etc.) when the
`embedder` is such a spec. The bring-your-own-key seam; ignored
for key-less embedders. See [`register_config()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.register_config).
* **store** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – where the vectors go. `None` → an in-memory `vd` backend; a
`vd` client → collections are created on it; a `vd` collection →
used directly (then only one config is supported); a backend-name
string → `vd.connect` of that backend.
* **cache** ([`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph)’s value cache — any
`MutableMapping`; `None` → an in-RAM `dict`.
* **auto_refresh** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – when `True`, the corpus is wrapped in a
[`ChangeDetectingCorpus`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus) and every edit made
*through* it is incrementally re-indexed into each materialized
config — the index stays live without an explicit
[`refresh()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.refresh). [`scan()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.scan) then also picks up out-of-band edits.
```pycon
>>> import numpy as np
>>> from ef import as_embedder
>>> e1 = as_embedder(lambda ts: np.array([[len(t)] for t in ts], float), model_id='len@1')
>>> sm = SourceManager(['hello', 'hi there'], embedder=e1)
>>> report = sm.materialize()
>>> report['segments']
2
>>> len(sm.search('hello')) >= 1
True
```
#### *property* configs *: [dict](https://docs.python.org/3/builtins/stdtypes.html#dict)[[str](https://docs.python.org/3/builtins/stdtypes.html#str), [str](https://docs.python.org/3/builtins/stdtypes.html#str)]*
The registered configs as a `{name: config_id}` mapping.
#### diagnose(config=None)
Report how far one config’s index has drifted from the corpus.
Computes the four staleness conditions — `orphan` / `missing` /
`stale` / `misconfigured` — by comparing the config’s `vd`
collection against the current corpus (see [`ef.diagnostics`](_autosummary/ef.diagnostics.html.md#module-ef.diagnostics)).
Read-only: it changes nothing. [`refresh()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.refresh) acts on the result.
* **Return type:**
[`StalenessReport`](_autosummary/ef.diagnostics.html.md#ef.diagnostics.StalenessReport)
```pycon
>>> sm = SourceManager({'a': 'alpha', 'b': 'beta'}, embedder='hashing')
>>> _ = sm.materialize()
>>> bool(sm.diagnose()) # freshly indexed — in sync
False
>>> sm.corpus['a'] = 'alpha rewritten'
>>> sm.diagnose().stale
('a',)
```
#### gc_orphans(config=None)
Delete the documents of sources that left the corpus — return the count.
The garbage-collection slice of [`refresh()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.refresh): it removes only
**orphan** documents (a source gone from the corpus) and prunes the
dead leaves out of the [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph),
touching nothing else. `gc_orphans()` is the orphan-deletion half of
`refresh(mode='full')`.
* **Return type:**
[`int`](https://docs.python.org/3/builtins/functions.html#int)
#### lineage(key)
The artifacts `key` was produced from — its provenance.
A thin pass-through to [`ArtifactGraph.ancestors`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.ancestors): “what produced this
vector / segment?”. The leaves of the returned set are the source
content hashes the artifact ultimately derives from.
* **Return type:**
[`frozenset`](https://docs.python.org/3/builtins/stdtypes.html#frozenset)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]
#### materialize(config=None, , sources=None)
Index the corpus through one config (or all) — corpus → segment → embed → `vd`.
For each source and each selected config: the source is added to the
[`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph) as a content-addressed leaf,
its `segment` and `embed` artifacts are [`materialize()`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph.materialize)d
(a cache hit if already computed — so a shared segment step runs once
across configs), and every `(segment, vector)` pair is written into
the config’s `vd` collection. Idempotent: re-materializing rewrites
identical documents.
* **Parameters:**
* **config** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the config to index — a name, a [`ConfigId`](_autosummary/ef.config.html.md#ef.config.ConfigId),
or `None` for *every* registered config.
* **sources** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – an optional subset of source ids to index — `None` (the
default) indexes the whole corpus. [`refresh()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.refresh) uses this to
re-index only the sources that changed; ids not in the corpus
are skipped.
* **Return type:**
[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]
* **Returns:**
a report `dict` with `configs` / `sources` / `segments` counts.
#### pipeline(config=None)
The [`PipelineSpec`](_autosummary/ef.config.html.md#ef.config.PipelineSpec) of a registered config.
* **Return type:**
[`PipelineSpec`](_autosummary/ef.config.html.md#ef.config.PipelineSpec)
#### rebuild(config=None)
Drop one config’s index entirely and re-index the whole corpus.
The heavy hammer: every document in the config’s `vd` collection is
deleted, then the whole corpus is re-[`materialize()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.materialize)d from
scratch. Use it to recover from a config whose `vd` documents were
produced by an earlier pipeline (every source `misconfigured`), or
whenever a guaranteed-clean index is wanted over the surgical
[`refresh()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.refresh). Every source is reported as `added` — the collection
was emptied and rebuilt.
* **Return type:**
[`RefreshReport`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshReport)
* **Returns:**
a [`RefreshReport`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshReport).
#### refresh(config=None, , sources=None, mode='full')
Re-sync one config’s `vd` index with the current corpus.
The explicit-refresh entry point. It [`diagnose()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.diagnose)s the config,
turns the staleness report into a [`RefreshPlan`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshPlan) for
the chosen `mode` ([`plan_refresh()`](_autosummary/ef.refresh.html.md#ef.refresh.plan_refresh)), deletes the
stale/orphan documents, re-[`materialize()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.materialize)s what changed, and
prunes the dead leaves out of the [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph).
Because it re-reads and re-hashes the corpus, it catches out-of-band
edits without any change-detection wrapper.
* **Parameters:**
* **config** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the config to refresh (name / id / `None` for the default
or sole config).
* **sources** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – an optional subset of source ids to refresh — sources
outside it are left untouched. Pair it with `mode='scoped_full'`
(or `'incremental'`); `'full'` rejects it.
* **mode** ([`Literal`](https://docs.python.org/3/library/typing.html#typing.Literal)[`'none'`, `'incremental'`, `'full'`, `'scoped_full'`]) – one of [`REFRESH_MODES`](_autosummary/ef.refresh.html.md#ef.refresh.REFRESH_MODES) — `'none'` (index
new/changed content, delete nothing), `'incremental'` (also
replace changed sources), `'full'` (also delete orphans — the
corpus is authoritative), `'scoped_full'` (`full` over the
`sources` subset). Default `'full'`.
* **Return type:**
[`RefreshReport`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshReport)
* **Returns:**
a [`RefreshReport`](_autosummary/ef.refresh.html.md#ef.refresh.RefreshReport).
```pycon
>>> sm = SourceManager({'a': 'alpha text', 'b': 'beta text'}, embedder='hashing')
>>> _ = sm.materialize()
>>> sm.corpus['b'] = 'beta rewritten' # edit
>>> sm.corpus['c'] = 'gamma text' # add
>>> del sm.corpus['a'] # delete
>>> report = sm.refresh()
>>> report.added, report.modified, report.deleted
(('c',), ('b',), ('a',))
>>> bool(sm.diagnose()) # in sync again
False
```
#### register_config(name='default', , segmenter=None, embedder=None, embedder_api_key=None)
Register a named segmenter+embedder pipeline; return its [`ConfigId`](_autosummary/ef.config.html.md#ef.config.ConfigId).
The segmenter and embedder are coerced through `ef`’s DI seams
([`as_segmenter()`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.as_segmenter) /
[`as_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.as_embedder); an absent `embedder`
defaults to [`DEFAULT_EMBEDDER`](_autosummary/ef.source_manager.html.md#ef.source_manager.DEFAULT_EMBEDDER)). Their ops are registered into the
shared [`ArtifactGraph`](_autosummary/ef.artifact_graph.html.md#ef.artifact_graph.ArtifactGraph) and a `vd` collection
is created for the config. Registering a config does *not* index it —
call [`materialize()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.materialize).
`embedder_api_key` is the per-call bring-your-own-key seam: when given
and `embedder` is a hosted-API provider spec (`"openai:…"` etc.),
the key is forwarded to that adapter instead of relying on a server
environment variable. Ignored for key-less embedders.
Re-registering the same `name` replaces it. Two configs that resolve
to the same [`PipelineSpec`](_autosummary/ef.config.html.md#ef.config.PipelineSpec) share one
[`ConfigId`](_autosummary/ef.config.html.md#ef.config.ConfigId) (and one collection).
* **Return type:**
[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)
```pycon
>>> import numpy as np
>>> from ef import as_embedder
>>> sm = SourceManager(['a', 'b'])
>>> cid = sm.register_config(
... 'mini', embedder=as_embedder(
... lambda ts: np.ones((len(ts), 2)), model_id='ones@2'))
>>> len(cid)
64
```
#### retrieve(query, , config=None, limit=10, filter=None)
Retrieve ranked [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)s — the RAG-plug-in surface.
The [`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager) counterpart of
[`SearchableCorpus.retrieve()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus.retrieve): plain segments in rank order, the
clean context shape an external RAG/agent framework consumes (`ef`
does not synthesize answers). [`search()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.search) is the scored counterpart.
* **Return type:**
[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]
#### scan()
Re-scan the corpus for out-of-band edits — needs `auto_refresh=True`.
Picks up changes made to the backing store *directly* (a file edited on
disk, an S3 object replaced) rather than through the corpus wrapper.
When the manager was built with `auto_refresh=True` the corpus is a
[`ChangeDetectingCorpus`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus); this calls its
[`scan()`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeDetectingCorpus.scan), which fires a
[`ChangeEvent`](_autosummary/ef.corpus.html.md#ef.corpus.ChangeEvent) per drift — each one auto-applied to
every materialized config. Returns the [`CorpusDiff`](_autosummary/ef.corpus.html.md#ef.corpus.CorpusDiff).
For a large bulk of out-of-band edits, prefer [`refresh()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.refresh) (one
corpus pass) over `scan` (one collection pass per changed source).
* **Raises:**
[**TypeError**](https://docs.python.org/3/builtins/exceptions.html#TypeError) – if the corpus is not change-detecting (the manager was
not built with `auto_refresh=True`).
* **Return type:**
[`CorpusDiff`](_autosummary/ef.corpus.html.md#ef.corpus.CorpusDiff)
#### search(query, , config=None, limit=10, filter=None)
Search one config’s index — return up to `limit` ranked [`SearchHit`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchHit)s.
`config` selects which config to query (a name / [`ConfigId`](_autosummary/ef.config.html.md#ef.config.ConfigId),
or `None` for the default or sole config). `query` is text or a
pre-computed query vector; `filter` is a `vd` metadata filter.
* **Return type:**
[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`SearchHit`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchHit)]
#### searchable(config=None)
Return a thin [`SearchableCorpus`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus) bound to one config’s index.
* **Return type:**
[`SearchableCorpus`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus)
### ef.source_manager.hits_to_segments(hits)
Project [`SearchHit`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchHit)s to plain [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)s.
The shape [`SearchableCorpus.retrieve()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus.retrieve) / [`SourceManager.retrieve()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.retrieve)
hand to an external RAG/agent framework — the RAG-plug-in surface (design
notes §F5). A [`SearchHit`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchHit) carries an `ef`-specific `score` and
`source_id`; a [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) is a plain `TypedDict` an
external framework already understands (and trivially adapts to a LangChain
`Document` / LlamaIndex `TextNode` / a Ragas `retrieved_contexts`
`list[str]` via `[s["text"] for s in segments]`).
The similarity `score` is dropped — [`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment) is the
interchange type and stays pure of a result-only key (call `search()`
when the score matters). Provenance is *not* lost: each hit’s `source_id`
is folded into the segment’s `metadata` under the conventional `"source"`
key (one of [`PROMOTED_METADATA_KEYS`](_autosummary/ef.segments.html.md#ef.segments.PROMOTED_METADATA_KEYS)), so a plain segment
still records which source document it was retrieved from. A segment that
already carries `metadata["source"]` keeps its own value.
* **Return type:**
[`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`Segment`](_autosummary/ef.segments.html.md#ef.segments.Segment)]
```pycon
>>> hit = SearchHit(segment={'text': 'hi', 'id': 'x'}, score=0.9, source_id='doc-1')
>>> segs = hits_to_segments([hit])
>>> segs[0]['text'], segs[0]['metadata']['source']
('hi', 'doc-1')
>>> 'score' in segs[0] # the score is not a Segment key
False
```
### ef.source_manager.ingest(sources, , segmenter=None, embedder=None, store=None, cache=None)
Index `sources` and return a ready-to-search [`SearchableCorpus`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus) — one call.
The light path: corpus in, `search`-ready object out, every component
defaulted. `ingest(['a', 'b', 'c']).search('query')` is the whole story.
It builds a single-config [`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager), [`materialize()`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager.materialize)s
it, and hands back the thin [`SearchableCorpus`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus).
* **Parameters:**
* **sources** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the corpus — coerced by [`as_corpus()`](_autosummary/ef.corpus.html.md#ef.corpus.as_corpus) (a mapping,
a directory path, an iterable of strings/mappings, or `None`).
* **segmenter** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the segmenter — [`as_segmenter()`](_autosummary/ef.segmenter_adapters.html.md#ef.segmenter_adapters.as_segmenter)
coerces it (`None` → the recursive default).
* **embedder** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the embedder — [`as_embedder()`](_autosummary/ef.embedder_adapters.html.md#ef.embedder_adapters.as_embedder)
coerces it; `None` → [`DEFAULT_EMBEDDER`](_autosummary/ef.source_manager.html.md#ef.source_manager.DEFAULT_EMBEDDER), the dependency-free
[`HashingEmbedder`](_autosummary/ef.embedders.html.md#ef.embedders.HashingEmbedder) (so this call needs nothing
beyond `pip install ef`). Pass an explicit embedder — e.g.
`"st:all-MiniLM-L6-v2"` — for neural semantic quality.
* **store** ([`Any`](https://docs.python.org/3/library/typing.html#typing.Any)) – the vector store — `None` → an in-memory `vd` backend. See
[`SourceManager`](_autosummary/ef.source_manager.html.md#ef.source_manager.SourceManager) for the other accepted forms.
* **cache** ([`MutableMapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – the artifact-graph value cache — any `MutableMapping`.
* **Return type:**
[`SearchableCorpus`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus)
* **Returns:**
a [`SearchableCorpus`](_autosummary/ef.source_manager.html.md#ef.source_manager.SearchableCorpus) over the freshly indexed corpus.
See the module docstring for a complete offline example.
# about-this-build.html.md
# About this build
This documentation was built on **2026-10-14 03:52 UTC** from commit 15ac259 on branch main, for **ef 0.1.22** (from pyproject.toml).
#### WARNING
The documentation and the package may be misaligned:
- The documented version (0.1.22) is behind the latest release on PyPI (0.1.23): `pip install ef` gives newer code than these docs describe.
## Source
| | |
|---------------------|------------------------------------------------------------------------------------------------------------------------------------------------------|
| Commit | 15ac259c80bdc5077b683499982a4e35a1eca089 |
| Branch | main |
| Tags at this commit | none |
| Working tree | clean |
| Remote | https://github.com/thorwhalen/ef |
## Continuous integration
| | |
|--------------|--------------------------------------------------------------------------------------------|
| Repository | thorwhalen/ef |
| Run | 38508395735 |
| Ref | refs/heads/main |
| Event commit | 15ac259c80bdc5077b683499982a4e35a1eca089 (in the history of the built commit) |
## Tools
| | |
|----------|---------|
| epythet | 0.2.17 |
| Sphinx | 9.1.0 |
| docutils | 0.22.4 |
| Python | 3.12.15 |
## Configuration as resolved
| | |
|---------------|-------------------------------------------------------------------|
| theme | auto (Sphinx theme sphinxawesome_theme) |
| accent | #783b83 |
| api_generator | autosummary |
| ignore | tests/, scrap/, examples/ |
| agent_outputs | true |
| aggregates | md |
| ai_artifacts | true |
## Package on PyPI
Latest release: 0.1.23, newer than the documented version (0.1.22).
## Reproduce
```bash
git clone https://github.com/thorwhalen/ef && cd ef
git checkout 15ac259c80bdc5077b683499982a4e35a1eca089
pip install "epythet==0.2.17"
epythet quickstart . --ignore tests/ scrap/ examples/
```
The same data, for machines: build_info.json (schema version 1).
# ai-agents.html.md
# For AI agents
`ef` ships artifacts for coding agents alongside its code. This page lists
them, says where each lives in the repository, and points at the
machine-readable copies of this documentation.
## Skills
Skills are folders holding a `SKILL.md` (the [Agent Skills](https://agentskills.io) format): a description that tells an agent when to use it and a body with the procedure. Install one into your agent with `gh skill` (any host: `--agent claude-code`, `copilot`, `cursor`, `codex`, `gemini`), or use the copy bundled in the wheel.
### `ef-architecture`
Developer skill: the target architecture of the ef (Embedding Flow) package. Load at the start of any non-trivial task in the ef repository — refactoring, adding the Embedder/Segmenter facades, the Corpus abstraction, the ArtifactGraph indexing core, or the “ready search” surface. Covers the five-layer model, the core contracts, the light/heavy progressive-disclosure split, and the hard boundaries. Trigger on “work on ef”, “refactor ef”, “implement the embedder/segmenter/corpus/artifact graph”, “ef search”.
```bash
gh skill install thorwhalen/ef ef-architecture --agent claude-code
```
Source: [`skills/ef-architecture`](https://github.com/thorwhalen/ef/tree/HEAD/skills/ef-architecture).
## Instruction files
Files agents read before working in this repository.
- [`.claude/CLAUDE.md`](https://github.com/thorwhalen/ef/tree/HEAD/.claude/CLAUDE.md): read by Claude Code
## Machine-readable documentation
This site publishes the same documentation in forms that fit an agent’s context window:
- [`llms.txt`](https://thorwhalen.github.io/ef/llms.txt): an index of every page with a one-line description ([llms.txt](https://llmstxt.org) format)
- [`ef.md`](https://thorwhalen.github.io/ef/ef.md): the whole documentation as one Markdown file
- `.html.md`: a rendered Markdown twin of every page, advertised from each page’s `` with ``
- [`objects.inv`](https://thorwhalen.github.io/ef/objects.inv): the Sphinx inventory: a symbol-to-URL index (`sphobjinv convert plain objects.inv -`)
# api.html.md
# API reference
| [`ef`](_autosummary/ef.html.md#module-ef) | ef (Embedding Flow) — a facade for semantic-embeddings user journeys. |
|-----------------------------------------------------------------|-------------------------------------------------------------------------|