ef.config¶

The config layer — declarative, content-addressable pipeline specs.

A pipeline in ef is not an object graph you wire by hand — it is data. A PipelineSpec declares the segment→embed chain; each step is a TransformSpec (an op key, an op version, and the keyword params that feed the content hash). Because a spec is fully serializable and content-hashed (config_id()), two configs that share a step share that step’s artifacts in the ArtifactGraph for free — config branching costs nothing beyond the divergent cone (design notes §3.1).

This module is the bridge between an ordinary Python call and a content- addressed graph node. For content addressing to be correct, two semantically-equal calls must produce identical params — so op(text) and op(text, size=512) (512 being the default) must hash the same. The canonical i2 idiom does exactly that normalization:

full_kwargs() — the raw recipe (design notes §3.8): a call (args, kwargs) → a fully-named, defaults-filled kwargs dict.
step_params() — its spec-building application: the canonical keyword params of an op, with the leading positional input slots (the upstream artifact ids) dropped, since those become ProducerSpec.inputs, not params.

Phase 4’s ArtifactGraph takes ProducerSpecs as given; this module (and ef.ops / ef.source_manager) is where ordinary calls become those specs.

Example — two pipelines, equal segment step, divergent embed step:

>>> seg = TransformSpec(op='segment:recursive', op_version='1', params={})
>>> embed_a = TransformSpec(op='embed:m-a', op_version='1', params={})
>>> embed_b = TransformSpec(op='embed:m-b', op_version='1', params={})
>>> pa = PipelineSpec(segment=seg, embed=embed_a)
>>> pb = PipelineSpec(segment=seg, embed=embed_b)
>>> config_id(pa) == config_id(pb)        # different embedders → different config
False
>>> config_id(pa) == config_id(PipelineSpec.from_dict(pa.as_dict()))
True

Module Attributes

ConfigId

A content-addressed pipeline identifier — the sha256 hex digest of a PipelineSpec (config_id()).

Functions

`full_kwargs`(op[, args, kwargs])	Normalize a call into a canonical, fully-named, defaults-filled kwargs dict.
`step_params`(op[, raw_params, n_inputs])	The canonical keyword params of an op step — its inputs dropped.
`config_id`(spec)	Content-addressed id of a `PipelineSpec` — `sha256` over its steps.

Classes

`ConfigId`	A content-addressed pipeline identifier — the `sha256` hex digest of a `PipelineSpec` (`config_id()`).
`TransformSpec`(op, op_version[, params])	One step of a pipeline — a serializable recipe for an op application.
`PipelineSpec`(segment, embed)	A whole indexing pipeline as data — the segment→embed chain.

ef.config.ConfigId¶: A content-addressed pipeline identifier — the sha256 hex digest of a PipelineSpec (config_id()). It is what ef writes into vd index metadata as config_hash so a later phase can tell which pipeline produced a vector.

class ef.config.PipelineSpec(segment, embed)[source]¶

Bases: object

A whole indexing pipeline as data — the segment→embed chain.

A PipelineSpec is the declarative description of one config: a TransformSpec for the segmenter and one for the embedder. It is fully serializable (as_dict() / from_dict()) and content-hashed by config_id(). Registering a second PipelineSpec that differs only in its embed step shares the segment artifacts in the ArtifactGraph — that is config branching, and the graph gives it for free.

The segment``→``embed ordering is the input wiring: the embed step consumes the segment step’s output. (Phase 5 fixes this two-step shape; a later phase may generalize it to an arbitrary step list.)

>>> seg = TransformSpec(op='segment:r', op_version='1', params={})
>>> emb = TransformSpec(op='embed:m@8', op_version='1', params={'input_type': 'document'})
>>> spec = PipelineSpec(segment=seg, embed=emb)
>>> spec.embed.params['input_type']
'document'
>>> PipelineSpec.from_dict(spec.as_dict()) == spec
True

as_dict()[source]¶

Return a plain-dict (JSON-serializable) form of the pipeline.

Return type:: dict[str, Any]

>>> seg = TransformSpec(op='segment:r', op_version='1', params={})
>>> emb = TransformSpec(op='embed:m', op_version='1', params={})
>>> PipelineSpec(segment=seg, embed=emb).as_dict()['segment']['op']
'segment:r'

classmethod from_dict(d)[source]¶

Rebuild a PipelineSpec from its as_dict() form.

Return type:: PipelineSpec

class ef.config.TransformSpec(op, op_version, params=<factory>)[source]¶

Bases: object

One step of a pipeline — a serializable recipe for an op application.

A TransformSpec names which op (op, a key resolved through the ArtifactGraph’s ops registry), *at which version* (op_version — it participates in every downstream hash), and with which keyword params (params). It does not record the step’s inputs: those are wired structurally by the PipelineSpec (the embed step consumes the segment step’s output). It is the per-step counterpart of a ProducerSpec.

The op key carries the component identity — e.g. "embed:openai:text- embedding-3-large@1024" or "segment:<segmenter-identity>" (see ef.ops) — so two configs with different components get disjoint artifact cones automatically.

>>> ts = TransformSpec(op='embed:m@8', op_version='2', params={'input_type': 'document'})
>>> ts.op, ts.op_version
('embed:m@8', '2')
>>> TransformSpec.from_dict(ts.as_dict()) == ts
True

as_dict()[source]¶

Return a plain-dict (JSON-serializable) form of the spec.

Return type:: dict[str, Any]

>>> TransformSpec(op='segment:r', op_version='1', params={}).as_dict()
{'op': 'segment:r', 'op_version': '1', 'params': {}}

classmethod from_dict(d)[source]¶

Rebuild a TransformSpec from its as_dict() form.

Return type:: TransformSpec

ef.config.config_id(spec)[source]¶

Content-addressed id of a PipelineSpec — sha256 over its steps.

The id is sha256 of the canonical (sorted-key) JSON of PipelineSpec.as_dict(). Two pipelines that describe the same computation hash identically; any change to a step’s op, version or params yields a different id. ef writes this digest into vd index metadata as config_hash.

Return type:: str

>>> seg = TransformSpec(op='segment:r', op_version='1', params={})
>>> emb = TransformSpec(op='embed:m@8', op_version='1', params={})
>>> cid = config_id(PipelineSpec(segment=seg, embed=emb))
>>> len(cid)
64
>>> cid == config_id(PipelineSpec(segment=seg, embed=emb))
True

ef.config.full_kwargs(op, args=(), kwargs=None)[source]¶

Normalize a call into a canonical, fully-named, defaults-filled kwargs dict.

The i2.Sig recipe of design notes §3.8. A call expressed as positional args + keyword kwargs is mapped onto op’s signature: every parameter is named, and any parameter the call left implicit is filled with its default. The result is therefore stable — op(x) and op(x, size=512) (512 the default) yield the same dict, which is what makes a content hash built from it correct.

Parameters:

op (Callable[..., Any]) – the callable whose signature defines the parameter names/defaults.
args (tuple[Any, ...]) – the call’s positional arguments — for an ef.ops op these are the upstream artifact ids; step_params() drops them again.
kwargs (Mapping[str, Any] | None) – the call’s keyword arguments.

Return type:

dict[str, Any]

>>> def embed(segments, *, input_type='document', dim=None): ...
>>> full_kwargs(embed, ('seg-1',), {'input_type': 'query'})
{'segments': 'seg-1', 'input_type': 'query', 'dim': None}
>>> full_kwargs(embed, ('seg-1',))                 # defaults filled in
{'segments': 'seg-1', 'input_type': 'document', 'dim': None}

ef.config.step_params(op, raw_params=None, *, n_inputs=1)[source]¶

The canonical keyword params of an op step — its inputs dropped.

An ef.ops op is called op(*input_values, **params): the first n_inputs positional parameters are upstream artifacts (they belong in ProducerSpec.inputs, not in the params), and every remaining parameter is a hashable step param. This runs full_kwargs() with placeholder inputs, then drops those input slots — leaving exactly the params, defaults filled, ready for a TransformSpec.

Parameters:

op (Callable[..., Any]) – the op callable whose signature is the param contract.
raw_params (Mapping[str, Any] | None) – the keyword params the caller actually supplied; anything omitted is defaulted.
n_inputs (int) – how many leading positional parameters are upstream inputs.

Return type:

dict[str, Any]

>>> def embed(segments, *, input_type='document'): ...
>>> step_params(embed, {'input_type': 'query'})
{'input_type': 'query'}
>>> step_params(embed)                              # default filled in
{'input_type': 'document'}
>>> def segment(source): ...
>>> step_params(segment)                            # no params at all
{}