ef.segments¶

The canonical Segment data model — ef’s interchange type for L2.

A segment is a piece of text carved out of a source document by a Segmenter. This module defines the one shape every segmenter, adapter and store in ef agrees on, so segments round-trip losslessly across the whole pipeline.

Two surfaces over the same data:

Segment — a TypedDict: the lightweight, streaming-friendly interchange type. A segment is just a dict; the type is passive and every helper here is a free function over it.
SegmentRecord — a frozen dataclass: the convenience surface for code that prefers attribute access and immutability.

Only text is genuinely required; id is required by convention and is content-derived by default (segment_id()) so that re-segmenting the same document yields the same ids — the basis of idempotent ingestion.

The two field names that are promoted to top level (rather than buried in metadata) are parent_id and index — hierarchy is expressed by a flat list with parent_id pointers, never a nested structure.

Example — build a segment, derive its id, view it as a record:

>>> seg = make_segment('Hello world.', index=0, tokens=3)
>>> seg['text']
'Hello world.'
>>> seg['id'] == segment_id('Hello world.')
True
>>> rec = segment_record(seg)
>>> rec.text, rec.index
('Hello world.', 0)
>>> rec.to_segment() == seg
True

Module Attributes

PROMOTED_METADATA_KEYS

Metadata keys that, by convention, every ef adapter must round-trip faithfully.

Functions

`segment_id`(text[, metadata])	Content-derived id for a segment — `sha256` of its normalized content.
`make_segment`(text, *[, id, parent_id, ...])	Build a `Segment`, deriving a content-based `id` if none given.
`as_segment`(x, **defaults)	Coerce `x` into a `Segment` — accepts a `str`, `Mapping` or record.
`segment_record`(x)	Coerce `x` (`str` / `Mapping` / `SegmentRecord`) into a record.

Classes

`Segment`	A piece of text carved from a source document — the interchange type.
`SegmentRecord`(text[, id, parent_id, start, ...])	Frozen-dataclass view over a `Segment` — the convenience surface.

ef.segments.PROMOTED_METADATA_KEYS = ('source', 'source_type', 'tokenizer', 'token_count', 'embedding_model', 'page', 'license', 'ingestion_run_id')¶: Metadata keys that, by convention, every ef adapter must round-trip faithfully. They live inside Segment["metadata"] (they are not promoted to top-level fields) but carry meaning the pipeline relies on. Recording tokenizer in particular is mandatory — omitting it is the single most common silent bug in chunking pipelines (a segment’s tokens count is meaningless without knowing which tokenizer produced it).

class ef.segments.Segment[source]¶

Bases: TypedDict

A piece of text carved from a source document — the interchange type.

A TypedDict: a Segment is a plain dict, which keeps it cheap to create and stream. total=False because most keys are optional; by convention text is always present and id is always set (derived from text if not supplied).

Keys:

text: The segment’s text. Required. id: Stable identifier — content-derived by default (segment_id()). parent_id: Id of the source document or parent segment this came from. start: Character offset of the segment’s start in the source text. end: Character offset of the segment’s end (exclusive) in the source. index: Ordinal position of the segment in its segmenter’s output. tokens: Token count of text (meaningful only with metadata key

tokenizer recording which tokenizer counted it).

metadata: Free-form mapping; framework-specific keys live here. See: PROMOTED_METADATA_KEYS for the conventional keys.

class ef.segments.SegmentRecord(text, id='', parent_id=None, start=None, end=None, index=None, tokens=None, metadata=<factory>)[source]¶

Bases: object

Frozen-dataclass view over a Segment — the convenience surface.

Use this when attribute access and immutability read better than dict access; use the Segment TypedDict directly on the hot path. The two convert losslessly via to_segment() and segment_record().

id is derived from text (and metadata) if left empty.

>>> rec = SegmentRecord('hello', index=2)
>>> rec.id == segment_id('hello')
True
>>> rec.to_segment()['index']
2

to_segment()[source]¶

Return an equivalent Segment dict (dropping empty keys).

Return type:: Segment

ef.segments.as_segment(x, **defaults)[source]¶

Coerce x into a Segment — accepts a str, Mapping or record.

This is the segment-side normalizer that lets a Segmenter accept loosely-typed output (a bare string, a partial dict) and still emit a well-formed Segment. **defaults supplies fallback values: a default fills a field only when x does not already carry it.

Accepts:

a SegmentRecord — converted via SegmentRecord.to_segment();
a str — becomes the text of a fresh segment;
a Mapping with a text key — its keys are carried through.

The id is always present on the result (derived if absent).

Raises:

TypeError – if x is none of the accepted forms.
ValueError – if x is a Mapping without a text key.

Return type:

Segment

>>> as_segment('plain text', index=3)['index']
3
>>> as_segment({'text': 'kept', 'index': 0}, index=9)['index']  # x wins
0

Return type:: Segment

ef.segments.make_segment(text, *, id=None, parent_id=None, start=None, end=None, index=None, tokens=None, metadata=None)[source]¶

Build a Segment, deriving a content-based id if none given.

Optional fields are omitted from the result when None — a segment dict carries only the keys it actually has.

Return type:: Segment

>>> seg = make_segment('chunk one', index=0, start=0, end=9)
>>> sorted(seg)
['end', 'id', 'index', 'start', 'text']
>>> seg['id'] == segment_id('chunk one')
True

ef.segments.segment_id(text, metadata=None)[source]¶

Content-derived id for a segment — sha256 of its normalized content.

The id is sha256(normalize(text) + canonical_json(metadata)) — the normalization and hashing primitives live in ef.hashing, the single source of truth ef content-addresses through. Because the id depends only on content, re-segmenting an unchanged document produces identical ids — which is what makes ingestion idempotent (a re-run is a no-op, not a duplicate). Pass metadata only when two segments could share identical text yet must stay distinct (e.g. same boilerplate from different sources).

Return type:: str

>>> segment_id('hello') == segment_id('hello')
True
>>> segment_id('hello') == segment_id('world')
False
>>> segment_id('x', {'source': 'a'}) == segment_id('x', {'source': 'b'})
False

ef.segments.segment_record(x)[source]¶

Coerce x (str / Mapping / SegmentRecord) into a record.

The dataclass counterpart of as_segment() — the convenience-surface constructor.

Return type:: SegmentRecord

>>> segment_record('hello').text
'hello'
>>> segment_record({'text': 'hi', 'tokens': 1}).tokens
1