ef.segmenters

The Segmenter facade — ef’s segment layer (L2).

A segmenter turns a document into a stream of Segment pieces. Like the embedder facade, its core is a tiny structural contract:

Segmenter = Callable[[str | Mapping], Iterable[Segment]]

Everything else — overlap, hierarchy, batching, the recursive splitting algorithm — is layered scaffolding over that one shape.

This module defines:

• Segmenter — the @runtime_checkable protocol (a callable). It is a protocol, not an ABC: any matching callable is a segmenter.

• BatchedSegmenter — an optional optimization trait (a batch method) for segmenters that can process many documents at once.

• BaseSegmenter — an implementation-convenience base (a default batch() and a repr). Subclassing is optional.

• RecursiveCharacterSegmenter — the default segmenter: recursive character splitting at ~512 tokens with ~12.5% overlap. The honest default — within a point or two of fancier methods on realistic corpora.

• line_segmenter() — a builtin that splits a document into its non-empty lines (a plain function — proof that a bare callable is a segmenter).

• FunctionSegmenter — wraps a bare text -> pieces callable into a full segmenter; the bridge for imbed’s registered segmenters.

• Composition helpers — with_overlap(), hierarchical(), materialise().

Streaming is first-class: a segmenter returns an iterable, not a list, so the heavy case (a huge corpus) never has to fit every segment in memory at once.

The dependency-injection seam as_segmenter() and the imbed registry bridge live in ef.segmenter_adapters.

Example — split a document into ~4-token chunks with one token of overlap:

>>> seg = RecursiveCharacterSegmenter(chunk_size=4, chunk_overlap=1)
>>> [c['text'] for c in seg('alpha beta gamma delta epsilon')]
['alpha beta gamma delta', 'delta epsilon']
>>> isinstance(seg, Segmenter)
True

ef.segmenters.APPROX_TOKENIZER = 'char-approx'

Identifier recorded as the tokenizer metadata key when token counts come from approx_token_count(). Recording which tokenizer counted a segment is mandatory — a tokens number is meaningless without it.

class ef.segmenters.BaseSegmenter

Implementation base for segmenters (optional — Segmenter is structural).

A subclass implements __call__; in return it gets a default batch() (one-document-at-a-time — override it for a genuinely batched backend) and a readable repr. Subclassing is purely a convenience; satisfying Segmenter structurally is what matters.

batch(docs: Iterable[str | Mapping[str, Any]], /) → Iterator[Iterable[Segment]]

Default: segment each document independently.

class ef.segmenters.BatchedSegmenter(*args, **kwargs)

Optional trait: a segmenter that can process many documents at once.

Plain Segmenter handles one document per call. A segmenter that can amortize per-call cost across documents (a shared model load, a vectorized pass) also offers batch. BaseSegmenter supplies a correct — if un-optimized — default, so every BaseSegmenter subclass is already a BatchedSegmenter.

batch(docs: Iterable[str | Mapping[str, Any]], /) → Iterable[Iterable[Segment]]

Segment each document in docs, yielding one segment-stream each.

ef.segmenters.DEFAULT_SEPARATORS: tuple[str, ...] = ('\n\n', '\n', '. ', ', ', ' ', '')

Separators tried, in order, by RecursiveCharacterSegmenter. Earlier (coarser) boundaries are preferred; the empty string is the last resort, splitting between characters so an oversized run is never left unsplit.

class ef.segmenters.FunctionSegmenter(func: ~typing.Callable[[...], ~typing.Any], *, pass_doc: bool = False, count_tokens: ~typing.Callable[[str], int] | None = <function approx_token_count>, tokenizer: str = 'char-approx')

Promote a bare text -> pieces callable to a full Segmenter.

The bridge for segmenter functions that carry no segment schema — most importantly imbed’s registered segmenters, which map text to an iterable of plain strings. Each piece the callable yields is normalized:

• a str piece becomes a Segment; its character offsets are recovered by locating it in the source text, and a token count plus tokenizer metadata are attached;

• a Mapping / SegmentRecord piece is coerced with as_segment(), gaining only an index / parent_id where it lacks them.

Parameters:

• func – The callable to wrap. By default it receives the document’s text; with pass_doc=True it receives the whole document.

• pass_doc – Pass the raw document instead of its text.

• count_tokens – Token counter for str pieces (None disables the tokens / tokenizer annotation).

• tokenizer – Name recorded as the tokenizer metadata key.

>>> fs = FunctionSegmenter(lambda text: text.split(','))
>>> [s['text'] for s in fs('a,b,c')]
['a', 'b', 'c']
>>> s = list(fs('a,b,c'))[1]
>>> (s['start'], s['end'], s['index'])
(2, 3, 1)

class ef.segmenters.RecursiveCharacterSegmenter(*, chunk_size: int = 512, chunk_overlap: int = 64, separators: ~typing.Sequence[str] = ('\n\n', '\n', '. ', ', ', ' ', ''), count_tokens: ~typing.Callable[[str], int] = <function approx_token_count>, tokenizer: str = 'char-approx')

The default segmenter — recursive character splitting.

Splits a document on a descending ladder of separators (paragraph → line → sentence → clause → word → character), merging the resulting pieces into chunks of about chunk_size tokens with chunk_overlap tokens shared between neighbours. This “good enough” default is, on realistic corpora, within a point or two of far more elaborate methods — upgrade only if you can measure the win.

Sizes are measured with count_tokens. The default counter is the dependency-free approx_token_count(); inject a real tokenizer (and its tokenizer name) when an exact count matters. Every emitted segment records the tokenizer in its metadata and carries exact start / end character offsets into the source.

Parameters:

• chunk_size – Target chunk size, in count_tokens units.

• chunk_overlap – Tokens of overlap between consecutive chunks (must be smaller than chunk_size).

• separators – The separator ladder, coarsest first; the last should be "" so an unsplittable run is still broken between characters.

• count_tokens – Maps a string to a token count.

• tokenizer – Name recorded as the tokenizer metadata key.

>>> seg = RecursiveCharacterSegmenter(chunk_size=3, chunk_overlap=1)
>>> [c['text'] for c in seg('one two three four')]
['one two three', 'three four']

class ef.segmenters.Segmenter(*args, **kwargs)

Structural type for a segmenter: str | Mapping -> Iterable[Segment].

A segmenter is just a callable. It accepts a document — a bare str or a Mapping with a text key (and optionally an id, which becomes the emitted segments’ parent_id) — and yields Segment pieces. It returns an iterable, not a list: streaming is the default so the heavy case scales.

Because the protocol’s only member is __call__, every callable satisfies isinstance(x, Segmenter). Treat it as documentation of intent, not a discriminating check — as_segmenter() is the seam that turns an arbitrary callable into a well-behaved segmenter.

ef.segmenters.approx_token_count(text: str) → int

Rough token count — the common ~4 characters per token heuristic.

A dependency-free default so ef can talk in “tokens” without pulling in tiktoken or a model tokenizer. Inject a real counter (and its name) into a segmenter when an exact count matters.

>>> approx_token_count('a' * 40)
10
>>> approx_token_count('')
0

ef.segmenters.hierarchical(segmenters: Sequence[Any], /) → Segmenter

Compose segmenters into levels — each segments the previous level’s output.

Level 0 applies segmenters[0] to the document. Level k applies segmenters[k] to every level-(k-1) segment’s text, setting each child’s parent_id to that segment’s id and shifting its offsets to stay absolute. The result is a flat stream of every segment of every level (level 0 first), with the tree expressed purely through parent_id pointers.

Each entry of segmenters is passed through as_segmenter(), so strings (e.g. "recursive") and bare callables are accepted alongside ready-made segmenters.

ef.segmenters.line_segmenter(doc: str | Mapping[str, Any], /) → Iterator[Segment]

Segment a document into its non-empty lines.

Each line is stripped of surrounding whitespace; blank lines are dropped. Character offsets point at the stripped content within the source.

A plain function — and therefore already a Segmenter. It carries the marker as_segmenter() uses to recognize a ready-made segmenter.

>>> [s['text'] for s in line_segmenter('  hello  \n\nworld\n')]
['hello', 'world']

ef.segmenters.materialise(x: Any) → Any

Force a streaming segmenter (or segment stream) to a concrete list.

Streaming is the default, but the light case often wants everything in hand. materialise() is polymorphic:

• given a segmenter (a callable), it returns a new segmenter whose output is a list rather than a lazy iterator;

• given an iterable of segments, it returns a list of them.

>>> eager = materialise(line_segmenter)
>>> out = eager('a\nb')
>>> isinstance(out, list)
True
>>> [s['text'] for s in out]
['a', 'b']
>>> materialise(s for s in [{'text': 'x'}])
[{'text': 'x'}]

ef.segmenters.with_overlap(segmenter: ~ef.segmenters.Segmenter, n_chars: int, /, *, count_tokens: ~typing.Callable[[str], int] = <function approx_token_count>, tokenizer: str = 'char-approx') → Segmenter

Wrap segmenter so consecutive segments overlap by n_chars characters.

Each segment is extended forward by up to n_chars characters (clamped to the source length), so segment K shares its tail with segment K+1’s head. A segment whose text changes has its id re-derived and its tokens recounted; segments lacking start / end offsets pass through untouched.

Overlap is expressed in characters here (this is a generic post-hoc helper) — for token-measured overlap use a segmenter’s own chunk_overlap.

>>> base = RecursiveCharacterSegmenter(chunk_size=3, chunk_overlap=0)
>>> [c['text'] for c in base('one two three four')]
['one two three', 'four']
>>> [c['text'] for c in with_overlap(base, 100)('one two three four')]
['one two three four', 'four']