ef.segmenters

The Segmenter facade — ef’s segment layer (L2).

A segmenter turns a document into a stream of Segment pieces. Like the embedder facade, its core is a tiny structural contract:

Segmenter = Callable[[str | Mapping], Iterable[Segment]]

Everything else — overlap, hierarchy, batching, the recursive splitting algorithm — is layered scaffolding over that one shape.

This module defines:

• Segmenter — the @runtime_checkable protocol (a callable). It is a protocol, not an ABC: any matching callable is a segmenter.

• BatchedSegmenter — an optional optimization trait (a batch method) for segmenters that can process many documents at once.

• BaseSegmenter — an implementation-convenience base (a default batch() and a repr). Subclassing is optional.

• RecursiveCharacterSegmenter — the default segmenter: recursive character splitting at ~512 tokens with ~12.5% overlap. The honest default — within a point or two of fancier methods on realistic corpora.

• line_segmenter() — a builtin that splits a document into its non-empty lines (a plain function — proof that a bare callable is a segmenter).

• FunctionSegmenter — wraps a bare text -> pieces callable into a full segmenter; the bridge for imbed’s registered segmenters.

• Composition helpers — with_overlap(), hierarchical(), materialise().

Streaming is first-class: a segmenter returns an iterable, not a list, so the heavy case (a huge corpus) never has to fit every segment in memory at once.

The dependency-injection seam as_segmenter() and the imbed registry bridge live in ef.segmenter_adapters.

Example — split a document into ~4-token chunks with one token of overlap:

>>> seg = RecursiveCharacterSegmenter(chunk_size=4, chunk_overlap=1)
>>> [c['text'] for c in seg('alpha beta gamma delta epsilon')]
['alpha beta gamma delta', 'delta epsilon']
>>> isinstance(seg, Segmenter)
True

Module Attributes

APPROX_TOKENIZER	Identifier recorded as the tokenizer metadata key when token counts come from approx_token_count().
DEFAULT_SEPARATORS	Separators tried, in order, by RecursiveCharacterSegmenter.

Functions

line_segmenter(doc, /)	Segment a document into its non-empty lines.
with_overlap(segmenter, n_chars, /, *[, ...])	Wrap segmenter so consecutive segments overlap by n_chars characters.
hierarchical(segmenters, /)	Compose segmenters into levels — each segments the previous level's output.
materialise(x)	Force a streaming segmenter (or segment stream) to a concrete list.
approx_token_count(text)	Rough token count — the common ~4 characters per token heuristic.

Classes

Segmenter(*args, **kwargs)	Structural type for a segmenter: str | Mapping -> Iterable[Segment].
BatchedSegmenter(*args, **kwargs)	Optional trait: a segmenter that can process many documents at once.
BaseSegmenter()	Implementation base for segmenters (optional — Segmenter is structural).
FunctionSegmenter(func, *[, pass_doc, ...])	Promote a bare text -> pieces callable to a full Segmenter.
RecursiveCharacterSegmenter(*[, chunk_size, ...])	The default segmenter — recursive character splitting.

ef.segmenters.APPROX_TOKENIZER = 'char-approx'

Identifier recorded as the tokenizer metadata key when token counts come from approx_token_count(). Recording which tokenizer counted a segment is mandatory — a tokens number is meaningless without it.

class ef.segmenters.BaseSegmenter

Bases: object

Implementation base for segmenters (optional — Segmenter is structural).

A subclass implements __call__; in return it gets a default batch() (one-document-at-a-time — override it for a genuinely batched backend) and a readable repr. Subclassing is purely a convenience; satisfying Segmenter structurally is what matters.

batch(docs, /)

Default: segment each document independently.

Return type:

Iterator[Iterable[Segment]]

class ef.segmenters.BatchedSegmenter(*args, **kwargs)

Bases: Protocol

Optional trait: a segmenter that can process many documents at once.

Plain Segmenter handles one document per call. A segmenter that can amortize per-call cost across documents (a shared model load, a vectorized pass) also offers batch. BaseSegmenter supplies a correct — if un-optimized — default, so every BaseSegmenter subclass is already a BatchedSegmenter.

batch(docs, /)

Segment each document in docs, yielding one segment-stream each.

Return type:

Iterable[Iterable[Segment]]

ef.segmenters.DEFAULT_SEPARATORS: tuple[str, ...] = ('\n\n', '\n', '. ', ', ', ' ', '')

Separators tried, in order, by RecursiveCharacterSegmenter. Earlier (coarser) boundaries are preferred; the empty string is the last resort, splitting between characters so an oversized run is never left unsplit.

class ef.segmenters.FunctionSegmenter(func, *, pass_doc=False, count_tokens=<function approx_token_count>, tokenizer='char-approx')

Bases: BaseSegmenter

Promote a bare text -> pieces callable to a full Segmenter.

The bridge for segmenter functions that carry no segment schema — most importantly imbed’s registered segmenters, which map text to an iterable of plain strings. Each piece the callable yields is normalized:

• a str piece becomes a Segment; its character offsets are recovered by locating it in the source text, and a token count plus tokenizer metadata are attached;

• a Mapping / SegmentRecord piece is coerced with as_segment(), gaining only an index / parent_id

  where it lacks them.

Parameters:

• func (Callable[..., Any]) – The callable to wrap. By default it receives the document’s text; with pass_doc=True it receives the whole document.

• pass_doc (bool) – Pass the raw document instead of its text.

• count_tokens (Optional[Callable[[str], int]]) – Token counter for str pieces (None disables the tokens / tokenizer annotation).

• tokenizer (str) – Name recorded as the tokenizer metadata key.

>>> fs = FunctionSegmenter(lambda text: text.split(','))
>>> [s['text'] for s in fs('a,b,c')]
['a', 'b', 'c']
>>> s = list(fs('a,b,c'))[1]
>>> (s['start'], s['end'], s['index'])
(2, 3, 1)

class ef.segmenters.RecursiveCharacterSegmenter(*, chunk_size=512, chunk_overlap=64, separators=('\\n\\n', '\\n', '. ', ', ', ' ', ''), count_tokens=<function approx_token_count>, tokenizer='char-approx')

Bases: BaseSegmenter

The default segmenter — recursive character splitting.

Splits a document on a descending ladder of separators (paragraph → line → sentence → clause → word → character), merging the resulting pieces into chunks of about chunk_size tokens with chunk_overlap tokens shared between neighbours. This “good enough” default is, on realistic corpora, within a point or two of far more elaborate methods — upgrade only if you can measure the win.

Sizes are measured with count_tokens. The default counter is the dependency-free approx_token_count(); inject a real tokenizer (and its tokenizer name) when an exact count matters. Every emitted segment records the tokenizer in its metadata and carries exact start / end character offsets into the source.

Parameters:

• chunk_size (int) – Target chunk size, in count_tokens units.

• chunk_overlap (int) – Tokens of overlap between consecutive chunks (must be smaller than chunk_size).

• separators (Sequence[str]) – The separator ladder, coarsest first; the last should be "" so an unsplittable run is still broken between characters.

• count_tokens (Callable[[str], int]) – Maps a string to a token count.

• tokenizer (str) – Name recorded as the tokenizer metadata key.

>>> seg = RecursiveCharacterSegmenter(chunk_size=3, chunk_overlap=1)
>>> [c['text'] for c in seg('one two three four')]
['one two three', 'three four']

class ef.segmenters.Segmenter(*args, **kwargs)

Bases: Protocol

Structural type for a segmenter: str | Mapping -> Iterable[Segment].

A segmenter is just a callable. It accepts a document — a bare str or a Mapping with a text key (and optionally an id, which becomes the emitted segments’ parent_id) — and yields Segment pieces. It returns an iterable, not a list: streaming is the default so the heavy case scales.

Because the protocol’s only member is __call__, every callable satisfies isinstance(x, Segmenter). Treat it as documentation of intent, not a discriminating check — as_segmenter() is the seam that turns an arbitrary callable into a well-behaved segmenter.

ef.segmenters.approx_token_count(text)

Rough token count — the common ~4 characters per token heuristic.

A dependency-free default so ef can talk in “tokens” without pulling in tiktoken or a model tokenizer. Inject a real counter (and its name) into a segmenter when an exact count matters.

Return type:

int

>>> approx_token_count('a' * 40)
10
>>> approx_token_count('')
0

ef.segmenters.hierarchical(segmenters, /)

Compose segmenters into levels — each segments the previous level’s output.

Level 0 applies segmenters[0] to the document. Level k applies segmenters[k] to every level-(k-1) segment’s text, setting each child’s parent_id to that segment’s id and shifting its offsets to stay absolute. The result is a flat stream of every segment of every level (level 0 first), with the tree expressed purely through parent_id pointers.

Each entry of segmenters is passed through as_segmenter(), so strings (e.g. "recursive") and bare callables are accepted alongside ready-made segmenters.

Return type:

Segmenter

ef.segmenters.line_segmenter(doc, /)

Segment a document into its non-empty lines.

Each line is stripped of surrounding whitespace; blank lines are dropped. Character offsets point at the stripped content within the source.

A plain function — and therefore already a Segmenter. It carries the marker as_segmenter() uses to recognize a ready-made segmenter.

Return type:

Iterator[Segment]

>>> [s['text'] for s in line_segmenter('  hello  \n\nworld\n')]
['hello', 'world']

ef.segmenters.materialise(x)

Force a streaming segmenter (or segment stream) to a concrete list.

Streaming is the default, but the light case often wants everything in hand. materialise() is polymorphic:

• given a segmenter (a callable), it returns a new segmenter whose output is a list rather than a lazy iterator;

• given an iterable of segments, it returns a list of them.

>>> eager = materialise(line_segmenter)
>>> out = eager('a\nb')
>>> isinstance(out, list)
True
>>> [s['text'] for s in out]
['a', 'b']
>>> materialise(s for s in [{'text': 'x'}])
[{'text': 'x'}]

Return type:

Any

ef.segmenters.with_overlap(segmenter, n_chars, /, *, count_tokens=<function approx_token_count>, tokenizer='char-approx')

Wrap segmenter so consecutive segments overlap by n_chars characters.

Each segment is extended forward by up to n_chars characters (clamped to the source length), so segment K shares its tail with segment K+1’s head. A segment whose text changes has its id re-derived and its tokens recounted; segments lacking start / end offsets pass through untouched.

Overlap is expressed in characters here (this is a generic post-hoc helper) — for token-measured overlap use a segmenter’s own chunk_overlap.

Return type:

Segmenter

>>> base = RecursiveCharacterSegmenter(chunk_size=3, chunk_overlap=0)
>>> [c['text'] for c in base('one two three four')]
['one two three', 'four']
>>> [c['text'] for c in with_overlap(base, 100)('one two three four')]
['one two three four', 'four']