ef.exploration

Layer L5 — explore a corpus: project to 2-D/3-D, cluster, label clusters.

This is ef’s visualization heritage, inherited from imbed and re-presented as a secondary surface. The primary surface of ef is search / RAG / corpus-indexing (see ef.ingest() and SourceManager); explore is the “see the shape of the corpus” tool — the backend an app_ef corpus map consumes.

Four free functions, each accepting a corpus or an already-computed vector matrix (whatever _resolve_vectors() understands):

  • project() — reduce embeddings to 2-D/3-D coordinates (PCA → UMAP).

  • cluster() — group embeddings into clusters (k-means / HDBSCAN).

  • label_clusters() — name clusters with an LLM (via imbed).

  • explore() — the orchestrator: project + cluster (+ optionally label) in one call, returning a structured, JSON-friendly ExploreResult (ids / coords / labels / cluster_titles) — the surface an app_ef corpus map (or ef.service.EfService.explore_corpus()) consumes directly.

The module imports with numpy only. UMAP, HDBSCAN and imbed are optional: each is imported lazily, inside the function that needs it, and only that heavy path requires the corresponding extra (ef[explore] / ef[imbed]). The numpy-only paths — PCA projection and k-means clustering — work out of the box with no install.

Example — project a small set of vectors to the plane (numpy only):

>>> import numpy as np
>>> vectors = np.random.RandomState(0).rand(12, 16)
>>> coords = project(vectors, dims=2, method='pca')
>>> coords.shape
(12, 2)
>>> labels = cluster(vectors, method='kmeans', n_clusters=3, random_state=0)
>>> labels.shape
(12,)

Each returned array’s rows are in the iteration order of the input — for a SearchableCorpus that is list(searchable.collection), which the caller zips back against to recover {id: coords}. explore() does that zip-back for you — its ExploreResult keeps ids aligned with coords and labels.

Functions

project(data, *[, dims, method, embedder, ...])

Project a corpus's embeddings to dims coordinates for visualization.

cluster(data, *[, method, n_clusters, ...])

Cluster a corpus's embeddings (use cases §G2).

label_clusters(segments, labels, *[, ...])

Name each cluster with a short LLM-generated title (use cases §G3).

explore(data, *[, dims, projection_method, ...])

Project, cluster and (optionally) label a corpus in one call.

Classes

ExploreResult

A structured, JSON-friendly corpus-exploration result.
class ef.exploration.ExploreResult[source]

Bases: TypedDict

A structured, JSON-friendly corpus-exploration result.

What explore() returns and ef.service.EfService.explore_corpus() serves. Every list is row-aligned: ids[i], coords[i] and labels[i] all describe the same item.

Keys:
ids: the per-item identifiers — a corpus’s keys, or positional "0",

"1", … for an id-less vector matrix.

coords: the projected coordinates — one [x, y] (or [x, y, z])

row per item.

labels: the cluster id of each item (HDBSCAN marks noise -1). cluster_titles: {cluster_id: title} — empty unless explore()

was called with label=True.

ef.exploration.cluster(data, *, method='kmeans', n_clusters=8, min_cluster_size=5, embedder=None, normalize=True, random_state=42)[source]

Cluster a corpus’s embeddings (use cases §G2).

method='kmeans' is the default and is dependency-free — a numpy Lloyd’s algorithm with k-means++ seeding. method='hdbscan' finds a density-based clustering (and a noise label -1) and needs the ef[explore] extra.

Parameters:

  • data (Any) – what to cluster — see project() / _resolve_vectors().

  • method (Literal['kmeans', 'hdbscan']) – 'kmeans' (numpy-only) or 'hdbscan' (optional extra).

  • n_clusters (int) – number of clusters for k-means (clamped to n_samples).

  • min_cluster_size (int) – smallest cluster HDBSCAN will form.

  • embedder (Any) – embedder used when data is raw text.

  • normalize (bool) – L2-normalize the vectors first, so Euclidean distance behaves like cosine similarity — the right default for semantic embeddings. Set False to cluster raw vectors.

  • random_state (int) – seed for k-means initialization.

Return type:

ndarray

Returns:

An ndarray of integer cluster labels, shape (n_samples,), in input order. HDBSCAN labels unclustered points -1.

>>> import numpy as np
>>> vectors = np.random.RandomState(2).rand(15, 8)
>>> labels = cluster(vectors, method='kmeans', n_clusters=3, random_state=0)
>>> labels.shape
(15,)
>>> set(labels.tolist()) <= {0, 1, 2}
True
ef.exploration.explore(data, *, dims=2, projection_method='auto', cluster_method='kmeans', n_clusters=8, min_cluster_size=5, label=False, context=' ', n_words=4, embedder=None, random_state=42)[source]

Project, cluster and (optionally) label a corpus in one call.

The layer-L5 orchestrator — project() + cluster() (+ optionally label_clusters()) wired into one structured result. Where project() / cluster() return bare arrays in input order, explore keeps every row tied to its id: it returns an ExploreResult with ids / coords / labels row-aligned (plus cluster_titles) — the JSON-friendly shape an app_ef corpus map (or ef.service.EfService.explore_corpus()) consumes directly.

Parameters:

  • data (Any) – the corpus to explore — anything _resolve_explorable() understands (a SourceManager / SearchableCorpus, a Corpus mapping, an iterable of texts or Segments, or a vector matrix).

  • dims (int) – projection target dimensionality — 2 or 3.

  • projection_method (Literal['auto', 'umap', 'pca']) – forwarded to project() — 'auto' / 'umap' / 'pca'.

  • cluster_method (Literal['kmeans', 'hdbscan']) – forwarded to cluster() — 'kmeans' / 'hdbscan'.

  • n_clusters (int) – number of k-means clusters.

  • min_cluster_size (int) – smallest cluster HDBSCAN will form.

  • label (bool) – when True, name each cluster with label_clusters() (needs the ef[imbed] extra, an LLM key, and text-bearing data); when False (the default) cluster_titles is empty.

  • context (str) – the corpus topic passed to label_clusters().

  • n_words (int) – maximum cluster-title length, in words.

  • embedder (Any) – embedder used when data is raw text.

  • random_state (int) – seed — projection and clustering are reproducible.

Return type:

ExploreResult

Returns:

an ExploreResult — ids, coords and labels row-aligned, and cluster_titles (empty unless label=True).

Raises:

ValueError – if the corpus is empty, has fewer than 2 samples, or label=True is asked of a text-less vector matrix.

>>> import numpy as np
>>> vectors = np.random.RandomState(0).rand(12, 16)
>>> result = explore(vectors, projection_method='pca', n_clusters=3, random_state=0)
>>> len(result['ids']), len(result['coords']), len(result['labels'])
(12, 12, 12)
>>> result['ids'][:3]
['0', '1', '2']
>>> len(result['coords'][0])                 # each row has `dims` coordinates
2
>>> set(result['labels']) <= {0, 1, 2}
True
>>> result['cluster_titles']                 # empty unless label=True
{}
ef.exploration.label_clusters(segments, labels, *, context=' ', n_words=4, n_samples=None, **labeler_kwargs)[source]

Name each cluster with a short LLM-generated title (use cases §G3).

A thin wrapper over imbed’s imbed.tools.ClusterLabeler: it samples each cluster’s segments, asks an LLM for a few-word title, and returns {cluster_id: title}. ef does not reimplement this — it reuses imbed (design notes §7).

Parameters:

  • segments (Iterable[Any]) – the segment texts — plain strings or Segment mappings (the "text" field is used). Aligned with labels.

  • labels (Iterable[Any]) – the cluster id of each segment — typically the output of cluster().

  • context (str) – the corpus’s overall topic, given to the LLM so titles describe how a cluster differs from the rest, not the shared topic.

  • n_words (int) – maximum title length, in words.

  • n_samples (int | None) – how many segments to sample per cluster when prompting; None keeps ClusterLabeler’s default.

  • **labeler_kwargs (Any) – forwarded to imbed.tools.ClusterLabeler (e.g. max_unique_clusters, prompt).

Return type:

dict[int, str]

Returns:

{cluster_id: title} for every distinct label.

Raises:

  • ValueError – if segments and labels differ in length.

  • ImportError – if imbed (and its pandas/oa deps) is missing.

>>> # needs ef[imbed] + an LLM key — illustrative only:
>>> # label_clusters(['neural net training ...', ...], [0, 1, 0, ...])
>>> # -> {0: 'Neural network training', 1: 'Dataset preprocessing'}
ef.exploration.project(data, *, dims=2, method='auto', embedder=None, n_neighbors=15, min_dist=0.1, metric='cosine', pca_components=50, random_state=42)[source]

Project a corpus’s embeddings to dims coordinates for visualization.

The canonical “show me the shape of the corpus” operation (use cases §G1). The default method='auto' runs the design-notes §8.4 recipe — **PCA → UMAP**, cosine metric, seeded: PCA first compresses high-dimensional embeddings to pca_components (cheap denoising), then UMAP lays them out in dims dimensions.

Parameters:

  • data (Any) – what to project — anything _resolve_vectors() understands: an ndarray (or nested sequence) of vectors, an iterable of texts (embedded with embedder), a Corpus mapping, or a SearchableCorpus / SourceManager (vectors pulled from the index).

  • dims (int) – target dimensionality — 2 or 3 for plotting.

  • method (Literal['auto', 'umap', 'pca']) – 'auto' (UMAP when available and the corpus is large enough, else PCA), 'umap' (force UMAP — needs the ef[explore] extra), or 'pca' (numpy-only, always available).

  • embedder (Any) – embedder used when data is raw text; defaults to the dependency-free DEFAULT_EMBEDDER.

  • n_neighbors (int) – UMAP neighborhood size (clamped to n_samples - 1).

  • min_dist (float) – UMAP minimum point separation in the embedding.

  • metric (str) – UMAP distance metric — 'cosine' for semantic embeddings.

  • pca_components (int) – PCA width for the pre-UMAP compression step; PCA is skipped when the source dimensionality is already smaller.

  • random_state (int) – seed — projection is reproducible.

Return type:

ndarray

Returns:

An ndarray of shape (n_samples, dims), rows in input order.

Raises:

  • ValueError – if there are fewer than 2 samples.

  • ImportError – if method='umap' and umap-learn is not installed.

>>> import numpy as np
>>> vectors = np.random.RandomState(1).rand(20, 32)
>>> project(vectors, dims=3, method='pca').shape
(20, 3)