> built 2026-09-22 14:31 UTC from 9ad945e (main) · hg 0.0.9. Details: build_info.json

# index.html.md

<!-- generated by epythet -->

# hg

Homogenous Groups — find items that recur together.

To install: `pip install hg`

`hg` bundles two complementary ways of discovering repeated structure in data,
with no third-party dependencies (pure standard library):

- **Duplication detection** — find and remove the largest repeated *contiguous*
  blocks in an ordered sequence (e.g. repeated line-blocks in text).
- **Frequent-itemset mining** — find sets of items that *co-occur* across
  transactions, optionally weighted by a per-transaction value.

## Duplication detection

The simplest case — de-duplicate repeated line-blocks in text, keeping the first
occurrence:

```python
>>> from hg import deduplicate_string_lines
>>> text = "A\nB\nC\nA\nB\nC\nD"
>>> final_text, removed = deduplicate_string_lines(text, min_block_size=3)
>>> print(final_text)
A
B
C
D
>>> removed
[RemovedBlock(removed_start=3, length=3, block_items=['A', 'B', 'C'])]

```

The same works on any sequence of items via `deduplicate_sequence` (or the
reusable `BlockDeduplicator`), with an optional `key` to control how items are
compared:

```python
>>> from hg import deduplicate_sequence
>>> deduped, removed = deduplicate_sequence([1, 2, 1, 2, 3], min_block_size=2)
>>> deduped
[1, 2, 3]

```

`min_block_size` is the smallest repeated run to detect; detected blocks are
then greedily extended to the largest repeated run.

## Frequent-itemset mining

Find sets of items that appear together in at least `minimum_support`
transactions:

```python
>>> from hg import find_frequent_itemsets
>>> transactions = [
...     ['bread', 'milk'],
...     ['bread', 'milk', 'eggs'],
...     ['milk', 'eggs'],
...     ['bread', 'butter'],
... ]
>>> for itemset in sorted(
...     find_frequent_itemsets(transactions, minimum_support=2),
...     key=lambda it: (-it.support, sorted(it.items)),
... ):
...     print(sorted(itemset.items), itemset.support)
['bread'] 3
['milk'] 3
['bread', 'milk'] 2
['eggs'] 2
['eggs', 'milk'] 2

```

### Value-weighted itemsets

The distinguishing feature: pass `transaction_values` to accumulate an arbitrary
per-transaction quantity (revenue, duration, …) alongside the count. Each
result is a `FrequentItemset(items, support, value)`:

```python
>>> prices = [4.0, 9.0, 5.0, 7.0]   # one value per transaction
>>> by_value = {
...     tuple(sorted(it.items)): it.value
...     for it in find_frequent_itemsets(
...         transactions, transaction_values=prices, minimum_support=2
...     )
... }
>>> by_value[('bread', 'milk')]     # baskets 0 (4.0) and 1 (9.0)
13.0

```

With no `transaction_values`, `value` simply equals `support`.

## When to use which

- Reach for **duplication detection** when *order matters* and you want to
  collapse repeated runs (deduping logs, transcripts, generated text).
- Reach for **frequent-itemset mining** when *co-occurrence* matters and order
  does not (market-basket analysis, tag/feature co-occurrence), especially when
  you want to weight occurrences by a value.

<p class="epythet-aggregates">This documentation as a single file: <a href="hg.md">hg.md</a> (Markdown, for agents).</p>


# _autosummary/hg.duplicates.html.md

# hg.duplicates

Duplicate detection and handling — find and remove the largest repeated
contiguous blocks in an ordered sequence.

Simple entry points (progressive disclosure — the common cases are one call):

- [`deduplicate_string_lines()`](_autosummary/hg.duplicates.html.md#hg.duplicates.deduplicate_string_lines) — de-duplicate repeated line-blocks in text.
- [`deduplicate_sequence()`](_autosummary/hg.duplicates.html.md#hg.duplicates.deduplicate_sequence) — the same for any indexable sequence of items.
- [`BlockDeduplicator`](_autosummary/hg.duplicates.html.md#hg.duplicates.BlockDeduplicator) — the reusable, configurable form behind both.

A “block” is a maximal run of consecutive items that occurs more than once; the
first occurrence is kept and later ones are removed.

```pycon
>>> text = "A\nB\nA\nB\nC"
>>> final_text, removed = deduplicate_string_lines(text, min_block_size=2)
>>> print(final_text)
A
B
C
>>> removed
[RemovedBlock(removed_start=2, length=2, block_items=['A', 'B'])]
```

### Functions

| [`deduplicate_sequence`](_autosummary/hg.duplicates.html.md#hg.duplicates.deduplicate_sequence)(sequence, \*[, ...])   | Functional facade over [`BlockDeduplicator`](_autosummary/hg.duplicates.html.md#hg.duplicates.BlockDeduplicator): remove the largest repeated contiguous blocks from `sequence`, keeping the first occurrence.   |
|----------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [`deduplicate_string_lines`](_autosummary/hg.duplicates.html.md#hg.duplicates.deduplicate_string_lines)(text, \*[, ...])   | De-duplicate repeated *line-blocks* in a string by splitting it into lines, running [`deduplicate_sequence()`](_autosummary/hg.duplicates.html.md#hg.duplicates.deduplicate_sequence), and re-joining.              |

### Classes

| [`BlockDeduplicator`](_autosummary/hg.duplicates.html.md#hg.duplicates.BlockDeduplicator)(\*[, min_block_size, key])     | A generic tool that finds repeated blocks in a sequence of items, then removes duplicate occurrences (retaining the first occurrence).   |
|---------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------|
| [`RemovedBlock`](_autosummary/hg.duplicates.html.md#hg.duplicates.RemovedBlock)(removed_start, length, block_items) | A contiguous block that was removed as a duplicate.                                                                                      |

### *class* hg.duplicates.BlockDeduplicator(, min_block_size=5, key=None)

Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)

A generic tool that finds repeated blocks in a sequence of items,
then removes duplicate occurrences (retaining the first occurrence).
It uses an initial block size for detection and extends the blocks
to find the largest repeated sequences.

Example of usage:

```pycon
>>> dedup = BlockDeduplicator(min_block_size=2)
>>> seq = [10, 20, 10, 20, 30]
>>> deduped, removed = dedup.deduplicate_sequence(seq)
>>> deduped
[10, 20, 30]
>>> removed
[RemovedBlock(removed_start=2, length=2, block_items=[10, 20])]
```

#### deduplicate_sequence(sequence)

Detect largest duplicate blocks, then remove the second
and subsequent occurrences of each block from ‘sequence’.

* **Parameters:**
  **sequence** ([`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)) – A list (or other indexable container) of items.
* **Returns:**
  `(deduped_sequence, removed_blocks)`
  - deduped_sequence: final list of items after removing duplicates
  - removed_blocks: list of [`RemovedBlock`](_autosummary/hg.duplicates.html.md#hg.duplicates.RemovedBlock) with the details of
    each removed occurrence

```pycon
>>> dedup = BlockDeduplicator(min_block_size=2)
>>> deduped, removed = dedup.deduplicate_sequence(
...     ["A", "B", "A", "B", "C"]
... )
>>> deduped
['A', 'B', 'C']
>>> removed
[RemovedBlock(removed_start=2, length=2, block_items=['A', 'B'])]
```

### *class* hg.duplicates.RemovedBlock(removed_start, length, block_items)

Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object)

A contiguous block that was removed as a duplicate.

- `removed_start`: index (in the original sequence) where the removed
  occurrence began.
- `length`: number of items in the block.
- `block_items`: the actual items of the block (taken from the first,
  retained, occurrence).

### hg.duplicates.deduplicate_sequence(sequence, , min_block_size=5, key=None)

Functional facade over [`BlockDeduplicator`](_autosummary/hg.duplicates.html.md#hg.duplicates.BlockDeduplicator): remove the largest
repeated contiguous blocks from `sequence`, keeping the first occurrence.

Returns `(deduped_sequence, removed_blocks)` where `removed_blocks` is a
list of [`RemovedBlock`](_autosummary/hg.duplicates.html.md#hg.duplicates.RemovedBlock).

```pycon
>>> deduped, removed = deduplicate_sequence([1, 2, 1, 2, 3], min_block_size=2)
>>> deduped
[1, 2, 3]
>>> removed
[RemovedBlock(removed_start=2, length=2, block_items=[1, 2])]
```

### hg.duplicates.deduplicate_string_lines(text, \*, min_block_size=5, key=<built-in function hash>, return_final_text=True, return_removed_blocks=True)

De-duplicate repeated *line-blocks* in a string by splitting it into lines,
running [`deduplicate_sequence()`](_autosummary/hg.duplicates.html.md#hg.duplicates.deduplicate_sequence), and re-joining.

* **Parameters:**
  * **text** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The input string.
  * **min_block_size** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – The size (in number of lines) for initial block match.
  * **key** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Optional key function mapping each line to a
    comparable/hashable value. Defaults to [`hash()`](https://docs.python.org/3/builtins/functions.html#hash)
    (lines are matched by hash, which is fine for text).
  * **return_final_text** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Include the deduplicated text in the result.
  * **return_removed_blocks** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Include the list of [`RemovedBlock`](_autosummary/hg.duplicates.html.md#hg.duplicates.RemovedBlock).
* **Returns:**
  - both flags true (default): `(final_text, removed_blocks)`
  - only `return_final_text`: `final_text`
  - only `return_removed_blocks`: `removed_blocks`
  - neither: `None`

```pycon
>>> text = "A\nB\nC\nA\nB\nC\nD"
>>> final_text, removed = deduplicate_string_lines(text, min_block_size=3)
>>> print(final_text)
A
B
C
D
>>> removed
[RemovedBlock(removed_start=3, length=3, block_items=['A', 'B', 'C'])]
>>> deduplicate_string_lines(text, min_block_size=3, return_removed_blocks=False)
'A\nB\nC\nD'
```


# _autosummary/hg.frequent_itemsets.html.md

# hg.frequent_itemsets

Frequent-itemset mining, augmented so each transaction can carry a numeric
*value* that is accumulated alongside the item *count* (support).

This lets you mine itemsets weighted by an arbitrary per-transaction quantity
(revenue, duration, …) in addition to plain co-occurrence support. Pass no
`transaction_values` to get ordinary (count-only) frequent-itemset mining.

The implementation is [Eclat](https://en.wikipedia.org/wiki/Association_rule_learning#Eclat_algorithm) (depth-first search over item *tidlists* — the
sets of transaction indices an item appears in), which is both correct and
simple: an itemset’s support is the size of the intersection of its items’
tidlists, and its accumulated value is the sum of those transactions’ values.

(Earlier versions of this module vendored an FP-growth fork whose conditional
tree was subtly wrong; this Eclat implementation replaces it and is verified
against a brute-force reference.)

### Functions

| [`find_frequent_itemsets`](_autosummary/hg.frequent_itemsets.html.md#hg.frequent_itemsets.find_frequent_itemsets)(transactions, \*[, ...])   | Find frequent itemsets in `transactions`, yielded as [`FrequentItemset`](_autosummary/hg.frequent_itemsets.html.md#hg.frequent_itemsets.FrequentItemset) (`(items, support, value)`).   |
|----------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------|

### Classes

| [`FrequentItemset`](_autosummary/hg.frequent_itemsets.html.md#hg.frequent_itemsets.FrequentItemset)(items, support, value)   | A frequent itemset result.   |
|-------------------------------------------------------------------------------------------|------------------------------|

### *class* hg.frequent_itemsets.FrequentItemset(items, support, value)

Bases: [`NamedTuple`](https://docs.python.org/3/library/typing.html#typing.NamedTuple)

A frequent itemset result.

Tuple-unpackable as `(items, support, value)`:

- `items`: the items making up the itemset (a list).
- `support`: number of transactions containing the itemset (the count).
- `value`: the accumulated per-transaction value over those transactions
  (equals `support` when no `transaction_values` were supplied).

#### items *: [list](https://docs.python.org/3/builtins/stdtypes.html#list)*

Alias for field number 0

#### support *: [int](https://docs.python.org/3/builtins/functions.html#int)*

Alias for field number 1

#### value *: [float](https://docs.python.org/3/builtins/functions.html#float)*

Alias for field number 2

### hg.frequent_itemsets.find_frequent_itemsets(transactions, , transaction_values=None, minimum_support=2)

Find frequent itemsets in `transactions`, yielded as
[`FrequentItemset`](_autosummary/hg.frequent_itemsets.html.md#hg.frequent_itemsets.FrequentItemset) (`(items, support, value)`).

`transactions` is any iterable of iterables of hashable items.
`transaction_values` is an optional parallel sequence of numeric weights
(one per transaction); when omitted, every transaction has weight 1 so
`value` equals `support` (ordinary frequent-itemset mining).
`minimum_support` is the minimum number of occurrences for an itemset to
be reported.

```pycon
>>> transactions = [
...     ['bread', 'milk'],
...     ['bread', 'milk', 'eggs'],
...     ['milk', 'eggs'],
...     ['bread', 'butter'],
... ]
>>> for itemset in sorted(
...     find_frequent_itemsets(transactions, minimum_support=2),
...     key=lambda it: (-it.support, sorted(it.items)),
... ):
...     print(sorted(itemset.items), itemset.support)
['bread'] 3
['milk'] 3
['bread', 'milk'] 2
['eggs'] 2
['eggs', 'milk'] 2
```

Pass `transaction_values` to weight each transaction (here by basket
price); the third field accumulates that weight over the matching
transactions:

```pycon
>>> prices = [4.0, 9.0, 5.0, 7.0]
>>> by_value = {
...     tuple(sorted(it.items)): it.value
...     for it in find_frequent_itemsets(
...         transactions, transaction_values=prices, minimum_support=2
...     )
... }
>>> by_value[('bread', 'milk')]  # baskets 0 (4.0) and 1 (9.0)
13.0
```


# _autosummary/hg.html.md

# hg

Homogenous Groups — discovering items that recur together.

Two complementary capabilities:

- **Duplication detection** — find and remove the largest repeated contiguous
  blocks in an ordered sequence (e.g. de-duplicate repeated line-blocks in text)
  via `deduplicate_string_lines()`, `deduplicate_sequence()`, and
  `BlockDeduplicator`.
- **Frequent-itemset mining** — find sets of items that co-occur across
  transactions, optionally weighted by a per-transaction value, via
  `find_frequent_itemsets()`.

### Modules

| [`duplicates`](_autosummary/hg.duplicates.html.md#module-hg.duplicates)               | Duplicate detection and handling — find and remove the largest repeated contiguous blocks in an ordered sequence.                            |
|------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------|
| [`frequent_itemsets`](_autosummary/hg.frequent_itemsets.html.md#module-hg.frequent_itemsets) | Frequent-itemset mining, augmented so each transaction can carry a numeric *value* that is accumulated alongside the item *count* (support). |


# about-this-build.html.md

<!-- generated by epythet -->

# About this build

This documentation was built on **2026-09-22 14:31 UTC** from commit <a href="https://github.com/thorwhalen/hg/commit/9ad945ebe3cd2b0097a4271e07880c11ae6830e7"><code>9ad945e</code></a> on branch <code>main</code>, for **hg 0.0.9** (from <code>pyproject.toml</code>).

#### WARNING
The documentation and the package may be misaligned:

- The documented version (0.0.9) is behind the latest release on PyPI (0.0.10): `pip install hg` gives newer code than these docs describe.

## Source

|                     |                                                                                                                                                      |
|---------------------|------------------------------------------------------------------------------------------------------------------------------------------------------|
| Commit              | <a href="https://github.com/thorwhalen/hg/commit/9ad945ebe3cd2b0097a4271e07880c11ae6830e7"><code>9ad945ebe3cd2b0097a4271e07880c11ae6830e7</code></a> |
| Branch              | <code>main</code>                                                                                                                                    |
| Tags at this commit | none                                                                                                                                                 |
| Working tree        | clean                                                                                                                                                |
| Remote              | <code>https://github.com/thorwhalen/hg</code>                                                                                                        |

## Continuous integration

|              |                                                                                            |
|--------------|--------------------------------------------------------------------------------------------|
| Repository   | <code>thorwhalen/hg</code>                                                                 |
| Run          | <a href="https://github.com/thorwhalen/hg/actions/runs/35740698358">35740698358</a>        |
| Ref          | <code>refs/heads/main</code>                                                               |
| Event commit | <code>9ad945ebe3cd2b0097a4271e07880c11ae6830e7</code> (in the history of the built commit) |

## Tools

|          |         |
|----------|---------|
| epythet  | 0.2.12  |
| Sphinx   | 9.1.0   |
| docutils | 0.22.4  |
| Python   | 3.12.14 |

## Configuration as resolved

|               |                                                                  |
|---------------|------------------------------------------------------------------|
| theme         | <code>auto</code> (Sphinx theme <code>shibuya</code>)            |
| accent        | <code>#006e59</code>                                             |
| api_generator | <code>autosummary</code>                                         |
| ignore        | <code>tests/</code>, <code>scrap/</code>, <code>examples/</code> |
| agent_outputs | <code>true</code>                                                |
| aggregates    | <code>md</code>                                                  |
| ai_artifacts  | <code>true</code>                                                |

## Package on PyPI

Latest release: <a href="https://pypi.org/project/hg/0.0.10/">0.0.10</a>, newer than the documented version (0.0.9).

## Reproduce

```bash
git clone https://github.com/thorwhalen/hg && cd hg
git checkout 9ad945ebe3cd2b0097a4271e07880c11ae6830e7
pip install "epythet==0.2.12"
epythet quickstart . --ignore tests/ scrap/ examples/
```

The same data, for machines: <a href="build_info.json"><code>build_info.json</code></a> (schema version 1).


# ai-agents.html.md

<!-- generated by epythet -->

# For AI agents

`hg` ships artifacts for coding agents alongside its code. This page lists
them, says where each lives in the repository, and points at the
machine-readable copies of this documentation.

## Skills

Skills are folders holding a `SKILL.md` (the [Agent Skills](https://agentskills.io) format): a description that tells an agent when to use it and a body with the procedure. Install one into your agent with `gh skill` (any host: `--agent claude-code`, `copilot`, `cursor`, `codex`, `gemini`), or use the copy bundled in the wheel.

### `hg-dev`

Use when developing or modifying the `hg` package itself — changing the public API, touching duplication detection (`hg/duplicates.py`) or frequent-itemset mining (`hg/frequent_itemsets.py`), adding algorithms, or working on its tests. Triggers on editing files under the hg repo, “add X to hg”, “fix the dedup/itemset code”, or changing what `hg/__init__.py` exports.

Source: [`.claude/skills/hg-dev`](https://github.com/thorwhalen/hg/tree/HEAD/.claude/skills/hg-dev).

### `hg-usage`

Use when you need to (a) de-duplicate repeated contiguous blocks in a sequence or text, or (b) mine frequent — optionally value-weighted — itemsets from transactions. Triggers on “remove repeated blocks/lines”, “dedupe a transcript/log/generated text”, “find the largest repeated run”, “frequent itemsets”, “market-basket / co-occurrence analysis”, “which items appear together”, or weighting co-occurrences by revenue/duration/etc. The package is `hg` (pure stdlib, `pip install hg`).

Source: [`.claude/skills/hg-usage`](https://github.com/thorwhalen/hg/tree/HEAD/.claude/skills/hg-usage).

## Instruction files

Files agents read before working in this repository.

- [`.claude/CLAUDE.md`](https://github.com/thorwhalen/hg/tree/HEAD/.claude/CLAUDE.md): read by Claude Code

## Machine-readable documentation

This site publishes the same documentation in forms that fit an agent’s context window:

- [`llms.txt`](https://thorwhalen.github.io/hg/llms.txt): an index of every page with a one-line description ([llms.txt](https://llmstxt.org) format)
- [`hg.md`](https://thorwhalen.github.io/hg/hg.md): the whole documentation as one Markdown file
- `<page>.html.md`: a rendered Markdown twin of every page, advertised from each page’s `<head>` with `<link rel="alternate" type="text/markdown">`
- [`objects.inv`](https://thorwhalen.github.io/hg/objects.inv): the Sphinx inventory: a symbol-to-URL index (`sphobjinv convert plain objects.inv -`)


# api.html.md

# API reference

| [`hg`](_autosummary/hg.html.md#module-hg)   | Homogenous Groups — discovering items that recur together.   |
|-----------------------------------------------------------------|--------------------------------------------------------------|