hg.duplicates

Duplicate detection and handling — find and remove the largest repeated contiguous blocks in an ordered sequence.

Simple entry points (progressive disclosure — the common cases are one call):

• deduplicate_string_lines() — de-duplicate repeated line-blocks in text.

• deduplicate_sequence() — the same for any indexable sequence of items.

• BlockDeduplicator — the reusable, configurable form behind both.

A “block” is a maximal run of consecutive items that occurs more than once; the first occurrence is kept and later ones are removed.

>>> text = "A\nB\nA\nB\nC"
>>> final_text, removed = deduplicate_string_lines(text, min_block_size=2)
>>> print(final_text)
A
B
C
>>> removed
[RemovedBlock(removed_start=2, length=2, block_items=['A', 'B'])]

class hg.duplicates.BlockDeduplicator(*, min_block_size: int = 5, key: Callable | None = None)

A generic tool that finds repeated blocks in a sequence of items, then removes duplicate occurrences (retaining the first occurrence). It uses an initial block size for detection and extends the blocks to find the largest repeated sequences.

Example of usage:

>>> dedup = BlockDeduplicator(min_block_size=2)
>>> seq = [10, 20, 10, 20, 30]
>>> deduped, removed = dedup.deduplicate_sequence(seq)
>>> deduped
[10, 20, 30]
>>> removed
[RemovedBlock(removed_start=2, length=2, block_items=[10, 20])]

deduplicate_sequence(sequence: Sequence)

Detect largest duplicate blocks, then remove the second and subsequent occurrences of each block from ‘sequence’.

Parameters:

sequence – A list (or other indexable container) of items.

Returns:

(deduped_sequence, removed_blocks) - deduped_sequence: final list of items after removing duplicates - removed_blocks: list of RemovedBlock with the details of

each removed occurrence

>>> dedup = BlockDeduplicator(min_block_size=2)
>>> deduped, removed = dedup.deduplicate_sequence(
...     ["A", "B", "A", "B", "C"]
... )
>>> deduped
['A', 'B', 'C']
>>> removed
[RemovedBlock(removed_start=2, length=2, block_items=['A', 'B'])]

class hg.duplicates.RemovedBlock(removed_start: int, length: int, block_items: list)

A contiguous block that was removed as a duplicate.

• removed_start: index (in the original sequence) where the removed occurrence began.

• length: number of items in the block.

• block_items: the actual items of the block (taken from the first, retained, occurrence).

hg.duplicates.deduplicate_sequence(sequence: Sequence, *, min_block_size: int = 5, key: Callable | None = None)

Functional facade over BlockDeduplicator: remove the largest repeated contiguous blocks from sequence, keeping the first occurrence.

Returns (deduped_sequence, removed_blocks) where removed_blocks is a list of RemovedBlock.

>>> deduped, removed = deduplicate_sequence([1, 2, 1, 2, 3], min_block_size=2)
>>> deduped
[1, 2, 3]
>>> removed
[RemovedBlock(removed_start=2, length=2, block_items=[1, 2])]

hg.duplicates.deduplicate_string_lines(text: str, *, min_block_size: int = 5, key: Callable | None = <built-in function hash>, return_final_text: bool = True, return_removed_blocks: bool = True)

De-duplicate repeated line-blocks in a string by splitting it into lines, running deduplicate_sequence(), and re-joining.

Parameters:

• text – The input string.

• min_block_size – The size (in number of lines) for initial block match.

• key – Optional key function mapping each line to a comparable/hashable value. Defaults to hash() (lines are matched by hash, which is fine for text).

• return_final_text – Include the deduplicated text in the result.

• return_removed_blocks – Include the list of RemovedBlock.

Returns:

• both flags true (default): (final_text, removed_blocks)

• only return_final_text: final_text

• only return_removed_blocks: removed_blocks

• neither: None

>>> text = "A\nB\nC\nA\nB\nC\nD"
>>> final_text, removed = deduplicate_string_lines(text, min_block_size=3)
>>> print(final_text)
A
B
C
D
>>> removed
[RemovedBlock(removed_start=3, length=3, block_items=['A', 'B', 'C'])]
>>> deduplicate_string_lines(text, min_block_size=3, return_removed_blocks=False)
'A\nB\nC\nD'