contaix.aggregation

Tools for aggregating contexts

contaix.aggregation.aggregate_store(store: Mapping, *, min_number_of_duplicated_lines: int | None = None, max_num_characters: int | None = None, exclude: Iterable | None = None, chk_size=None, egress: str | Callable | None = None, **store_aggregate_kwargs)

Aggregates and processes text files from a store.

Parameters:

• store – The source store (e.g., TextFiles instance)

• min_number_of_duplicated_lines (int) – If not None, minimum block size for deduplication.

• max_num_characters (int) – Maximum number of characters per file.

• exclude (set) – Set of filenames to exclude.

• chk_size (int) – Chunk size for aggregation. If None, won’t chunk

• egress (str or callable) – Template for output filenames or function to call on each chunk

• **store_aggregate_kwargs – Additional keyword arguments to pass to store_aggregate.