contaix.aggregation

Tools for aggregating contexts

Functions

aggregate_store(store, *[, ...])

Aggregates and processes text files from a store.
contaix.aggregation.aggregate_store(store, *, min_number_of_duplicated_lines=None, max_num_characters=None, exclude=None, chk_size=None, egress=None, **store_aggregate_kwargs)[source]

Aggregates and processes text files from a store.

Parameters:

  • store (Mapping) – The source store (e.g., TextFiles instance)

  • min_number_of_duplicated_lines (int | None) – If not None, minimum block size for deduplication.

  • max_num_characters (int | None) – Maximum number of characters per file.

  • exclude (Iterable | None) – Set of filenames to exclude.

  • chk_size (int) – Chunk size for aggregation. If None, won’t chunk

  • egress (str | Callable | None) – Template for output filenames or function to call on each chunk

  • **store_aggregate_kwargs – Additional keyword arguments to pass to store_aggregate.