> built 2026-09-22 13:47 UTC from 8e7afad (main) · contaix 0.0.19. Details: build_info.json # index.html.md # contaix Tools to make contexts (knowledge bases) for AI. contaix turns diverse sources — code repos, documentation sites, file collections — into clean markdown contexts for AI agents to consume. ```bash pip install contaix ``` ## Skills This package ships agent skills you can install into any agent host with [`gh skill`](https://cli.github.com/manual/gh_skill) (don’t have it? [install gh](https://cli.github.com/)): ```bash gh skill install thorwhalen/contaix contaix-acquire-docs --agent claude-code gh skill install thorwhalen/contaix contaix_docs_to_markdown gh skill install thorwhalen/contaix contaix_web_aggregate ``` | Skill | Use it when… | |----------------------------|--------------------------------------------------------------------------------------------------------------------| | `contaix-acquire-docs` | acquiring a tool’s docs (plus OpenAPI/MCP/GraphQL specs) as one agent-ready markdown bundle, cheapest source first | | `contaix_docs_to_markdown` | converting a documentation website into a single clean markdown file | | `contaix_web_aggregate` | aggregating arbitrary (non-docs) web pages into structured markdown, with manual control | ## Quick Start ### Documentation site → single markdown file ```python from contaix.web import site_to_markdown site_to_markdown( 'https://fal.ai/docs/documentation', output_file='~/Downloads/fal_ai_docs.md', verbose=True, ) ``` `site_to_markdown` first probes the site for a publisher-provided `/llms-full.txt` bundle (Mintlify, Docusaurus, Fern, and others ship one). If found, it returns that directly — one HTTP request, clean markdown, no scraping. Otherwise it falls back to discovering navigation, fetching every page (with caching), and aggregating. To bypass the fast path or grab the bundle yourself: ```python from contaix.web import find_llms_full_url, fetch_llms_full bundle_url = find_llms_full_url('https://fal.ai/docs') # str or None markdown = fetch_llms_full('https://fal.ai/docs') # str or None ``` ### Code base → single markdown file ```python from contaix import code_aggregate # A local package md = code_aggregate('contaix') # A directory md = code_aggregate('/path/to/project') # A GitHub repo (downloaded via hubcap) md = code_aggregate('https://github.com/thorwhalen/contaix') # Save instead of return code_aggregate('contaix', egress='/tmp/contaix_code.md') ``` ### Files → markdown ```python from contaix import bytes_to_markdown, bytes_store_to_markdown_store from dol import Files # Single file (auto-detects format) md = bytes_to_markdown(open('report.pdf', 'rb').read(), input_format='pdf') # A whole folder src = Files('/path/to/documents') target = {} bytes_store_to_markdown_store(src, target) ``` Markdown conversion is delegated to [`dn.src`](https://github.com/thorwhalen/dn) and supports PDF, DOCX, XLSX, PPTX, HTML, IPYNB, and more. ### Cleanup of messy markdown (e.g. ChatGPT exports) ```python from contaix import remove_hyperlink_crap, remove_improperly_double_newlines # Reads from clipboard if no arg, copies result back clean = remove_hyperlink_crap() ``` Pure transforms live in [`dn.repair`](https://github.com/thorwhalen/dn); the contaix wrappers add clipboard integration. ### URLs in prose → article PDFs ```python from contaix import extract_urls, download_articles, verify_urls # Inspect what URLs are in a string list(extract_urls("[OpenAI](https://openai.com) and https://anthropic.com")) # -> [('OpenAI', 'https://openai.com'), ('https://anthropic.com', 'https://anthropic.com')] # Verify they resolve verify_urls(['https://openai.com', 'https://anthropic.com']) # -> {'https://openai.com': 200, 'https://anthropic.com': 200} # Download as PDFs (delegates to pdfdol) download_articles(some_md_string, save_dir='~/Downloads/articles') ``` ## Package Map ```default contaix/ web.py # site_to_markdown, find_llms_full_url, RSC extraction code.py # code_aggregate, PackageCodeContexts, GitHub fetch markdown.py # bytes_to_markdown (delegates to dn.src) urls.py # extract_urls, verify_urls, download_articles aggregation.py # aggregate_store with deduplication and chunking util.py # markdown_of_site, scrape utilities, clipboard helpers skills.py # discover/install OTHER ecosystem packages' skills ``` contaix’s own agent skills live at the repo root in `skills/` (gh-distributed — see the Skills section above), symlinked into `.claude/skills/`. ## Architecture - Functions are the primary interface; classes are reserved for stateful workflows (e.g. `PackageCodeContexts`). - `dol` store abstractions for file/data access throughout. - Smart defaults with keyword-only configurability — out of the box for common cases, fully parameterizable when needed. - Pure transforms live in domain packages (`dn`, `pdfdol`); contaix is the orchestration and UX layer. ## Optional dependencies - `pdfdol` — for `download_articles` and PDF round-trips (`pip install contaix[pdf]`) - `dn[all]` — for the full set of file-format converters (`pip install contaix[all]`) ## Related packages - [`dn`](https://github.com/thorwhalen/dn) — pure markdown conversion and repair - [`pdfdol`](https://github.com/thorwhalen/pdfdol) — PDF reading, writing, and format conversion - [`dol`](https://github.com/i2mint/dol) — store abstractions used everywhere - [`scraped`](https://github.com/thorwhalen/scraped) — scraping primitives - [`hubcap`](https://github.com/thorwhalen/hubcap) — GitHub repo access

This documentation as a single file: contaix.md (Markdown, for agents).

# _autosummary/contaix.aggregation.html.md # contaix.aggregation Tools for aggregating contexts ### Functions | [`aggregate_store`](_autosummary/contaix.aggregation.html.md#contaix.aggregation.aggregate_store)(store, \*[, ...]) | Aggregates and processes text files from a store. | |--------------------------------------------------------------------------------------|-----------------------------------------------------| ### contaix.aggregation.aggregate_store(store, , min_number_of_duplicated_lines=None, max_num_characters=None, exclude=None, chk_size=None, egress=None, \*\*store_aggregate_kwargs) Aggregates and processes text files from a store. * **Parameters:** * **store** ([`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)) – The source store (e.g., TextFiles instance) * **min_number_of_duplicated_lines** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – If not None, minimum block size for deduplication. * **max_num_characters** ([`int`](https://docs.python.org/3/builtins/functions.html#int) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Maximum number of characters per file. * **exclude** ([`Iterable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterable) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Set of filenames to exclude. * **chk_size** ([*int*](https://docs.python.org/3/builtins/functions.html#int)) – Chunk size for aggregation. If None, won’t chunk * **egress** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – Template for output filenames or function to call on each chunk * **\*\*store_aggregate_kwargs** – Additional keyword arguments to pass to store_aggregate. # _autosummary/contaix.code.html.md # contaix.code Tools to make AI contexts from code bases ### Module Attributes | [`DFLT_AGGREGATE_EXCLUDE_PATTERNS`](_autosummary/contaix.code.html.md#contaix.code.DFLT_AGGREGATE_EXCLUDE_PATTERNS) | Default regex patterns matching section paths that are (almost) never useful as AI context: vendored deps, VCS internals, build/cache outputs, lockfiles, and minified bundles/sourcemaps. | |------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ### Functions | [`code_aggregate`](_autosummary/contaix.code.html.md#contaix.code.code_aggregate)(code_src, \*[, egress, ...]) | Aggregates all code segments from the given code source (folder, github url, store). | |--------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------| | [`is_local_pkg_name`](_autosummary/contaix.code.html.md#contaix.code.is_local_pkg_name)(name) | Returns True if and only if name is the name of a local package. | | [`iter_pruned_aggregate_lines`](_autosummary/contaix.code.html.md#contaix.code.iter_pruned_aggregate_lines)(lines, \*[, ...]) | Stream `lines` of an aggregated-code markdown, yielding only the lines that belong to sections whose header path does *not* match `exclude`. | | [`prune_code_aggregate`](_autosummary/contaix.code.html.md#contaix.code.prune_code_aggregate)(src, \*[, exclude, ...]) | Remove bloated sections (vendored deps, build output, lockfiles, minified bundles, ...) from an aggregated-code markdown. | | [`resolve_code_source`](_autosummary/contaix.code.html.md#contaix.code.resolve_code_source)(code_src[, keys_filt, ...]) | Will resolve code_src to a Mapping whose values are the code strings | | [`resolve_code_source_dir_path`](_autosummary/contaix.code.html.md#contaix.code.resolve_code_source_dir_path)(code_src) | Resolves code_src to a directory path string. | ### Classes | [`PackageCodeContexts`](_autosummary/contaix.code.html.md#contaix.code.PackageCodeContexts)([save_folder]) | Manages aggregation and saves of the code of local packages | |---------------------------------------------------------------------------------------|---------------------------------------------------------------| ### contaix.code.DFLT_AGGREGATE_EXCLUDE_PATTERNS *= ('(^|/)node_modules/', '(^|/)\\\\.git/', '(^|/)bower_components/', '(^|/)(dist|build|out|target|\\\\.next|\\\\.nuxt|\\\\.turbo|\\\\.cache|\\\\.parcel-cache|coverage|_\_pycache_\_|\\\\.pytest_cache|\\\\.mypy_cache|\\\\.tox|\\\\.venv|venv|site-packages|vendor)/', '(^|/)(package-lock\\\\.json|yarn\\\\.lock|pnpm-lock\\\\.yaml|npm-shrinkwrap\\\\.json|poetry\\\\.lock|Pipfile\\\\.lock|Cargo\\\\.lock|composer\\\\.lock|Gemfile\\\\.lock|go\\\\.sum)$', '\\\\.min\\\\.(js|mjs|cjs|css)$', '\\\\.(js|mjs|cjs|css)\\\\.map$', '\\\\.map$')* Default regex patterns matching section paths that are (almost) never useful as AI context: vendored deps, VCS internals, build/cache outputs, lockfiles, and minified bundles/sourcemaps. Patterns are matched (`re.search`) against each section’s path. ### *class* contaix.code.PackageCodeContexts(save_folder='.') Bases: [`object`](https://docs.python.org/3/builtins/functions.html#object) Manages aggregation and saves of the code of local packages #### multiple_pkgs_code(name=None, pkgs=(), , pkg_secion_marker='#', section_sepator='\\\\n\\\\n\\\\n') Aggregates the code of multiple local packages in separate sections. Saves the result to a file if a name is provided. #### save_multiple_pkgs_code(name=None, pkgs=(), , pkg_secion_marker='#', section_sepator='\\\\n\\\\n\\\\n') Aggregates the code of multiple local packages in separate sections. Saves the result to a file if a name is provided. #### save_single(pkg) Aggregates and saves the code of a single local package. ### Example To save a single package’s code in a single file, in the current folder: ```pycon >>> PackageCodeContexts().save_single('aix') ``` or, to save multiple package’s code in a single file, in a specific folder. ```pycon >>> PackageCodeContexts('some/folder/path').save_multiple_pkgs_code(['aix', 'dol']) ``` ### contaix.code.code_aggregate(code_src, \*, egress=, kv_to_item=>, keys_filt='\\\\\\\\.py$', keys_exclude=None, include_readme=, \*\*store_aggregate_kwargs) Aggregates all code segments from the given code source (folder, github url, store). This is useful when you want to use AI to search and respond to questions about a specific code base. * **Parameters:** * **code_src** (`Union`[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Mapping`](https://docs.python.org/3/library/typing.html#typing.Mapping)]) – A dictionary where keys are references to the code (e.g., paths) and values are code snippets or content. * **egress** (`Union`[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – A function to apply to the aggregate before returning. If a string, the aggregate will be saved to the file. * **kv_to_item** (*Callable*) – A function that converts a key-value pairs to the items that should be aggregated. * **keys_filt** (`Union`[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – A function or regex string to filter the keys of the code store. If a string, it will be compiled to a regex pattern. * **keys_exclude** – Optional regex string, iterable of regex strings, or `path->bool` predicate identifying keys to *exclude* from the aggregate (applied after `keys_filt`). Use `DFLT_AGGREGATE_EXCLUDE_PATTERNS` to skip vendored deps, build outputs, lockfiles and minified bundles. * **include_readme** ([`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)) – A function that takes the code_store as input and returns the content of a README file to include in the aggregate. If None, no README will be included. * **\*\*store_aggregate_kwargs** – Additional keyword arguments to pass to store_aggregate. * **Return type:** [*Any*](https://docs.python.org/3/library/typing.html#typing.Any) See dol.store_aggregate for more details. * **Returns:** The aggregated code content, or the result of the egress function. * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) ### Example ```pycon >>> code_src = { ... 'module1.py': 'def foo(): pass', ... 'module2.py': 'def bar(): pass', ... 'module3.py': 'class Baz: pass', ... } >>> print(code_aggregate(code_src)) ## module1.py ``` ```python def foo(): pass ``` ## module2.py ```python def bar(): pass ``` ## module3.py ```python class Baz: pass ``` Here, let’s input an imported (third party) package, and have the function save the result to a temporary file ```pycon >>> from tempfile import NamedTemporaryFile >>> temp_file_name = NamedTemporaryFile().name >>> import aix >>> _ = code_aggregate(aix, egress=temp_file_name) >>> print(open(temp_file_name).read(15)) ## __init__.py ``` #### TIP You can also import directly from the name (string) of the package by doing `code_aggregate(__import__('aix'))` or more robustly, `importlib.import_module('aix')`. If you have hubcap installed, you can even get an aggregate of code from a GitHub repository. ```pycon >>> string = code_aggregate('https://github.com/thorwhalen/aix') ``` ### contaix.code.is_local_pkg_name(name) Returns True if and only if name is the name of a local package. * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### contaix.code.iter_pruned_aggregate_lines(lines, , exclude=('(^|/)node_modules/', '(^|/)\\\\\\\\.git/', '(^|/)bower_components/', '(^|/)(dist|build|out|target|\\\\\\\\.next|\\\\\\\\.nuxt|\\\\\\\\.turbo|\\\\\\\\.cache|\\\\\\\\.parcel-cache|coverage|_\_pycache_\_|\\\\\\\\.pytest_cache|\\\\\\\\.mypy_cache|\\\\\\\\.tox|\\\\\\\\.venv|venv|site-packages|vendor)/', '(^|/)(package-lock\\\\\\\\.json|yarn\\\\\\\\.lock|pnpm-lock\\\\\\\\.yaml|npm-shrinkwrap\\\\\\\\.json|poetry\\\\\\\\.lock|Pipfile\\\\\\\\.lock|Cargo\\\\\\\\.lock|composer\\\\\\\\.lock|Gemfile\\\\\\\\.lock|go\\\\\\\\.sum)$', '\\\\\\\\.min\\\\\\\\.(js|mjs|cjs|css)$', '\\\\\\\\.(js|mjs|cjs|css)\\\\\\\\.map$', '\\\\\\\\.map$'), header_prefix='## ', fence='\`\`\`', on_drop=None) Stream `lines` of an aggregated-code markdown, yielding only the lines that belong to sections whose header path does *not* match `exclude`. A section header is detected *structurally*: a `header_prefix` line whose remainder looks like a file path **and** whose next non-blank line opens a code fence. This is deliberately not a simple “toggle on every fence” scheme – file *content* in these aggregates routinely contains stray `` lines (vendored ``.md`/`.ts`/`.json`), which corrupts fence-toggle state and lets bloat slip through. Requiring the header-then-fence signature makes section boundaries robust to that. Because it consumes and produces a line iterable, it works equally on a small in-memory string (via `str.splitlines(keepends=True)`) and on a multi-gigabyte file (by iterating the open file object), never holding more than a handful of lines. * **Parameters:** * **lines** ([`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – Iterable of lines. Whatever line endings are present are preserved. * **exclude** – Regex string, iterable of regex strings, or `path->bool` predicate selecting sections to drop. Defaults to `DFLT_AGGREGATE_EXCLUDE_PATTERNS`. * **header_prefix** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Prefix marking a section header; the remainder is the path. * **fence** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Code-fence marker that opens a section’s content block. * **on_drop** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]]) – Optional callback invoked with each dropped section’s path (useful for logging/counting what was removed). * **Yields:** The lines to keep, in original order. * **Return type:** [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] ### Example ```pycon >>> md = '''## a.py ... ... ```python ... x = 1 # a heading-looking line: ## not a header ... ``` ... ... ## node_modules/dep/index.js ... ... ```python ... huge minified bundle ... ``` ... ''' >>> kept = ''.join(iter_pruned_aggregate_lines(md.splitlines(keepends=True))) >>> '## a.py' in kept and 'x = 1' in kept True >>> 'node_modules' in kept False ``` ### contaix.code.prune_code_aggregate(src, \*, exclude=('(^|/)node_modules/', '(^|/)\\\\\\\\.git/', '(^|/)bower_components/', '(^|/)(dist|build|out|target|\\\\\\\\.next|\\\\\\\\.nuxt|\\\\\\\\.turbo|\\\\\\\\.cache|\\\\\\\\.parcel-cache|coverage|_\_pycache_\_|\\\\\\\\.pytest_cache|\\\\\\\\.mypy_cache|\\\\\\\\.tox|\\\\\\\\.venv|venv|site-packages|vendor)/', '(^|/)(package-lock\\\\\\\\.json|yarn\\\\\\\\.lock|pnpm-lock\\\\\\\\.yaml|npm-shrinkwrap\\\\\\\\.json|poetry\\\\\\\\.lock|Pipfile\\\\\\\\.lock|Cargo\\\\\\\\.lock|composer\\\\\\\\.lock|Gemfile\\\\\\\\.lock|go\\\\\\\\.sum)$', '\\\\\\\\.min\\\\\\\\.(js|mjs|cjs|css)$', '\\\\\\\\.(js|mjs|cjs|css)\\\\\\\\.map$', '\\\\\\\\.map$'), egress=, header_prefix='## ', fence='\`\`\`', on_drop=None) Remove bloated sections (vendored deps, build output, lockfiles, minified bundles, …) from an aggregated-code markdown. This is the post-hoc complement to `code_aggregate`’s `keys_exclude`: use it to slim down an aggregate that was already generated (e.g. one that accidentally included a `node_modules/` tree). It streams, so it handles arbitrarily large files. * **Parameters:** * **src** (`Union`[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Iterable`](https://docs.python.org/3/library/typing.html#typing.Iterable)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]]) – The aggregate as a markdown string, a path to a markdown file, or any iterable of lines. * **exclude** – Sections to drop (see `iter_pruned_aggregate_lines`). * **egress** (`Union`[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]) – What to do with the result. The default (`identity`) returns the pruned markdown as a string. Pass a filepath string to stream the result to that file (memory-safe for huge inputs) and return the filepath. * **header_prefix** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Aggregate format markers (see `iter_pruned_aggregate_lines`). * **fence** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Aggregate format markers (see `iter_pruned_aggregate_lines`). * **on_drop** ([`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`Callable`](https://docs.python.org/3/library/typing.html#typing.Callable)[[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str)], [`Any`](https://docs.python.org/3/library/typing.html#typing.Any)]]) – Optional callback invoked with each dropped section’s path. * **Return type:** [`Any`](https://docs.python.org/3/library/typing.html#typing.Any) * **Returns:** The pruned markdown string, or the output filepath when `egress` is a path. ### Example ```pycon >>> md = "## keep.py\n\n```python\nok\n```\n## dist/bundle.js\n\n```python\nx\n```\n" >>> "dist/bundle.js" in prune_code_aggregate(md) False ``` ### contaix.code.resolve_code_source(code_src, keys_filt=>, \*, keys_exclude=None) Will resolve code_src to a Mapping whose values are the code strings * **Parameters:** * **code_src** (`Union`[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Mapping`](https://docs.python.org/3/library/typing.html#typing.Mapping)]) – The source of the code. Can be an explicit `Mapping`, a directory path string, a GitHub URL, or an imported package (must contain a \_\_path_\_ atribute) * **keys_filt** (*Callable*) – A function or regex string to filter the keys of the code store. If a string, it will be compiled to a regex pattern. * **keys_filt** – A function to filter the keys. Defaults to lambda x: x.endswith(‘.py’). * **keys_exclude** – Optional regex string, iterable of regex strings, or `path->bool` predicate identifying keys to *exclude* (e.g. vendored deps, build outputs). Applied on top of (after) `keys_filt`. Pass `DFLT_AGGREGATE_EXCLUDE_PATTERNS` to skip the usual bloat. * **Return type:** [`Mapping`](https://docs.python.org/3/library/typing.html#typing.Mapping) ### contaix.code.resolve_code_source_dir_path(code_src) Resolves code_src to a directory path string. If the input is a package object or name, it will return the local path where the package is located. If the input is a github URL, this repository will be DOWNLOAD to a local temporary directory and the path of that directory will be returned. * **Parameters:** **code_src** (`Union`[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Mapping`](https://docs.python.org/3/library/typing.html#typing.Mapping)]) – The source of the code. Can be a directory path string, a GitHub URL, or an imported package (must contain a \_\_path_\_ atribute) * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) * **Returns:** The resolved directory path. * **Raises:** [**AssertionError**](https://docs.python.org/3/builtins/exceptions.html#AssertionError) – If the resolved path is not a valid directory. # _autosummary/contaix.html.md # contaix Tools to make contexts (knowledge bases for AI agents) ### Examples Download all articles from a markdown string and save them as PDF files: ```pycon >>> download_articles(md_string) ``` Verify URLs in a markdown string by checking their status codes (useful when trying to verify if AI hallucinated the urls) ```pycon >>> verify_urls(md_string) ``` Make an md file with all the code in a directory: ```pycon >>> md_string = code_aggregate(package_or_folder_or_github_url) ``` ### Modules | [`aggregation`](_autosummary/contaix.aggregation.html.md#module-contaix.aggregation) | Tools for aggregating contexts | |-------------------------------------------------------------------------------------------|-------------------------------------------------------------| | [`code`](_autosummary/contaix.code.html.md#module-contaix.code) | Tools to make AI contexts from code bases | | [`markdown`](_autosummary/contaix.markdown.html.md#module-contaix.markdown) | Converting things to markdown | | [`skills`](_autosummary/contaix.skills.html.md#module-contaix.skills) | Skill discovery and installation for the contaix ecosystem. | | [`urls`](_autosummary/contaix.urls.html.md#module-contaix.urls) | Work with urls | | [`util`](_autosummary/contaix.util.html.md#module-contaix.util) | General utilities for contaix | | [`web`](_autosummary/contaix.web.html.md#module-contaix.web) | Web content extraction and aggregation tools. | # _autosummary/contaix.markdown.html.md # contaix.markdown Converting things to markdown This module provides tools to convert various file formats (PDF, Word, Excel, PowerPoint, HTML) to markdown format. It includes: - Default converters for common file formats - Functions to convert bytes to markdown - Store-based converters for processing multiple files # _autosummary/contaix.skills.html.md # contaix.skills Skill discovery and installation for the contaix ecosystem. Claude Code skills are stored in `~/.claude/skills/` as folders containing `SKILL.md` files. When packages are pip-installed, their skills live in `/data/skills/` and need to be symlinked into the global directory. This module discovers skills from installed packages and creates those symlinks. Usage: ```default # From the command line: python -m contaix.skills # From Python: from contaix.skills import install_skills install_skills() # List available skills without installing: from contaix.skills import discover_skills for name, path in discover_skills().items(): print(f" {name}: {path}") ``` ### Functions | [`discover_skills`](_autosummary/contaix.skills.html.md#contaix.skills.discover_skills)([packages]) | Discover skills from installed ecosystem packages. | |---------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------| | [`install_skills`](_autosummary/contaix.skills.html.md#contaix.skills.install_skills)([packages, force, verbose]) | Create symlinks in `~/.claude/skills/` for discovered skills. | | [`main`](_autosummary/contaix.skills.html.md#contaix.skills.main)() | CLI entry point: discover and install skills. | | [`uninstall_skills`](_autosummary/contaix.skills.html.md#contaix.skills.uninstall_skills)([packages, verbose]) | Remove symlinks created by [`install_skills()`](_autosummary/contaix.skills.html.md#contaix.skills.install_skills). | ### contaix.skills.discover_skills(packages=('contaix', 'dn', 'pdfdol')) Discover skills from installed ecosystem packages. Scans each package’s `data/skills/` directory for folders containing a `SKILL.md` file. * **Parameters:** **packages** ([*tuple*](https://docs.python.org/3/builtins/stdtypes.html#tuple) *of* [*str*](https://docs.python.org/3/builtins/stdtypes.html#str)) – Package names to scan. * **Returns:** Mapping of skill name to its `Path` in the installed package. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`Path`](https://docs.python.org/3/library/pathlib.html#pathlib.Path)] ### contaix.skills.install_skills(packages=('contaix', 'dn', 'pdfdol'), , force=False, verbose=True) Create symlinks in `~/.claude/skills/` for discovered skills. * **Parameters:** * **packages** ([*tuple*](https://docs.python.org/3/builtins/stdtypes.html#tuple) *of* [*str*](https://docs.python.org/3/builtins/stdtypes.html#str)) – Package names to scan for skills. * **force** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – If True, replace existing symlinks (but not regular directories). * **verbose** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Print progress. ### contaix.skills.main() CLI entry point: discover and install skills. ### contaix.skills.uninstall_skills(packages=('contaix', 'dn', 'pdfdol'), , verbose=True) Remove symlinks created by [`install_skills()`](_autosummary/contaix.skills.html.md#contaix.skills.install_skills). Only removes symlinks (not regular directories) to avoid data loss. * **Parameters:** * **packages** ([*tuple*](https://docs.python.org/3/builtins/stdtypes.html#tuple) *of* [*str*](https://docs.python.org/3/builtins/stdtypes.html#str)) – Package names whose skills should be removed. * **verbose** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Print progress. # _autosummary/contaix.urls.html.md # contaix.urls Work with urls ### Functions | [`download_articles`](_autosummary/contaix.urls.html.md#contaix.urls.download_articles)([md_string, save_dir, ...]) | Download PDF articles from markdown text. | |----------------------------------------------------------------------------------------------------|--------------------------------------------------------------------| | [`download_articles_by_section`](_autosummary/contaix.urls.html.md#contaix.urls.download_articles_by_section)([md_string, ...]) | Download articles by markdown section. | | [`extract_html_links`](_autosummary/contaix.urls.html.md#contaix.urls.extract_html_links)([markdown]) | Extract URLs from HTML anchor tags. | | [`extract_markdown_links`](_autosummary/contaix.urls.html.md#contaix.urls.extract_markdown_links)([markdown, pattern]) | Extract markdown links from a string. | | [`extract_urls`](_autosummary/contaix.urls.html.md#contaix.urls.extract_urls)([markdown, pattern, extractor]) | Extract URLs and their context from a markdown string. | | [`extract_urls_only`](_autosummary/contaix.urls.html.md#contaix.urls.extract_urls_only)([markdown]) | Extract URLs with empty context. | | [`extract_with_surrounding_context`](_autosummary/contaix.urls.html.md#contaix.urls.extract_with_surrounding_context)([markdown, ...]) | Extract URLs with surrounding text as context. | | `get_from_clipboard`() | | | [`remove_hyperlink_crap`](_autosummary/contaix.urls.html.md#contaix.urls.remove_hyperlink_crap)([string, ...]) | Remove unwanted hyperlinks and citations from a string. | | [`verify_urls`](_autosummary/contaix.urls.html.md#contaix.urls.verify_urls)([src]) | Verifies URLs in a markdown string by checking their status codes. | ### contaix.urls.download_articles(md_string=None, save_dir='/home/runner/Downloads', , save_non_pdf=False, verbose=True) Download PDF articles from markdown text. Wrapper around `pdfdol.download`. If `md_string` is None, reads from clipboard (requires pyperclip). See `pdfdol.download.download_articles` for full documentation. ### contaix.urls.download_articles_by_section(md_string=None, rootdir=None, save_non_pdf=False, , section_marker='###') Download articles by markdown section. Wrapper around `pdfdol.download`. If `md_string` is None, reads from clipboard (requires pyperclip). See `pdfdol.download.download_articles_by_section` for full documentation. ### contaix.urls.extract_html_links(markdown=None) Extract URLs from HTML anchor tags. * **Parameters:** **markdown** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – The markdown or HTML string to process * **Return type:** [`Iterator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterator)[[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]] * **Returns:** Iterator of (anchor_text, url) pairs ### contaix.urls.extract_markdown_links(markdown=None, pattern=None) Extract markdown links from a string. #### NOTE By default this also extracts bare `http(s)://...` URLs found in the text. For those, the returned context/name is the URL itself. * **Parameters:** * **markdown** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – The markdown string to process * **pattern** ([`Pattern`](https://docs.python.org/3/library/re.html#re.Pattern) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – A compiled regex pattern to match markdown links * **Return type:** [`Iterator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterator)[[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]] * **Returns:** Iterator of (context, url) pairs ```pycon >>> text = "[Google](https://google.com) and https://github.com" >>> list(extract_markdown_links(text)) [('Google', 'https://google.com'), ('https://github.com', 'https://github.com')] ``` ### contaix.urls.extract_urls(markdown=None, pattern=None, extractor=None) Extract URLs and their context from a markdown string. * **Parameters:** * **markdown** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – The markdown string to process * **pattern** ([`Pattern`](https://docs.python.org/3/library/re.html#re.Pattern) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – A compiled regex pattern to match URLs and their context. Defaults to matching both markdown hyperlinks `[context](url)` and bare `http(s)://...` URLs. * **extractor** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)[[[`Match`](https://docs.python.org/3/library/re.html#re.Match)], [`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]] | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – A function that extracts `(context, url)` from a match. Defaults to: - `(context, url)` for markdown hyperlinks - `(url, url)` for bare URLs * **Return type:** [`Iterator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterator)[[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]] * **Returns:** Iterator of (context, url) pairs ```pycon >>> text = "[Google](https://google.com) and https://github.com" >>> list(extract_urls(text)) [('Google', 'https://google.com'), ('https://github.com', 'https://github.com')] ``` ### contaix.urls.extract_urls_only(markdown=None) Extract URLs with empty context. * **Parameters:** **markdown** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – String containing markdown text * **Return type:** [`Iterator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterator)[[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]] * **Returns:** Iterator of tuples with empty string and extracted URLs ```pycon >>> list(extract_urls_only("Check [this link](https://example.com) and https://github.com/user/repo)")) [('', 'https://example.com'), ('', 'https://github.com/user/repo')] ``` ### contaix.urls.extract_with_surrounding_context(markdown=None, context_chars=30) Extract URLs with surrounding text as context. * **Parameters:** * **markdown** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – The markdown string to process * **context_chars** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Number of characters to include before and after URL * **Return type:** [`Iterator`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Iterator)[[`tuple`](https://docs.python.org/3/builtins/stdtypes.html#tuple)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)]] * **Returns:** Iterator of (context, url) pairs ```pycon >>> text = "Check this link: [Google](https://google.com) and [GitHub](https://github.com)" >>> list(extract_with_surrounding_context(text, context_chars=5)) [('gle](https://google.com) and', 'https://google.com)'), ('Hub](https://github.com)', 'https://github.com)')] ``` ### contaix.urls.remove_hyperlink_crap(string=None, copy_to_clipboard=True) Remove unwanted hyperlinks and citations from a string. Typically used to clean up text copied from ChatGPT or Claude. Delegates to `dn.repair.remove_hyperlink_crap` for the pure transform, adding clipboard integration on top. If no string is specified, reads from clipboard (requires pyperclip). ### contaix.urls.verify_urls(src=None) Verifies URLs in a markdown string by checking their status codes. * **Parameters:** **src** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`list`](https://docs.python.org/3/builtins/stdtypes.html#list) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – The markdown string or list containing URLs. If a string starting with `[`, parsed as a JSON list of URLs. Otherwise, URLs are extracted from the markdown text. * **Returns:** A dictionary with URLs as keys and their status codes as values. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`int`](https://docs.python.org/3/builtins/functions.html#int) | [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] ```pycon >>> verify_urls(['https://example.com']) {'https://example.com': 200} ``` # _autosummary/contaix.util.html.md # contaix.util General utilities for contaix This module provides core utility functions used throughout the contaix package, including: - File path handling (fullpath) - URL detection and content retrieval (is_url, url_to_contents) - File saving utilities (save_to_file_and_return_file) - Basic helper functions (identity) ### Functions | [`fullpath`](_autosummary/contaix.util.html.md#contaix.util.fullpath)(path) | Returns the full path of the given path. | |----------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [`identity`](_autosummary/contaix.util.html.md#contaix.util.identity)(x) | Returns the input unchanged. | | [`is_url`](_autosummary/contaix.util.html.md#contaix.util.is_url)(path) | Check if the given path is a URL. | | [`remove_improperly_double_newlines`](_autosummary/contaix.util.html.md#contaix.util.remove_improperly_double_newlines)(string, \*) | Remove improperly double newlines from a string. | | [`save_to_file_and_return_file`](_autosummary/contaix.util.html.md#contaix.util.save_to_file_and_return_file)([obj, encoder, key]) | Save `encoder(obj)` to a file using a random name in `rootdir` (or a temp directory if not provided). | | [`source_first_arg_from_clipboard_if_none`](_autosummary/contaix.util.html.md#contaix.util.source_first_arg_from_clipboard_if_none)(func) | Decorator that sources a `string` argument from the clipboard when it's None and copies the function's return value back to the clipboard when the keyword argument `copy_to_clipboard` is truthy. | | [`url_to_contents`](_autosummary/contaix.util.html.md#contaix.util.url_to_contents)(url, \*[, kind, timeout]) | Fetch the content from a URL. | ### contaix.util.fullpath(path) Returns the full path of the given path. * **Parameters:** **path** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The path to convert to a full path. * **Returns:** The full path. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### Example ```pycon >>> fullpath('~/Downloads') '/home/user/Downloads' ``` ```pycon >>> fullpath('.') '/home/user/python_projects/aix/aix' ``` ### contaix.util.identity(x) Returns the input unchanged. * **Parameters:** **x** – Any input * **Returns:** The input unchanged ### contaix.util.is_url(path) Check if the given path is a URL. * **Parameters:** **path** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Path to check * **Returns:** True if the path is a URL, False otherwise * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### contaix.util.remove_improperly_double_newlines(string, , copy_to_clipboard=True) Remove improperly double newlines from a string. Delegates to `dn.repair.remove_improperly_double_newlines` for the pure transform, adding clipboard integration via the decorator. * **Parameters:** **string** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None)) – The input string. * **Returns:** The string with improperly double newlines removed. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### Example ```pycon >>> text = "This is a test.\n\nThis should be one newline.\n \nThis too." >>> result = remove_improperly_double_newlines(text, copy_to_clipboard=False) >>> assert result == 'This is a test.\n\nThis should be one newline.\nThis too.' ``` ### contaix.util.save_to_file_and_return_file(obj=None, \*, encoder=, key=None) Save `encoder(obj)` to a file using a random name in `rootdir` (or a temp directory if not provided). Returns the full path to the saved file. If `obj` is None, returns a partial function with preconfigured `encoder` and `rootdir`. * **Parameters:** * **obj** – The object to save. If None, return a partial function. * **encoder** – A function to encode the object into text or bytes. Defaults to identity. * **key** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)) – The key (by default, filepath) to write to. If None, a temporary file is created. If a string starting with ‘\*’, the ‘\*’ is replaced with a unique temporary filename. If a string that has a ‘\*’ somewhere in the middle, what’s on the left of if is used as a directory and the ‘\*’ is replaced with a unique temporary filename. For example ‘/tmp/\*_file.ext’ would be replaced with ‘/tmp/oiu8fj9873_file.ext’. If a callable, it will be called with obj as input to get the key. One use case is to use a function that generates a key based on the object. * **Returns:** Full path to the saved file, or a partial function if `obj` is None. * **Return type:** [*str*](https://docs.python.org/3/builtins/stdtypes.html#str) ### Examples ```pycon >>> from pathlib import Path >>> filepath = save_to_file_and_return_file("hello world") >>> import os >>> Path(filepath).read_text() 'hello world' ``` The default encoder is identity, so you can save binary data as well: ```pycon >>> filepath = save_to_file_and_return_file(b"binary data", encoder=lambda x: x) >>> Path(filepath).read_bytes() b'binary data' ``` ### contaix.util.source_first_arg_from_clipboard_if_none(func) Decorator that sources a `string` argument from the clipboard when it’s None and copies the function’s return value back to the clipboard when the keyword argument `copy_to_clipboard` is truthy. The wrapped function must accept a parameter named `string`. It may also accept a keyword-only parameter named `copy_to_clipboard` (default True). ### contaix.util.url_to_contents(url, , kind='binary', timeout=10) Fetch the content from a URL. * **Parameters:** * **url** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The URL to fetch. * **kind** ([*str*](https://docs.python.org/3/builtins/stdtypes.html#str)) – The kind of content to fetch (‘text’, ‘binary’, or ‘response’). Defaults to ‘binary’. * **timeout** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Timeout for the request in seconds. Default is 10 seconds. * **Returns:** The content of the URL if successful, None otherwise. * **Return type:** [`Optional`](https://docs.python.org/3/library/typing.html#typing.Optional)[[`bytes`](https://docs.python.org/3/builtins/stdtypes.html#bytes)] # _autosummary/contaix.web.html.md # contaix.web Web content extraction and aggregation tools. Functions for extracting structured content from websites and converting to markdown, with support for navigation discovery, caching, and content aggregation. Main entry points: > site_to_markdown(url) # Full pipeline: discover nav, fetch pages, aggregate > extract_site_nav(url) # Get ordered navigation structure from a doc site > fetch_page(url) # Fetch a single page’s HTML with optional caching Example: ```default >>> md = site_to_markdown( ... 'https://platform.claude.com/docs/en/home', ... cache_dir='/tmp/claude_docs_cache', ... output_file='~/Downloads/claude_docs.md', ... ) ``` ### Functions | [`extract_rsc_page_content`](_autosummary/contaix.web.html.md#contaix.web.extract_rsc_page_content)(rsc_text) | Extract page content from RSC flight data. | |-----------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------| | [`extract_site_nav`](_autosummary/contaix.web.html.md#contaix.web.extract_site_nav)(url, \*[, html]) | Extract navigation structure from a documentation site. | | [`fetch_llms_full`](_autosummary/contaix.web.html.md#contaix.web.fetch_llms_full)(url, \*[, timeout]) | Fetch a publisher-provided single-doc bundle if available. | | [`fetch_nextjs_rsc`](_autosummary/contaix.web.html.md#contaix.web.fetch_nextjs_rsc)(url, \*[, cache_dir, force, ...]) | Fetch page content via Next.js RSC flight endpoint. | | [`fetch_page`](_autosummary/contaix.web.html.md#contaix.web.fetch_page)(url, \*[, cache_dir, force, timeout]) | Fetch a page's HTML content, with optional disk caching. | | [`fetch_pages`](_autosummary/contaix.web.html.md#contaix.web.fetch_pages)(urls, \*[, cache_dir, force, verbose]) | Fetch multiple pages with caching. | | [`find_llms_full_url`](_autosummary/contaix.web.html.md#contaix.web.find_llms_full_url)(url, \*[, timeout]) | Probe for a publisher-provided single-doc bundle. | | [`html_to_clean_markdown`](_autosummary/contaix.web.html.md#contaix.web.html_to_clean_markdown)(html, \*[, ...]) | Convert HTML to clean markdown using html2text. | | [`is_nextjs_site`](_autosummary/contaix.web.html.md#contaix.web.is_nextjs_site)(html) | Detect if a page is served by Next.js (App Router). | | [`list_site_pages`](_autosummary/contaix.web.html.md#contaix.web.list_site_pages)(url) | List all pages found in a documentation site's navigation. | | [`parse_rsc_flight`](_autosummary/contaix.web.html.md#contaix.web.parse_rsc_flight)(rsc_text) | Parse RSC flight data into a dict mapping keys to parsed JSON data. | | [`rsc_tree_to_markdown`](_autosummary/contaix.web.html.md#contaix.web.rsc_tree_to_markdown)(node, \*[, \_registry, ...]) | Recursively extract markdown from a React Server Component tree node. | | [`site_to_markdown`](_autosummary/contaix.web.html.md#contaix.web.site_to_markdown)(url, \*[, cache_dir, ...]) | Download a documentation site and produce a single aggregated markdown. | | [`url_to_cache_key`](_autosummary/contaix.web.html.md#contaix.web.url_to_cache_key)(url) | Convert a URL to a filesystem-safe cache key. | ### contaix.web.extract_rsc_page_content(rsc_text) Extract page content from RSC flight data. Works on both: - RSC flight endpoint responses (`text/x-component`) - Inline RSC payloads extracted from HTML Returns markdown text if content was found, None otherwise. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None) ### contaix.web.extract_site_nav(url, , html=None) Extract navigation structure from a documentation site. Returns a dict with keys: > - `base_url`: The base URL of the site > - `tabs`: List of tab dicts, each with `label` and `groups` > - `pages`: Flat ordered list of `{path, title}` dicts (all tabs) Tries, in order: > 1. Next.js RSC payload (embedded in SSR HTML) > 2. HTML `` link extraction with path-based grouping * **Parameters:** * **url** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The documentation site URL (e.g. a homepage or any page on the site). * **html** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Pre-fetched HTML. If None, fetches from `url`. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### contaix.web.fetch_llms_full(url, , timeout=30) Fetch a publisher-provided single-doc bundle if available. See [`find_llms_full_url()`](_autosummary/contaix.web.html.md#contaix.web.find_llms_full_url) for how the URL is discovered. Returns the markdown text, or `None` if no bundle was found. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None) ### contaix.web.fetch_nextjs_rsc(url, , cache_dir=None, force=False, timeout=30) Fetch page content via Next.js RSC flight endpoint. Next.js App Router sites serve React Server Component payloads when the `RSC: 1` header is present. These contain the full page content that would otherwise require JavaScript rendering. * **Parameters:** * **url** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The page URL. * **cache_dir** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Directory to cache RSC responses. * **force** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Re-fetch even if cached. * **timeout** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Request timeout. * **Returns:** The RSC flight data as text. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### contaix.web.fetch_page(url, , cache_dir=None, force=False, timeout=30) Fetch a page’s HTML content, with optional disk caching. * **Parameters:** * **url** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The URL to fetch. * **cache_dir** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Directory to cache HTML files. If None, no caching. * **force** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – If True, re-fetch even if cached. * **timeout** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Request timeout in seconds. * **Returns:** The page’s HTML content. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### contaix.web.fetch_pages(urls, , cache_dir=None, force=False, verbose=False) Fetch multiple pages with caching. Returns a dict mapping URL -> HTML content. Pages that fail to fetch are skipped with a warning. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)[[`str`](https://docs.python.org/3/builtins/stdtypes.html#str), [`str`](https://docs.python.org/3/builtins/stdtypes.html#str)] ### contaix.web.find_llms_full_url(url, , timeout=30) Probe for a publisher-provided single-doc bundle. Many documentation generators (Mintlify, Docusaurus, Fern, etc.) expose the entire docs as a single markdown file at `/llms-full.txt` (or `/llms.txt` for an index). When present, this is far better than scraping page-by-page. Tries, in order: 1. Same path as the input URL with the doc-root replaced (e.g. `https://site.com/docs/foo` -> `https://site.com/docs/llms-full.txt`). 2. Site root (e.g. `https://site.com/llms-full.txt`). * **Parameters:** * **url** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Any URL on the documentation site. * **timeout** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – HEAD request timeout. * **Returns:** The first URL that returns HTTP 200 with non-trivial content, or `None` if none of the candidates exist. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`None`](https://docs.python.org/3/builtins/constants.html#None) ### contaix.web.html_to_clean_markdown(html, , body_width=0, ignore_images=True, include_links=True, \*\*html2text_options) Convert HTML to clean markdown using html2text. * **Parameters:** * **html** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Raw HTML content. * **body_width** ([`int`](https://docs.python.org/3/builtins/functions.html#int)) – Line width for wrapping. 0 = no wrapping. * **ignore_images** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to skip image tags. * **include_links** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Whether to include hyperlinks in output. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### contaix.web.is_nextjs_site(html) Detect if a page is served by Next.js (App Router). Looks for `self.__next_f.push` in the HTML, which is the RSC streaming payload signature. * **Return type:** [`bool`](https://docs.python.org/3/builtins/functions.html#bool) ### contaix.web.list_site_pages(url) List all pages found in a documentation site’s navigation. Returns a list of dicts with keys: path, title, url, group, tab. * **Return type:** [`list`](https://docs.python.org/3/builtins/stdtypes.html#list)[[`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict)] ```pycon >>> pages = list_site_pages('https://platform.claude.com/docs/en/home') >>> len(pages) 89 ``` ### contaix.web.parse_rsc_flight(rsc_text) Parse RSC flight data into a dict mapping keys to parsed JSON data. Each RSC record starts with `:`. Records come in three flavors: - `I[...]`: module import descriptors (skipped here). - `T,`: a raw text chunk whose payload occupies exactly `hex_size` bytes (often markdown, code, or compiled MDX). The payload may contain newlines, so we read by byte count rather than splitting on `\n`. - Anything else: a JSON value terminated by the next record header (`\n:`) or end-of-text. * **Parameters:** **rsc_text** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Raw RSC flight response text (`text/x-component`). * **Returns:** Mapping of hex keys to their parsed payloads. * **Return type:** [`dict`](https://docs.python.org/3/builtins/stdtypes.html#dict) ### contaix.web.rsc_tree_to_markdown(node, , \_registry=None, \_depth=0) Recursively extract markdown from a React Server Component tree node. The RSC tree uses the format: ```default ["$", "tagName", key, {"children": ..., "className": ...}] ``` or plain strings for text nodes. Component references like `"$L2a"` are resolved via `_registry` (from [`parse_rsc_flight()`](_autosummary/contaix.web.html.md#contaix.web.parse_rsc_flight)). * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### contaix.web.site_to_markdown(url, , cache_dir=None, output_file=None, tab_filter=None, page_fetcher=None, content_extractor=None, section_separator='\\\\n\\\\n---\\\\n\\\\n', collapse_blank_lines=True, use_llms_full=True, verbose=False) Download a documentation site and produce a single aggregated markdown. * **Parameters:** * **url** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – The documentation site’s root URL. * **cache_dir** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Directory for caching fetched HTML. Defaults to `~/.cache/contaix/web`. * **output_file** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Path to save the markdown file. If None, returns the string. * **tab_filter** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str) | [`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)) – If a string, only include pages from tabs whose label contains this string (case-insensitive). If a callable, `tab_filter(tab_label) -> bool`. * **page_fetcher** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)) – Custom `(url) -> html` function. Defaults to `fetch_page`. * **content_extractor** ([`Callable`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Callable)) – Custom `(html) -> markdown` function. Defaults to auto-detection. * **section_separator** ([`str`](https://docs.python.org/3/builtins/stdtypes.html#str)) – Separator between page sections in the output. * **use_llms_full** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – If True (default), first try to find a publisher-provided `/llms-full.txt` bundle (Mintlify, Docusaurus, etc. ship these). When found, returns it directly instead of scraping page-by-page. Disable to force the scraping pipeline. Skipped automatically when `page_fetcher` or `tab_filter` is set, since those imply the caller wants the scraping path. * **verbose** ([`bool`](https://docs.python.org/3/builtins/functions.html#bool)) – Print progress information. * **Returns:** If `output_file` is None, returns the markdown string. Otherwise, returns the output file path. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ### contaix.web.url_to_cache_key(url) Convert a URL to a filesystem-safe cache key. * **Return type:** [`str`](https://docs.python.org/3/builtins/stdtypes.html#str) ```pycon >>> url_to_cache_key('https://example.com/docs/en/foo') 'example.com__docs__en__foo.html' ``` # about-this-build.html.md # About this build This documentation was built on **2026-09-22 13:47 UTC** from commit 8e7afad on branch main, for **contaix 0.0.19** (from pyproject.toml). #### WARNING The documentation and the package may be misaligned: - The documented version (0.0.19) is behind the latest release on PyPI (0.0.20): `pip install contaix` gives newer code than these docs describe. ## Source | | | |---------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------| | Commit | 8e7afad3f150efc4765bd12934b61891243261b5 | | Branch | main | | Tags at this commit | none | | Working tree | clean | | Remote | https://github.com/thorwhalen/contaix | ## Continuous integration | | | |--------------|--------------------------------------------------------------------------------------------| | Repository | thorwhalen/contaix | | Run | 35735523995 | | Ref | refs/heads/main | | Event commit | 8e7afad3f150efc4765bd12934b61891243261b5 (in the history of the built commit) | ## Tools | | | |----------|---------| | epythet | 0.2.12 | | Sphinx | 9.1.0 | | docutils | 0.22.4 | | Python | 3.12.14 | ## Configuration as resolved | | | |---------------|------------------------------------------------------------------| | theme | auto (Sphinx theme furo) | | accent | #8c325d | | api_generator | autosummary | | ignore | tests/, scrap/, examples/ | | agent_outputs | true | | aggregates | md | | ai_artifacts | true | ## Package on PyPI Latest release: 0.0.20, newer than the documented version (0.0.19). ## Reproduce ```bash git clone https://github.com/thorwhalen/contaix && cd contaix git checkout 8e7afad3f150efc4765bd12934b61891243261b5 pip install "epythet==0.2.12" epythet quickstart . --ignore tests/ scrap/ examples/ ``` The same data, for machines: build_info.json (schema version 1). # ai-agents.html.md # For AI agents `contaix` ships artifacts for coding agents alongside its code. This page lists them, says where each lives in the repository, and points at the machine-readable copies of this documentation. ## Skills Skills are folders holding a `SKILL.md` (the [Agent Skills](https://agentskills.io) format): a description that tells an agent when to use it and a body with the procedure. Install one into your agent with `gh skill` (any host: `--agent claude-code`, `copilot`, `cursor`, `codex`, `gemini`), or use the copy bundled in the wheel. ### `contaix-acquire-docs` Acquire a tool’s documentation as a single agent-ready markdown bundle, plus any machine-readable specs (OpenAPI, MCP, AsyncAPI, GraphQL). Use whenever the user provides one or more URLs (or a tool/product name) and wants the docs / manual / API reference as a context bundle for an AI agent. Triggers on “acquire the docs”, “get me the docs”, “bundle this manual”, “make a context for tool X”, “download the API reference”, “give me everything as one md”, “fetch the OpenAPI spec”, “grab the MCP spec”, or any URL pointing at a documentation / developer / API / manual site. Tries cheapest paths first — GitHub source, llms.txt, raw-markdown variants of doc URLs — and only falls back to HTML scraping (via contaix_docs_to_markdown / contaix_web_aggregate) when nothing better is available. ```bash gh skill install thorwhalen/contaix contaix-acquire-docs --agent claude-code ``` Source: [`skills/contaix-acquire-docs`](https://github.com/thorwhalen/contaix/tree/HEAD/skills/contaix-acquire-docs). ### `contaix_docs_to_markdown` Convert a documentation website into a single, clean markdown file. Use when the user provides a documentation URL and wants it as markdown, or says “get me the docs”, “download documentation”, “make a markdown of this site”, “aggregate these docs”, or gives a URL to a docs site. Also use for converting API reference sites, developer guides, product documentation, or knowledge bases. ```bash gh skill install thorwhalen/contaix contaix_docs_to_markdown --agent claude-code ``` Source: [`skills/contaix_docs_to_markdown`](https://github.com/thorwhalen/contaix/tree/HEAD/skills/contaix_docs_to_markdown). ### `contaix_web_aggregate` Aggregate web content from multiple pages into structured markdown. Use when the user wants to aggregate a non-documentation website (company info, product pages), when contaix_docs_to_markdown doesn’t work for a specific site, when the user wants manual control over which pages to include, or when dealing with JavaScript-rendered sites that need WebFetch. ```bash gh skill install thorwhalen/contaix contaix_web_aggregate --agent claude-code ``` Source: [`skills/contaix_web_aggregate`](https://github.com/thorwhalen/contaix/tree/HEAD/skills/contaix_web_aggregate). ## Instruction files Files agents read before working in this repository. - [`.claude/CLAUDE.md`](https://github.com/thorwhalen/contaix/tree/HEAD/.claude/CLAUDE.md): read by Claude Code ## Machine-readable documentation This site publishes the same documentation in forms that fit an agent’s context window: - [`llms.txt`](https://thorwhalen.github.io/contaix/llms.txt): an index of every page with a one-line description ([llms.txt](https://llmstxt.org) format) - [`contaix.md`](https://thorwhalen.github.io/contaix/contaix.md): the whole documentation as one Markdown file - `.html.md`: a rendered Markdown twin of every page, advertised from each page’s `` with `` - [`objects.inv`](https://thorwhalen.github.io/contaix/objects.inv): the Sphinx inventory: a symbol-to-URL index (`sphobjinv convert plain objects.inv -`) # api.html.md # API reference | [`contaix`](_autosummary/contaix.html.md#module-contaix) | Tools to make contexts (knowledge bases for AI agents) | |---------------------------------------------------------------------------|----------------------------------------------------------|