contaix.web

Web content extraction and aggregation tools.

Functions for extracting structured content from websites and converting to markdown, with support for navigation discovery, caching, and content aggregation.

Main entry points:

site_to_markdown(url) # Full pipeline: discover nav, fetch pages, aggregate extract_site_nav(url) # Get ordered navigation structure from a doc site fetch_page(url) # Fetch a single page’s HTML with optional caching

Example:

>>> md = site_to_markdown(
...     'https://platform.claude.com/docs/en/home',
...     cache_dir='/tmp/claude_docs_cache',
...     output_file='~/Downloads/claude_docs.md',
... )

Functions

extract_rsc_page_content(rsc_text)

Extract page content from RSC flight data.

extract_site_nav(url, *[, html])

Extract navigation structure from a documentation site.

fetch_llms_full(url, *[, timeout])

Fetch a publisher-provided single-doc bundle if available.

fetch_nextjs_rsc(url, *[, cache_dir, force, ...])

Fetch page content via Next.js RSC flight endpoint.

fetch_page(url, *[, cache_dir, force, timeout])

Fetch a page's HTML content, with optional disk caching.

fetch_pages(urls, *[, cache_dir, force, verbose])

Fetch multiple pages with caching.

find_llms_full_url(url, *[, timeout])

Probe for a publisher-provided single-doc bundle.

html_to_clean_markdown(html, *[, ...])

Convert HTML to clean markdown using html2text.

is_nextjs_site(html)

Detect if a page is served by Next.js (App Router).

list_site_pages(url)

List all pages found in a documentation site's navigation.

parse_rsc_flight(rsc_text)

Parse RSC flight data into a dict mapping keys to parsed JSON data.

rsc_tree_to_markdown(node, *[, _registry, ...])

Recursively extract markdown from a React Server Component tree node.

site_to_markdown(url, *[, cache_dir, ...])

Download a documentation site and produce a single aggregated markdown.

url_to_cache_key(url)

Convert a URL to a filesystem-safe cache key.
contaix.web.extract_rsc_page_content(rsc_text)[source]

Extract page content from RSC flight data.

Works on both:

  • RSC flight endpoint responses (text/x-component)

  • Inline RSC payloads extracted from HTML

Returns markdown text if content was found, None otherwise.

Return type:

str | None

contaix.web.extract_site_nav(url, *, html=None)[source]

Extract navigation structure from a documentation site.

Returns a dict with keys:

  • base_url: The base URL of the site

  • tabs: List of tab dicts, each with label and groups

  • pages: Flat ordered list of {path, title} dicts (all tabs)

Tries, in order:

  1. Next.js RSC payload (embedded in SSR HTML)

  2. HTML <a> link extraction with path-based grouping

Parameters:

  • url (str) – The documentation site URL (e.g. a homepage or any page on the site).

  • html (str) – Pre-fetched HTML. If None, fetches from url.

Return type:

dict

contaix.web.fetch_llms_full(url, *, timeout=30)[source]

Fetch a publisher-provided single-doc bundle if available.

See find_llms_full_url() for how the URL is discovered.

Returns the markdown text, or None if no bundle was found.

Return type:

str | None

contaix.web.fetch_nextjs_rsc(url, *, cache_dir=None, force=False, timeout=30)[source]

Fetch page content via Next.js RSC flight endpoint.

Next.js App Router sites serve React Server Component payloads when the RSC: 1 header is present. These contain the full page content that would otherwise require JavaScript rendering.

Parameters:

  • url (str) – The page URL.

  • cache_dir (str) – Directory to cache RSC responses.

  • force (bool) – Re-fetch even if cached.

  • timeout (int) – Request timeout.

Returns:

The RSC flight data as text.

Return type:

str

contaix.web.fetch_page(url, *, cache_dir=None, force=False, timeout=30)[source]

Fetch a page’s HTML content, with optional disk caching.

Parameters:

  • url (str) – The URL to fetch.

  • cache_dir (str) – Directory to cache HTML files. If None, no caching.

  • force (bool) – If True, re-fetch even if cached.

  • timeout (int) – Request timeout in seconds.

Returns:

The page’s HTML content.

Return type:

str

contaix.web.fetch_pages(urls, *, cache_dir=None, force=False, verbose=False)[source]

Fetch multiple pages with caching.

Returns a dict mapping URL -> HTML content. Pages that fail to fetch are skipped with a warning.

Return type:

dict[str, str]

contaix.web.find_llms_full_url(url, *, timeout=30)[source]

Probe for a publisher-provided single-doc bundle.

Many documentation generators (Mintlify, Docusaurus, Fern, etc.) expose the entire docs as a single markdown file at /llms-full.txt (or /llms.txt for an index). When present, this is far better than scraping page-by-page.

Tries, in order:

  1. Same path as the input URL with the doc-root replaced (e.g. https://site.com/docs/foo -> https://site.com/docs/llms-full.txt).

  2. Site root (e.g. https://site.com/llms-full.txt).

Parameters:

  • url (str) – Any URL on the documentation site.

  • timeout (int) – HEAD request timeout.

Returns:

The first URL that returns HTTP 200 with non-trivial content, or None if none of the candidates exist.

Return type:

str | None

contaix.web.html_to_clean_markdown(html, *, body_width=0, ignore_images=True, include_links=True, **html2text_options)[source]

Convert HTML to clean markdown using html2text.

Parameters:

  • html (str) – Raw HTML content.

  • body_width (int) – Line width for wrapping. 0 = no wrapping.

  • ignore_images (bool) – Whether to skip image tags.

  • include_links (bool) – Whether to include hyperlinks in output.

Return type:

str

contaix.web.is_nextjs_site(html)[source]

Detect if a page is served by Next.js (App Router).

Looks for self.__next_f.push in the HTML, which is the RSC streaming payload signature.

Return type:

bool

contaix.web.list_site_pages(url)[source]

List all pages found in a documentation site’s navigation.

Returns a list of dicts with keys: path, title, url, group, tab.

Return type:

list[dict]

>>> pages = list_site_pages('https://platform.claude.com/docs/en/home')
>>> len(pages)
89
contaix.web.parse_rsc_flight(rsc_text)[source]

Parse RSC flight data into a dict mapping keys to parsed JSON data.

Each RSC record starts with <hex_key>:<payload>. Records come in three flavors:

  • I[...]: module import descriptors (skipped here).

  • T<hex_size>,<bytes>: a raw text chunk whose payload occupies exactly hex_size bytes (often markdown, code, or compiled MDX). The payload may contain newlines, so we read by byte count rather than splitting on \n.

  • Anything else: a JSON value terminated by the next record header (\n<hex>:) or end-of-text.

Parameters:

rsc_text (str) – Raw RSC flight response text (text/x-component).

Returns:

Mapping of hex keys to their parsed payloads.

Return type:

dict

contaix.web.rsc_tree_to_markdown(node, *, _registry=None, _depth=0)[source]

Recursively extract markdown from a React Server Component tree node.

The RSC tree uses the format:

["$", "tagName", key, {"children": ..., "className": ...}]

or plain strings for text nodes. Component references like "$L2a" are resolved via _registry (from parse_rsc_flight()).

Return type:

str

contaix.web.site_to_markdown(url, *, cache_dir=None, output_file=None, tab_filter=None, page_fetcher=None, content_extractor=None, section_separator='\\n\\n---\\n\\n', collapse_blank_lines=True, use_llms_full=True, verbose=False)[source]

Download a documentation site and produce a single aggregated markdown.

Parameters:

  • url (str) – The documentation site’s root URL.

  • cache_dir (str) – Directory for caching fetched HTML. Defaults to ~/.cache/contaix/web.

  • output_file (str) – Path to save the markdown file. If None, returns the string.

  • tab_filter (str | Callable) – If a string, only include pages from tabs whose label contains this string (case-insensitive). If a callable, tab_filter(tab_label) -> bool.

  • page_fetcher (Callable) – Custom (url) -> html function. Defaults to fetch_page.

  • content_extractor (Callable) – Custom (html) -> markdown function. Defaults to auto-detection.

  • section_separator (str) – Separator between page sections in the output.

  • use_llms_full (bool) – If True (default), first try to find a publisher-provided /llms-full.txt bundle (Mintlify, Docusaurus, etc. ship these). When found, returns it directly instead of scraping page-by-page. Disable to force the scraping pipeline. Skipped automatically when page_fetcher or tab_filter is set, since those imply the caller wants the scraping path.

  • verbose (bool) – Print progress information.

Returns:

If output_file is None, returns the markdown string. Otherwise, returns the output file path.

Return type:

str

contaix.web.url_to_cache_key(url)[source]

Convert a URL to a filesystem-safe cache key.

Return type:

str

>>> url_to_cache_key('https://example.com/docs/en/foo')
'example.com__docs__en__foo.html'