contaix.web

Web content extraction and aggregation tools.

Functions for extracting structured content from websites and converting to markdown, with support for navigation discovery, caching, and content aggregation.

Main entry points:

site_to_markdown(url) # Full pipeline: discover nav, fetch pages, aggregate extract_site_nav(url) # Get ordered navigation structure from a doc site fetch_page(url) # Fetch a single page’s HTML with optional caching

Example:

>>> md = site_to_markdown(
...     'https://platform.claude.com/docs/en/home',
...     cache_dir='/tmp/claude_docs_cache',
...     output_file='~/Downloads/claude_docs.md',
... )
contaix.web.extract_rsc_page_content(rsc_text: str) → str | None[source]

Extract page content from RSC flight data.

Works on both: - RSC flight endpoint responses (text/x-component) - Inline RSC payloads extracted from HTML

Returns markdown text if content was found, None otherwise.

contaix.web.extract_site_nav(url: str, *, html: str = None) → dict[source]

Extract navigation structure from a documentation site.

Returns a dict with keys:

  • base_url: The base URL of the site

  • tabs: List of tab dicts, each with label and groups

  • pages: Flat ordered list of {path, title} dicts (all tabs)

Tries, in order:

  1. Next.js RSC payload (embedded in SSR HTML)

  2. HTML <a> link extraction with path-based grouping

Parameters:

  • url (str) – The documentation site URL (e.g. a homepage or any page on the site).

  • html (str, optional) – Pre-fetched HTML. If None, fetches from url.

contaix.web.fetch_llms_full(url: str, *, timeout: int = 30) → str | None[source]

Fetch a publisher-provided single-doc bundle if available.

See find_llms_full_url() for how the URL is discovered.

Returns the markdown text, or None if no bundle was found.

contaix.web.fetch_nextjs_rsc(url: str, *, cache_dir: str = None, force: bool = False, timeout: int = 30) → str[source]

Fetch page content via Next.js RSC flight endpoint.

Next.js App Router sites serve React Server Component payloads when the RSC: 1 header is present. These contain the full page content that would otherwise require JavaScript rendering.

Parameters:

  • url (str) – The page URL.

  • cache_dir (str, optional) – Directory to cache RSC responses.

  • force (bool) – Re-fetch even if cached.

  • timeout (int) – Request timeout.

Returns:

The RSC flight data as text.

Return type:

str

contaix.web.fetch_page(url: str, *, cache_dir: str = None, force: bool = False, timeout: int = 30) → str[source]

Fetch a page’s HTML content, with optional disk caching.

Parameters:

  • url (str) – The URL to fetch.

  • cache_dir (str, optional) – Directory to cache HTML files. If None, no caching.

  • force (bool) – If True, re-fetch even if cached.

  • timeout (int) – Request timeout in seconds.

Returns:

The page’s HTML content.

Return type:

str

contaix.web.fetch_pages(urls: Iterable[str], *, cache_dir: str = None, force: bool = False, verbose: bool = False) → dict[str, str][source]

Fetch multiple pages with caching.

Returns a dict mapping URL -> HTML content. Pages that fail to fetch are skipped with a warning.

contaix.web.find_llms_full_url(url: str, *, timeout: int = 30) → str | None[source]

Probe for a publisher-provided single-doc bundle.

Many documentation generators (Mintlify, Docusaurus, Fern, etc.) expose the entire docs as a single markdown file at /llms-full.txt (or /llms.txt for an index). When present, this is far better than scraping page-by-page.

Tries, in order:

  1. Same path as the input URL with the doc-root replaced (e.g. https://site.com/docs/foo -> https://site.com/docs/llms-full.txt).

  2. Site root (e.g. https://site.com/llms-full.txt).

Parameters:

  • url (str) – Any URL on the documentation site.

  • timeout (int) – HEAD request timeout.

Returns:

The first URL that returns HTTP 200 with non-trivial content, or None if none of the candidates exist.

Return type:

str or None

contaix.web.html_to_clean_markdown(html: str, *, body_width: int = 0, ignore_images: bool = True, include_links: bool = True, **html2text_options) → str[source]

Convert HTML to clean markdown using html2text.

Parameters:

  • html (str) – Raw HTML content.

  • body_width (int) – Line width for wrapping. 0 = no wrapping.

  • ignore_images (bool) – Whether to skip image tags.

  • include_links (bool) – Whether to include hyperlinks in output.

contaix.web.is_nextjs_site(html: str) → bool[source]

Detect if a page is served by Next.js (App Router).

Looks for self.__next_f.push in the HTML, which is the RSC streaming payload signature.

contaix.web.list_site_pages(url: str) → list[dict][source]

List all pages found in a documentation site’s navigation.

Returns a list of dicts with keys: path, title, url, group, tab.

>>> pages = list_site_pages('https://platform.claude.com/docs/en/home')
>>> len(pages)
89
contaix.web.parse_rsc_flight(rsc_text: str) → dict[source]

Parse RSC flight data into a dict mapping keys to parsed JSON data.

Each RSC record starts with <hex_key>:<payload>. Records come in three flavors:

  • I[...]: module import descriptors (skipped here).

  • T<hex_size>,<bytes>: a raw text chunk whose payload occupies exactly hex_size bytes (often markdown, code, or compiled MDX). The payload may contain newlines, so we read by byte count rather than splitting on \n.

  • Anything else: a JSON value terminated by the next record header (\n<hex>:) or end-of-text.

Parameters:

rsc_text (str) – Raw RSC flight response text (text/x-component).

Returns:

Mapping of hex keys to their parsed payloads.

Return type:

dict

contaix.web.rsc_tree_to_markdown(node, *, _registry: dict = None, _depth: int = 0) → str[source]

Recursively extract markdown from a React Server Component tree node.

The RSC tree uses the format:

["$", "tagName", key, {"children": ..., "className": ...}]

or plain strings for text nodes. Component references like "$L2a" are resolved via _registry (from parse_rsc_flight()).

contaix.web.site_to_markdown(url: str, *, cache_dir: str = None, output_file: str = None, tab_filter: str | Callable = None, page_fetcher: Callable = None, content_extractor: Callable = None, section_separator: str = '\n\n---\n\n', collapse_blank_lines: bool = True, use_llms_full: bool = True, verbose: bool = False) → str[source]

Download a documentation site and produce a single aggregated markdown.

Parameters:

  • url (str) – The documentation site’s root URL.

  • cache_dir (str, optional) – Directory for caching fetched HTML. Defaults to ~/.cache/contaix/web.

  • output_file (str, optional) – Path to save the markdown file. If None, returns the string.

  • tab_filter (str or callable, optional) – If a string, only include pages from tabs whose label contains this string (case-insensitive). If a callable, tab_filter(tab_label) -> bool.

  • page_fetcher (callable, optional) – Custom (url) -> html function. Defaults to fetch_page.

  • content_extractor (callable, optional) – Custom (html) -> markdown function. Defaults to auto-detection.

  • section_separator (str) – Separator between page sections in the output.

  • use_llms_full (bool) – If True (default), first try to find a publisher-provided /llms-full.txt bundle (Mintlify, Docusaurus, etc. ship these). When found, returns it directly instead of scraping page-by-page. Disable to force the scraping pipeline. Skipped automatically when page_fetcher or tab_filter is set, since those imply the caller wants the scraping path.

  • verbose (bool) – Print progress information.

Returns:

If output_file is None, returns the markdown string. Otherwise, returns the output file path.

Return type:

str

contaix.web.url_to_cache_key(url: str) → str[source]

Convert a URL to a filesystem-safe cache key.

>>> url_to_cache_key('https://example.com/docs/en/foo')
'example.com__docs__en__foo.html'