contaix.urls

Work with urls

Functions

download_articles([md_string, save_dir, ...])	Download PDF articles from markdown text.
download_articles_by_section([md_string, ...])	Download articles by markdown section.
extract_html_links([markdown])	Extract URLs from HTML anchor tags.
extract_markdown_links([markdown, pattern])	Extract markdown links from a string.
extract_urls([markdown, pattern, extractor])	Extract URLs and their context from a markdown string.
extract_urls_only([markdown])	Extract URLs with empty context.
extract_with_surrounding_context([markdown, ...])	Extract URLs with surrounding text as context.
get_from_clipboard()
remove_hyperlink_crap([string, ...])	Remove unwanted hyperlinks and citations from a string.
verify_urls([src])	Verifies URLs in a markdown string by checking their status codes.

contaix.urls.download_articles(md_string=None, save_dir='/home/runner/Downloads', *, save_non_pdf=False, verbose=True)

Download PDF articles from markdown text. Wrapper around pdfdol.download.

If md_string is None, reads from clipboard (requires pyperclip). See pdfdol.download.download_articles for full documentation.

contaix.urls.download_articles_by_section(md_string=None, rootdir=None, save_non_pdf=False, *, section_marker='###')

Download articles by markdown section. Wrapper around pdfdol.download.

If md_string is None, reads from clipboard (requires pyperclip). See pdfdol.download.download_articles_by_section for full documentation.

contaix.urls.extract_html_links(markdown=None)

Extract URLs from HTML anchor tags.

Parameters:

markdown (str | None) – The markdown or HTML string to process

Return type:

Iterator[tuple[str, str]]

Returns:

Iterator of (anchor_text, url) pairs

contaix.urls.extract_markdown_links(markdown=None, pattern=None)

Extract markdown links from a string.

Note

By default this also extracts bare http(s)://... URLs found in the text. For those, the returned context/name is the URL itself.

Parameters:

• markdown (str | None) – The markdown string to process

• pattern (Pattern | None) – A compiled regex pattern to match markdown links

Return type:

Iterator[tuple[str, str]]

Returns:

Iterator of (context, url) pairs

>>> text = "[Google](https://google.com) and https://github.com"
>>> list(extract_markdown_links(text))
[('Google', 'https://google.com'), ('https://github.com', 'https://github.com')]

contaix.urls.extract_urls(markdown=None, pattern=None, extractor=None)

Extract URLs and their context from a markdown string.

Parameters:

• markdown (str | None) – The markdown string to process

• pattern (Pattern | None) – A compiled regex pattern to match URLs and their context. Defaults to matching both markdown hyperlinks [context](url) and bare http(s)://... URLs.

• extractor (Callable[[Match], tuple[str, str]] | None) –

  A function that extracts (context, url) from a match. Defaults to:

  • (context, url) for markdown hyperlinks

  • (url, url) for bare URLs

Return type:

Iterator[tuple[str, str]]

Returns:

Iterator of (context, url) pairs

>>> text = "[Google](https://google.com) and https://github.com"
>>> list(extract_urls(text))
[('Google', 'https://google.com'), ('https://github.com', 'https://github.com')]

contaix.urls.extract_urls_only(markdown=None)

Extract URLs with empty context.

Parameters:

markdown (str | None) – String containing markdown text

Return type:

Iterator[tuple[str, str]]

Returns:

Iterator of tuples with empty string and extracted URLs

>>> list(extract_urls_only("Check [this link](https://example.com) and https://github.com/user/repo)"))
[('', 'https://example.com'), ('', 'https://github.com/user/repo')]

contaix.urls.extract_with_surrounding_context(markdown=None, context_chars=30)

Extract URLs with surrounding text as context.

Parameters:

• markdown (str | None) – The markdown string to process

• context_chars (int) – Number of characters to include before and after URL

Return type:

Iterator[tuple[str, str]]

Returns:

Iterator of (context, url) pairs

>>> text = "Check this link: [Google](https://google.com) and [GitHub](https://github.com)"
>>> list(extract_with_surrounding_context(text, context_chars=5))
[('gle](https://google.com) and', 'https://google.com)'), ('Hub](https://github.com)', 'https://github.com)')]

contaix.urls.remove_hyperlink_crap(string=None, copy_to_clipboard=True)

Remove unwanted hyperlinks and citations from a string.

Typically used to clean up text copied from ChatGPT or Claude. Delegates to dn.repair.remove_hyperlink_crap for the pure transform, adding clipboard integration on top.

If no string is specified, reads from clipboard (requires pyperclip).

contaix.urls.verify_urls(src=None)

Verifies URLs in a markdown string by checking their status codes.

Parameters:

src (str | list | None) – The markdown string or list containing URLs. If a string starting with [, parsed as a JSON list of URLs. Otherwise, URLs are extracted from the markdown text.

Returns:

A dictionary with URLs as keys and their status codes as values.

Return type:

dict[str, int | str]

>>> verify_urls(['https://example.com'])
{'https://example.com': 200}