Contextractor for Python (PyPI)

This Python wrapper for Contextractor is currently an alpha, experimental release — not fully tested or officially supported, though still maintained.

contextractor on PyPI lets you crawl web pages and extract clean main-content text — txt, markdown, json, html, or the original raw HTML — from Python.

The package is a thin, typed wrapper that drives the bundled Node engine — it reimplements nothing. A self-contained Node runtime is installed automatically as a dependency, so no separate Node.js install is required. To embed extraction in your own Node.js code instead, see the npm library page.

Install

pip install contextractor
python -m contextractor install   # one-time: download Chromium

Requires Python 3.12+. Platform wheels are published for macOS (arm64, x86_64), Linux (x86_64, aarch64; glibc ≥ 2.28), and Windows (x64). The musl C library is not supported.

Quick start

Crawl a page and write the extracted content to an output directory:

import contextractor

summary = contextractor.extract(
    ["https://example.com"],
    save=["markdown-kvs"],
    output_dir="./out",
    max_requests_per_crawl=10,
)
print(summary)
# ExtractSummary(total=1, succeeded=1, failed=0, skipped=0,
#                output_dir='/abs/out', manifest_path='/abs/out/manifest.json')

urls accepts a single string or a list. Extracted files and a manifest.json index are written to output_dir (default: ./contextractor-output). The manifest is a JSON array of records, each tagged status: "success", "failed", or "skipped".

Async

aextract is the awaitable counterpart of extract, with the same options:

import asyncio
import contextractor

async def main():
    summary = await contextractor.aextract(
        ["https://example.com", "https://example.org"],
        save=["markdown-dataset", "original-kvs"],
        output_dir="./out",
        max_concurrency=5,
    )
    print(summary.succeeded, "of", summary.total)

asyncio.run(main())

Single-page extraction

extract_one() crawls exactly one URL (no link-following) and returns the extracted content directly — no output directory, nothing written to disk:

import contextractor

url = "https://en.wikipedia.org/wiki/Web_scraping"
markdown = contextractor.extract_one(url)
print(markdown)  # str — markdown is the default format

Request several formats to get a dict keyed by format:

contents = contextractor.extract_one(
    "https://example.com",
    formats=["markdown", "json", "original"],
)
print(contents["markdown"])  # extracted markdown
print(contents["original"])  # raw page HTML

With one requested format the return value is a str; with several it is a dict[str, str]. extract_one accepts the single-page subset of the crawl options (the ExtractOneOptions TypedDict) — e.g. proxy, mode, user_agent, cookies, headers, headless — but not crawl-frontier options like globs or max_crawl_depth. A page that cannot be fetched or extracted raises ContextractorError. If the page yields no content for one of several requested formats, that format's key is absent from the returned dict; when the single requested format yields no content, ContextractorError is raised.

aextract_one is the awaitable counterpart:

import asyncio
import contextractor

async def main():
    url = "https://en.wikipedia.org/wiki/Web_scraping"
    markdown = await contextractor.aextract_one(url)
    print(markdown)

asyncio.run(main())

Return value

extract() and aextract() return an ExtractSummary (a frozen dataclass):

Partial failures (some URLs failed) do not raise — they are reflected in summary.failed. Validation and configuration errors, and real crawl failures, raise ContextractorError (see the Errors section below).

Options

All crawl options are typed keyword arguments, surfaced as the ExtractOptions TypedDict so editor autocomplete and type-checkers see every field. A selection:

The output format set is txt, markdown, json, html, and original.

Breaking change. save now takes single format-destination tokens; the separate save_destination option was removed. Rewrite save=["markdown"] plus save_destination=["key-value-store"] as save=["markdown-kvs"].

Config file

Share configuration across runs with a JSON config file. Keys use camelCase:

{
  "mode": "precision",
  "save": ["markdown-kvs", "json-dataset"],
  "maxRequestsPerCrawl": 25,
  "maxCrawlDepth": 2
}

contextractor.extract(
    ["https://example.com"],
    config_file="config.json",
    output_dir="./out",
)

Keyword arguments override values from the config file.

Proxies

Only http, https, socks4, and socks5 proxy URLs are accepted; an unsupported scheme raises ProxySchemeError before anything runs. Proxy credentials are never echoed in errors or logs.

contextractor.extract(
    ["https://example.com"],
    proxy=["http://user:pass@proxy-host:3128"],
    proxy_rotation="per-request",
    output_dir="./out",
)

proxy_rotation controls how the proxy pool is cycled: recommended (default), per-request, or until-failure. It is accepted by extract()/aextract() and extract_one()/aextract_one().

Browser provisioning

Browsers are not bundled in the wheel. Run python -m contextractor install once to download Chromium for the browser-based engines (adaptive, firefox, chromium). The cheerio engine fetches over plain HTTP and needs no browser, so it works without this step. The standard PLAYWRIGHT_BROWSERS_PATH environment variable is honored. You can also call it programmatically:

import contextractor

contextractor.install("chromium")

Errors

Partial failures do not raise; only the conditions above do.

Advanced

• CONTEXTRACTOR_NODE_PATH — point at a host Node binary to use instead of the bundled runtime.
• storage_dir — reuse a Crawlee storage directory across runs (defaults to a private temporary directory cleaned up after each call).
• timeout — per-process wall-clock limit, in seconds.
• PLAYWRIGHT_BROWSERS_PATH — honored for browser provisioning.

Where to go next

• Help hub — the index of every Contextractor help page.
• Playground — enter a URL and preview extraction results in your browser.
• Apify Actor — run extraction at scale on the Apify platform.
• npm CLI — the command-line tool, flag reference, and JSON config.
• npm library — embed extraction in your own Node.js application.

Updated: July 5, 2026