gnosis-markdown

Prove where every document came from.

Web → clean, RAG-ready Markdown — with proof of origin baked into every byte.

Markdown is a projection; the raw bytes + WARC are the source of truth.

For RAG engineers, compliance & data-governance teams, and security-sensitive researchers who need to prove — not just assume — where a document came from, gnosis-markdown fetches and crawls any page into LLM-ready Markdown while stamping every file with byte-level SHA-256, WARC archival, and Ed25519 signatures you can verify independently.

Byte-level provenance. Re-fetchable. Re-verifiable. No sidecar bookkeeping.

It started with an audit. I tested my own scraper against a page I'd written, and its "completeness" metric reported 106% retention on a document that had lost a third of its text. The number was arithmetically valid; every test was green. The fix — and the open problem it exposed — are in the self-audit and spec §4.4.

Known limitations, stated plainly: retention_ratio measures how much text survived extraction, not which text (a single dropped table in a long document barely moves it — see §4.4 of the spec). JS rendering is opt-in via a sidecar. The SSRF guard covers direct connections, not proxies. Full disclosure in SECURITY.md.

Why gnosis-markdown

The citation/provenance layer for LLM pipelines — not another scraper.

You can scrape the web with a dozen tools. You can only prove where a document came from with one. gnosis-markdown turns every fetch into an auditable Capture Record: hash the bytes, archive the raw response, record the consent signals, and sign the result — so the document in your RAG index is traceable to the bytes on the wire.

Firecrawl, Crawl4AI and Jina Reader win on speed, scale and hosting. gnosis-markdown doesn't compete there — it wins on auditability, the one axis none of them ship as a first-class feature.

The honest comparison

This table is deliberately not a feature matrix. It claims only the provenance / audit / consent / signing surface gnosis-markdown ships in its own tree — and nothing about speed, scale, or hosting, where we do not claim to compete.

Legend: ✅ first-class, shipped in-tree and verifiable · ⚠️ partial / not the same thing · ❌ not offered as a documented feature · ➖ opaque (managed service; behavior not verifiable in a self-hosted deployment).

As of September 2026. gnosis-markdown claims are verifiable against this repository (--sign, --warc, --profile, gnosis-keygen, gnosis-verify). Competitor columns reflect their public docs at time of writing — re-verify each project against its own repository before relying on this table. Firecrawl and Jina Reader are hosted services: some server-side behavior (e.g. SSRF handling) exists but cannot be verified by a third party in a self-hosted build, hence ➖.

Sources (verified 2026-09-04): gnosis-markdown — 280-test suite and the Capture Record spec · Firecrawl — docs.firecrawl.dev · Crawl4AI — github.com/unclecode/crawl4ai · Jina Reader — jina.ai/reader.

Related MCP server: AIMLPM/markcrawl

Verify it yourself

The provenance claim is the product. After a fetch:

gnosis https://docs.python.org/3/tutorial/ -o out/ --warc

# One command: extract the hash from the frontmatter and re-verify the stored body.
h=$(sed -n 's/^bytes_sha256: *//p' out/*.md | head -1); printf '%s  %s\n' "$h" "out/.gnosis-store/$h" | shasum -a 256 -c -
# prints: <hash>: OK

Every markdown file is re-fetchable and re-verifiable — no sidecar bookkeeping.

Reproducible benchmark evidence: see BENCHMARKS.md.

Quick start

Requires Python 3.12+.

pip install gnosis-markdown

# One page → one markdown file (written to ./ by default; use -o to change it)
gnosis https://docs.python.org/3/tutorial/

# Crawl an entire section
gnosis https://docs.python.org/3/tutorial/ --all -o ./python-docs/

# Archive the raw bytes to WARC + a content-addressed store, and sign the record
gnosis https://docs.python.org/3/tutorial/ --warc --sign --sign-key key.pem

# Emit per-chunk citation manifests for RAG
gnosis https://docs.example.com --chunk

# Export with provenance (JSON / JSONL / Parquet)
gnosis https://docs.example.com --format json

# Evaluate yourself against a corpus (one URL per line)
printf 'https://docs.python.org/3/\nhttps://example.com/\n' > urls.txt
gnosis-bench --urls urls.txt

# Convert a PDF/Office doc to Markdown (pip install gnosis-markdown[docs])
gnosis-doc report.pdf -o report.md

JS rendering is opt-in via a sidecar binary (default obscura); install the Obscura binary, then gnosis https://my-spa.example --render (or set render.engine in config).

Features

Provenance & audit — the moat

• bytes_sha256 — SHA-256 of the response body bytes (after content decoding). You hash the bytes, not the derived text.

• content_hash — SHA-256 of the emitted Markdown, so transforms are auditable too.

• WARC archival (--warc) — WARC-grade evidence, replayable via pywb, plus a content-addressed store keyed on bytes_sha256. Every file is re-fetchable and re-verifiable — no sidecar bookkeeping.

• Ed25519 signing — seal of origin — --sign cryptographically signs each record; gnosis-keygen mints keypairs and gnosis-verify checks them against a pinned key. Prove a document came from a capture you made, untouched.

• Data cards — every scrape/crawl job writes a data-card.json: sources, sizes, licenses encountered, ai.txt/llms.txt coverage, and compliance decisions — one artifact an auditor reads instead of opening every file.

Consent & compliance policy

• ai.txt / llms.txt consent recording — a host's ai.txt directives and llms.txt presence are captured into the frontmatter of every affected file.

• Compliance policy engine — per-page allow_if / deny_if rules with deny-overrides semantics, matched on license, ai.txt directives, and URL path. Decisions are recorded in the frontmatter and data card, not just applied.

• --profile presets — strict-optout (block training/data opt-outs and Disallow: paths) and open-only (permissive/open licenses only).

ai.txt is advisory, not enforced by default. gnosis records a site's ai.txt opt-out but does not, by default, refuse to scrape. To stop scraping at opt-outs, use --profile strict-optout or an explicit deny_if rule. "We record the opt-out and scrape anyway" is exactly the behavior a regulator will ask about — decide it deliberately.

Security

• IP-pinned SSRF guard — blocks loopback, RFC1918, link-local, multicast, CGNAT/6to4/Teredo/NAT64, and every redirect hop — closing the DNS-rebinding TOCTOU by resolving once, validating every address, and dialing only pinned IPs (TLS SNI still uses the hostname, so pinning never weakens TLS).

• robots.txt + politeness respected (per-host rate limiting, Crawl-delay capped), fail-open on errors.

• Auth/custom headers are sent only to the original origin — never replayed to cross-origin redirect targets.

• Secrets via ${ENV_VAR} — keep credentials out of config files and shell history.

Extraction & output

• Clean, main-content Markdown with valid GFM tables, metadata extraction, and boilerplate stripping — plus a retention_ratio / stripped_elements / low_content audit trail over the transform itself.

• Token-aware chunking (--chunk) — stable chunk IDs, heading paths, and exact byte offsets in a per-page .chunks.json citation manifest.

• Multi-format export — --format json|jsonl|parquet, each record carrying full provenance.

• llms.txt / llms-full.txt emission on every crawl.

Crawling at scale

• Incremental crawl + conditional GET — If-None-Match / 304 skip unchanged downloads; a hash-native checkpoint makes --all resumable, growing past max_pages across runs.

Integrations

• MCP server (gnosis-mcp) — expose gnosis as an MCP fetch_and_convert tool that returns provenance-stamped Markdown ([mcp] extra).

• LlamaIndex reader and LangChain document loader — return provenance- stamped Documents ([llamaindex] / [langchain] extras).

• Plugin hooks — pre_fetch / post_fetch / post_process for custom auth, filtering, and post-processing.

• Companion CLIs — gnosis-bench (reproducible scorecard), gnosis-doc (PDF/Office → Markdown), gnosis-keygen / gnosis-verify (signing).

Provenance: the contract

The full machine-readable contract — every field, its exact semantics, and conformance rules — is in the Capture Record Specification.

---
title: Quickstart
url: https://docs.example.com/quickstart
fetched_at: '2026-09-02T08:41:44Z'
content_hash: 1549512c...16fd     # SHA-256 of the markdown body
bytes_sha256: 85052df6...bcb31    # SHA-256 of the response body bytes
status_code: 200
generator: gnosis/2.2.0
etag: '"61e917f4..."'
last_modified: Fri, 31 Jul 2026 16:07:37 GMT
---

Standard YAML, parseable by python-frontmatter, Jekyll, Hugo, Obsidian, and any downstream pipeline. Opt out with --no-frontmatter.

CLI reference

gnosis URL [OPTIONS]

Also available: gnosis-bench (reproducible scorecard), gnosis-doc (document → Markdown), gnosis-keygen (generate a signing keypair), gnosis-verify (verify a signed document), and gnosis-mcp (MCP server).

Installation

pip install gnosis-markdown                  # core
pip install 'gnosis-markdown[sign]'          # Ed25519 signing (cryptography)
pip install 'gnosis-markdown[parquet]'       # Parquet export (pyarrow)
pip install 'gnosis-markdown[mcp]'           # MCP server
pip install 'gnosis-markdown[llamaindex]'    # LlamaIndex reader
pip install 'gnosis-markdown[langchain]'     # LangChain loader
pip install 'gnosis-markdown[docs]'          # document conversion (MarkItDown)
pip install 'gnosis-markdown[qmd]'           # QMD vector-DB indexing

Requires Python 3.12+. See gnosis/config/default.yaml for the full configuration reference.

Development

git clone https://github.com/SHCV-it/gnosis.git
cd gnosis
pip install -e '.[test]'
python -m pytest tests/ -q          # offline suite (localhost fixtures)

Contributing

Contributions are welcome — open an issue first to discuss. Report security issues privately (see SECURITY). See ROADMAP.md for the project plan.

License

MIT — see LICENSE.

Authors: Steffen Hoehne & Ali Zahid Raja, SHCV.IT