Crawl API
Collect the useful pages, not an unbounded website.
Set depth, page count and path filters, then inspect a summary of every collected page and any failures.
The public playground runs a deliberately small crawl. Higher limits require an account.Bounded collectionControl page count and depth before work starts.
Path filtersInclude useful sections and exclude noisy routes.
Per-page traceInspect status, depth, cache state and failures.
Predictable site ingestion
Move from one URL to a controlled dataset.
CrawlVolt discovers links, keeps work on the selected origin and returns successful pages separately from failures. The limits remain explicit in both the request and the usage ledger.
- Documentation and knowledge-base ingestion
- Catalog or changelog collection
- Small monitoring corpora with stable boundaries
POST /v1/crawl
{
"url": "https://example.com/docs",
"max_pages": 5,
"max_depth": 2,
"include_patterns": ["/docs/**"]
}
200 OK
{ "pages": [...], "failures": [...] }Try the section you need to collect.
Follow the documentation collection guide to save Markdown and keep a manifest of sources and failures.
Start with a small live crawl, inspect every page, then increase capacity without changing the endpoint.