Crawl and search
Discover URLs and collect website pages with bounded CrawlVolt Map, Crawl and Search requests.
CrawlVolt exposes two independent collection primitives: bounded site crawls
and live web search. Use v1.crawl for POST /v1/crawl and v1.search for
POST /v1/search.
Search the live web
curl --request POST https://www.crawlvolt.com/v1/search \
--header "Authorization: Bearer $CRAWLVOLT_API_KEY" \
--header "Content-Type: application/json" \
--data '{
"query": "browser automation APIs",
"limit": 10,
"sources": ["web", "news", "images"],
"categories": ["github"],
"country": "US",
"tbs": "qdr:m"
}'Results are grouped under data.web, data.news, and data.images. Each
requested source returns up to limit results, from 1 to 100.
| Field | Description |
|---|---|
query | Required query, up to 500 characters |
sources | Any combination of web, news, and images; defaults to web |
categories | Optional github, research, and pdf constraints |
include_domains | Search only these hostnames, up to 20 |
exclude_domains | Exclude these hostnames, up to 20 |
tbs | qdr:d, qdr:w, qdr:m, or qdr:y |
country | Two-letter country code |
location | Optional location phrase |
timeout_ms | Request timeout from 1,000 to 120,000 ms |
{
"success": true,
"id": "1d9d7e67-832d-46c8-861c-64af4f189d53",
"request_id": "1d9d7e67-832d-46c8-861c-64af4f189d53",
"query": "browser automation APIs",
"data": {
"web": [
{
"title": "Browser automation documentation",
"description": "API reference and examples.",
"url": "https://example.com/docs",
"position": 1
}
],
"news": [],
"images": []
},
"result_count": 1,
"credits_used": 3,
"warning": null,
"billing": { "billable": true, "units": 3, "unit": "page" }
}Extract search results
Add scrape_options to include page content with web and news results:
{
"query": "CrawlVolt API",
"sources": ["web", "news"],
"limit": 5,
"scrape_options": {
"formats": ["markdown", "links", "screenshot"]
}
}Supported formats are markdown, html, raw_html, links, and
screenshot. Extraction is applied to the first 10 web and news results. A
warning in the response reports a source or extraction limitation without
hiding successful results.
Search uses one page unit per requested source for each block of 10 results. Extracted result pages use one additional page unit each.
Crawl a website
Use POST /v1/map first when you need the bounded URL inventory without
choosing pages in advance. Map accepts the same origin, depth, page-limit, and
path-filter fields as Crawl and returns normalized URLs under links.
curl --request POST https://www.crawlvolt.com/v1/map \
--header "Authorization: Bearer $CRAWLVOLT_API_KEY" \
--header "Idempotency-Key: docs-map-42" \
--header "Content-Type: application/json" \
--data '{"url":"https://example.com/docs/","max_pages":20,"max_depth":2}'curl --request POST https://www.crawlvolt.com/v1/crawl \
--header "Authorization: Bearer $CRAWLVOLT_API_KEY" \
--header "Idempotency-Key: docs-release-42" \
--header "Content-Type: application/json" \
--data '{
"url": "https://example.com/docs/",
"max_pages": 10,
"max_depth": 2,
"include_patterns": ["/docs/*"],
"exclude_patterns": ["*/archive/*"],
"cache_ttl_secs": 300
}'The crawler follows exact-origin HTTP and HTTPS links breadth-first. It removes fragments and common tracking parameters, skips common binary assets, and returns a summary for every collected page.
| Boundary | Free | Pro |
|---|---|---|
| Successful pages per crawl | 10 | 50 |
| Maximum depth | 3 | 3 |
| Maximum crawl duration | 120 seconds | 120 seconds |
| Crawl retention | 24 hours | 24 hours |
Each successful crawl page uses one page unit. Failed pages release their
reservation. A stable Idempotency-Key keeps page billing stable across
retries.