Crawl and search

CrawlVolt can crawl a small, controlled section of a site and retain the result for ranked retrieval. API keys need the v1.crawl scope for both endpoints.

Start a crawl

curl --request POST https://www.crawlvolt.com/v1/crawl \
  --header "Authorization: Bearer $CRAWLVOLT_API_KEY" \
  --header "Idempotency-Key: docs-release-42" \
  --header "Content-Type: application/json" \
  --data '{
    "url": "https://example.com/docs/",
    "max_pages": 10,
    "max_depth": 2,
    "include_patterns": ["/docs/*"],
    "exclude_patterns": ["*/archive/*"],
    "cache_ttl_secs": 300
  }'

The crawler uses breadth-first traversal and returns a summary for each stored page. Filters are glob patterns over the path and query string. The starting URL is always attempted; filters apply to discovered links.

{
  "crawl_id": "crw_0123456789abcdef0123456789abcdef",
  "status": "completed",
  "page_count": 3,
  "failed_count": 0,
  "truncated": false,
  "retention_hours": 24,
  "pages": [
    {
      "url": "https://example.com/docs/",
      "depth": 0,
      "title": "Documentation",
      "summary": "Documentation ...",
      "from_cache": false
    }
  ],
  "billing": { "billable": true, "units": 3, "unit": "page" }
}

Crawl boundaries

Only exact-origin HTTP and HTTPS links are followed. CrawlVolt removes URL fragments and common tracking parameters, skips common binary assets, caps URL discovery and attempts at most twice the requested successful-page count. Every browser request still passes through the scraper egress policy.

Each successful page commits one scrape_page unit. Failed pages release their reservation. A stable Idempotency-Key makes each page's billing identity stable across retries. Cache policy is evaluated per page; cache hits remain commercially billable but record zero browser-runtime units.

Search retained content

curl --request POST https://www.crawlvolt.com/v1/search \
  --header "Authorization: Bearer $CRAWLVOLT_API_KEY" \
  --header "Content-Type: application/json" \
  --data '{
    "crawl_id": "crw_0123456789abcdef0123456789abcdef",
    "query": "authentication token",
    "limit": 10
  }'

Search uses PostgreSQL full-text ranking over the retained title and page text. It returns up to 20 results with url, title, description, a highlighted snippet, score, and page metadata. A crawl is visible only to the account and project that created it. Missing, expired, incomplete, or foreign crawls return 404 crawl_not_found.

Search consumes request-rate capacity but no page quota. Responses are private and return Cache-Control: private, no-store.