Keep the first collection small.
Choose a documentation section you own or have permission to collect. This example maps at most five successful pages at depth two, stays on the exact origin and selects only the starting path and its descendants. It excludes archive paths, then makes at most five sequential Scrape requests.
Map discovers an inventory. Crawl also collects content internally but returns page summaries; this guide uses Map followed by Scrape to save full Markdown locally. A Crawl response's summary is not the full page body.
Create a CrawlVolt API key. Give it a name for this project and copy the secret when it appears. It is shown once.
Already registered? Sign in and continue to API keys. Keep the secret in your local environment or credential store.
Budget for both operations: up to five successful Map pages plus five successful Scrape requests in this example. Map can attempt additional pages while filling its successful-page budget. Check reported credits in Activity; URL discovery is not free.
Download two files into the same folder.
Use Node.js 24. Download collect_docs.mjs and its request helper, page_to_markdown.mjs. The helper is explained in the single-page Node.js guide.
export CRAWLVOLT_API_KEY="your_api_key"
node collect_docs.mjs https://www.crawlvolt.com/docs docs-sampleThe output directory must not exist yet. The script creates it before calling the API, writes the raw Map response to map.json and updates manifest.json after every extraction. Markdown filenames come from the source URL, so page titles cannot create unsafe local paths.
Keep discovery and extraction visible.
Map returns objects under links; their url property contains the URL. The script deduplicates that inventory and checks its origin and path again before extraction.
import { createHash } from 'node:crypto';
import { mkdir, writeFile } from 'node:fs/promises';
import { join } from 'node:path';
import { pathToFileURL } from 'node:url';
import { publicUrl, request, requireMarkdown } from './page_to_markdown.mjs';
export async function collectDocs(url, directory, call = request) {
const start = publicUrl(url);
const path = start.pathname.replace(/\/$/, '');
const maxPages = 5;
// Refuse an existing output directory before making any paid request.
await mkdir(directory);
const mapped = await call('map', {
url: start.href,
max_pages: maxPages,
max_depth: 2,
include_patterns: path ? [path, `${path}/*`] : ['/*'],
exclude_patterns: ['*/archive/*'],
timeout_secs: 20,
}, { timeoutMs: 130_000 });
await writeFile(join(directory, 'map.json'), JSON.stringify(mapped, null, 2));
if (!Array.isArray(mapped.links)) throw new Error('Map returned no links array; inspect map.json.');
const urls = [...new Set(mapped.links.map((link) => link.url))].filter((value) => {
try {
const candidate = publicUrl(value);
return candidate.origin === start.origin &&
(candidate.pathname === path || candidate.pathname.startsWith(`${path}/`));
} catch { return false; }
}).slice(0, maxPages);
const report = {
start_url: start.href,
map: { status: mapped.status, truncated: mapped.truncated, stop_reason: mapped.stop_reason },
credits_reported: mapped.credits_used ?? 0,
pages: [],
failures: (mapped.failures ?? []).map((failure) => ({ stage: 'map', ...failure })),
stopped_early: false,
unattempted: [],
};
const saveReport = () => writeFile(join(directory, 'manifest.json'), JSON.stringify(report, null, 2));
await saveReport();
for (const sourceUrl of urls) {
try {
const result = await call('scrape', {
url: sourceUrl, formats: ['markdown'], only_main_content: true,
});
report.credits_reported += result.credits_used ?? 0;
const markdown = requireMarkdown(result);
const filename = `${createHash('sha256').update(sourceUrl).digest('hex').slice(0, 16)}.md`;
await writeFile(join(directory, filename), markdown, { flag: 'wx' });
report.pages.push({ source_url: sourceUrl, markdown_file: filename, characters: markdown.length });
} catch (error) {
report.failures.push({ stage: 'scrape', url: sourceUrl, error: error.message });
if ([401, 402, 403, 429].includes(error.status) || error.name === 'TimeoutError') {
report.stopped_early = true;
report.unattempted = urls.slice(urls.indexOf(sourceUrl) + 1);
}
}
await saveReport();
if (report.stopped_early) break;
}
return report;
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
try {
const [url = 'https://www.crawlvolt.com/docs', directory = 'docs-sample'] = process.argv.slice(2);
const report = await collectDocs(url, directory);
console.log(`${report.pages.length} saved, ${report.failures.length} failures. Inspect ${directory}/manifest.json.`);
if (!report.pages.length || report.failures.length || report.stopped_early) process.exitCode = 2;
} catch (error) {
console.error(error.message);
process.exitCode = 1;
}
}The Map call has a longer client timeout because it visits several pages. Scrape requests stay sequential. Authentication, quota and rate-limit errors stop the remaining extraction work. Ordinary page failures are recorded while the other selected pages continue.
Read the manifest before trusting the dataset.
For each saved page, the manifest records source_url, markdown_file and characters. The failures array distinguishes discovery from extraction failures. A stopped job lists its remaining URLs under unattempted.
Check map.truncated and map.stop_reason in every run. A page budget, depth limit, timeout or an unlinked page can leave relevant documentation undiscovered. Five saved files do not prove that a site has only five pages.
Exit code 2 means the sample has no saved pages, a reported failure or an early stop. It still preserves successful files. A bounded, error-free sample can exit 0 and still have truncated: true; use the manifest to assess coverage.
Open a getting-started page, an API reference page and a page with code or tables. Compare each against its source. If a documentation link points to another origin, collect that origin separately. Review timeout requests in Activity before retrying: this script does not automatically replay them.
Scale from an inspected sample.
Increase scope only after the files are useful and the observed credit usage fits your budget. Keep the manifest with each run so you can trace content back to its URL and distinguish missing pages from empty ones.
Continue with Markdown to a RAG dataset, or read the Map and Crawl reference for the current limits and request fields.